I am trying to run partial incremental PCA with the following
But I get the following error when I trying this:
file[‘eigenModes’] = decomposer.components_
OverflowError: cannot serialize a bytes object larger than 4 GiB
I found that link https://stackoverflow.com/questions/29704139/pickle-in-python3-doesnt-work-for-large-data-saving where it says that I need to point protocol 4 for pickle. I checked that the protocol you are using is 3, and I am wondering if there is any way to use protocol 4 through shelve or if you have any other idea?