Cannot serialize a bytes object larger than 4 GiB - shelve

Hello,
I am trying to run partial incremental PCA with the following

decomposer.partial_fit

But I get the following error when I trying this:

file[‘eigenModes’] = decomposer.components_
OverflowError: cannot serialize a bytes object larger than 4 GiB

I found that link https://stackoverflow.com/questions/29704139/pickle-in-python3-doesnt-work-for-large-data-saving where it says that I need to point protocol 4 for pickle. I checked that the protocol you are using is 3, and I am wondering if there is any way to use protocol 4 through shelve or if you have any other idea?

Thanks,
Eleni

The solution is described at the link - you need to specify the pickle protocol:

pickle.dump(d, open("file", 'w'), protocol=4)

Yes, true, I was thinking more if there is a way to use pickle protocol 4 from shelve lib, but I will use it directly.
Thanks!

I don’t think shelve library is bundled with Slicer, so you can install any version and configure it any way you want.

For example, you can specify protocol in shelve.open (https://docs.python.org/3/library/shelve.html).

1 Like