Cannot serialize a bytes object larger than 4 GiB - shelve

Development
1

Hello,
I am trying to run partial incremental PCA with the following

decomposer.partial_fit

But I get the following error when I trying this:

file[‘eigenModes’] = decomposer.components_
OverflowError: cannot serialize a bytes object larger than 4 GiB

I found that link python - _pickle in python3 doesn't work for large data saving - Stack Overflow where it says that I need to point protocol 4 for pickle. I checked that the protocol you are using is 3, and I am wondering if there is any way to use protocol 4 through shelve or if you have any other idea?

Thanks,
Eleni

2

The solution is described at the link - you need to specify the pickle protocol:

pickle.dump(d, open("file", 'w'), protocol=4)
3

Yes, true, I was thinking more if there is a way to use pickle protocol 4 from shelve lib, but I will use it directly.
Thanks!

4

I don’t think shelve library is bundled with Slicer, so you can install any version and configure it any way you want.

For example, you can specify protocol in shelve.open (https://docs.python.org/3/library/shelve.html).

1 Like