Cannot serialize a bytes object larger than 4 GiB - shelve

I am trying to run partial incremental PCA with the following


But I get the following error when I trying this:

file[‘eigenModes’] = decomposer.components_
OverflowError: cannot serialize a bytes object larger than 4 GiB

I found that link where it says that I need to point protocol 4 for pickle. I checked that the protocol you are using is 3, and I am wondering if there is any way to use protocol 4 through shelve or if you have any other idea?


The solution is described at the link - you need to specify the pickle protocol:

pickle.dump(d, open("file", 'w'), protocol=4)

Yes, true, I was thinking more if there is a way to use pickle protocol 4 from shelve lib, but I will use it directly.

I don’t think shelve library is bundled with Slicer, so you can install any version and configure it any way you want.

For example, you can specify protocol in (

1 Like