# Cannot serialize a bytes object larger than 4 GiB - shelve

**URL:** <https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454>\
**Category:** Development\
**Created:** [July 9, 2020, 2:20am UTC](https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454 "2020-07-09T02:20:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![siaeleni](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@siaeleni](https://discourse.slicer.org/u/siaeleni)\
**Post date:** [July 9, 2020, 2:20am UTC](https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454/1 "2020-07-09T02:20:29Z")

</div>

Hello,  
I am trying to run partial incremental PCA with the following

> decomposer.partial\_fit

But I get the following error when I trying this:

> file[‘eigenModes’] = decomposer.components\_  
> OverflowError: cannot serialize a bytes object larger than 4 GiB

I found that link [python - \_pickle in python3 doesn't work for large data saving - Stack Overflow](https://stackoverflow.com/questions/29704139/pickle-in-python3-doesnt-work-for-large-data-saving) where it says that I need to point protocol 4 for pickle. I checked that the protocol you are using is 3, and I am wondering if there is any way to use protocol 4 through shelve or if you have any other idea?

Thanks,  
Eleni

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [July 9, 2020, 2:28am UTC](https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454/2 "2020-07-09T02:28:44Z")

</div>

The solution is described at the link - you need to specify the pickle protocol:

```
pickle.dump(d, open("file", 'w'), protocol=4)
```

---

<div class="post-metadata">

**Author:** ![siaeleni](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@siaeleni](https://discourse.slicer.org/u/siaeleni)\
**Post date:** [July 9, 2020, 5:04pm UTC](https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454/3 "2020-07-09T17:04:39Z")

</div>

Yes, true, I was thinking more if there is a way to use pickle protocol 4 from shelve lib, but I will use it directly.  
Thanks!

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [July 9, 2020, 5:21pm UTC](https://discourse.slicer.org/t/cannot-serialize-a-bytes-object-larger-than-4-gib-shelve/12454/4 "2020-07-09T17:21:15Z")

</div>

I don’t think shelve library is bundled with Slicer, so you can install any version and configure it any way you want.

For example, you can specify protocol in `shelve.open` ([https://docs.python.org/3/library/shelve.html](https://docs.python.org/3/library/shelve.html)).