There is this python library that supports parallel zip and looks like it is MIT license. Would it be possible to incorporate this into Slicer and use it for MRB creation?
The current implementation of MRB creation is in C++ with libarchive, but it’s very simple (just save everything to a tmp directory and then zip it) so making a python workalike implementation with this could be good. It would be good to have tests to confirm both methods create the same result.
I’ve been thinking for a while though about making a zarr container for data and scenes, which may have some of the same benefits in a more scalable fashion so we should think about where we want to invest time.
As a drop-in replacement, we have plan to switch
zlib-ng. See COMP: Switch to modern maintained zlib-ng fork of zlib by jamesobutler · Pull Request #6708 · Slicer/Slicer · GitHub
Instead of transitioning to
fastzip (currently unmaintained), we will likely add support for GitHub - facebook/zstd: Zstandard - Fast real-time compression algorithm
Related ITK discussions:
Thanks Jc. I didn’t see any zip support in zstd, but it looks like zlib-ng provides something we could use: