Parallel zip library for Slicer

Support
1

There is this python library that supports parallel zip and looks like it is MIT license. Would it be possible to incorporate this into Slicer and use it for MRB creation?

GitHub - fastzip/fastzip: Fast parallel zip file creation

@pieper @jcfr

2

The current implementation of MRB creation is in C++ with libarchive, but it’s very simple (just save everything to a tmp directory and then zip it) so making a python workalike implementation with this could be good. It would be good to have tests to confirm both methods create the same result.

I’ve been thinking for a while though about making a zarr container for data and scenes, which may have some of the same benefits in a more scalable fashion so we should think about where we want to invest time.

3

As a drop-in replacement, we have plan to switch zlib with zlib-ng. See COMP: Switch to modern maintained zlib-ng fork of zlib by jamesobutler · Pull Request #6708 · Slicer/Slicer · GitHub

Instead of transitioning to fastzip (currently unmaintained), we will likely add support for GitHub - facebook/zstd: Zstandard - Fast real-time compression algorithm

1 Like
4

Related ITK discussions:

[…] giving up on zstd as zlib-ng was integrated via #2708 and #2803, adding zstd to NRRD and MetaIO is major undertaking, and there is ongoing OME-Zarr effort.

See Use zstd compression in NRRD and MetaIO · Issue #348 · InsightSoftwareConsortium/ITK · GitHub and Integrate modern optimized zlib · Issue #416 · InsightSoftwareConsortium/ITK · GitHub

5

Thanks Jc. I didn’t see any zip support in zstd, but it looks like zlib-ng provides something we could use: