# Parallel zip library for Slicer

**URL:** <https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290>\
**Category:** Support\
**Created:** [December 7, 2023, 5:46pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290 "2023-12-07T17:46:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [December 7, 2023, 5:46pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290/1 "2023-12-07T17:46:31Z")

</div>

There is this python library that supports parallel zip and looks like it is MIT license. Would it be possible to incorporate this into Slicer and use it for MRB creation?

[GitHub - fastzip/fastzip: Fast parallel zip file creation](https://github.com/fastzip/fastzip)

@pieper @jcfr

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [December 7, 2023, 6:08pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290/2 "2023-12-07T18:08:21Z")

</div>

The current implementation of MRB creation is in C++ with libarchive, but it’s very simple (just save everything to a tmp directory and then zip it) so making a python workalike implementation with this could be good. It would be good to have tests to confirm both methods create the same result.

I’ve been thinking for a while though about making a zarr container for data and scenes, which may have some of the same benefits in a more scalable fashion so we should think about where we want to invest time.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [December 7, 2023, 6:22pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290/3 "2023-12-07T18:22:06Z")

</div>

As a drop-in replacement, we have plan to switch `zlib` with `zlib-ng`. See [COMP: Switch to modern maintained zlib-ng fork of zlib by jamesobutler · Pull Request #6708 · Slicer/Slicer · GitHub](https://github.com/Slicer/Slicer/pull/6708)

Instead of transitioning to `fastzip` (currently unmaintained), we will likely add support for [GitHub - facebook/zstd: Zstandard - Fast real-time compression algorithm](https://github.com/facebook/zstd)

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [December 7, 2023, 6:23pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290/4 "2023-12-07T18:23:56Z")

</div>

Related ITK discussions:

> […] giving up on `zstd` as `zlib-ng` was integrated via [#2708](https://github.com/InsightSoftwareConsortium/ITK/pull/2708) and [#2803](https://github.com/InsightSoftwareConsortium/ITK/pull/2803), adding zstd to NRRD and MetaIO is major undertaking, and there is ongoing OME-Zarr effort.

See [Use zstd compression in NRRD and MetaIO · Issue #348 · InsightSoftwareConsortium/ITK · GitHub](https://github.com/InsightSoftwareConsortium/ITK/issues/348) and [Integrate modern optimized zlib · Issue #416 · InsightSoftwareConsortium/ITK · GitHub](https://github.com/InsightSoftwareConsortium/ITK/issues/416)

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [December 7, 2023, 6:27pm UTC](https://discourse.slicer.org/t/parallel-zip-library-for-slicer/33290/5 "2023-12-07T18:27:40Z")

</div>

Thanks Jc. I didn’t see any zip support in zstd, but it looks like zlib-ng provides something we could use:

> **[GitHub - zlib-ng/minizip-ng: Fork of the popular zip manipulation library...](https://github.com/zlib-ng/minizip-ng)**
>
> Fork of the popular zip manipulation library found in the zlib distribution. - GitHub - zlib-ng/minizip-ng: Fork of the popular zip manipulation library found in the zlib distribution.