Is anyone else using ninja to build on mac? Lately I’m noticing that ninja -j2 in Slicer-build runs 8 clang processes simultaneously, which floors the (2-core) computer and the build absolutely crawls (one .o file every 2-3 seconds). I expect this for some parallel projects during superbuild, because I don’t think we’re using job pool support. But within Slicer-build I thought it should only be building one project so cmake+ninja should respect -j2.
(CMake 3.9.4, macOS 10.12)
The way forward could be to:
Use CMake 3.11 to benefit from this fix: Ninja: add CMAKE_JOB_POOLS variable as default for JOBS_POOLS
Improve ExternalProjectDependency (and update the version of the module user in Slicer and CTK) so that it pass the variables CMAKE_JOB_POOLS, CMAKE_JOB_POOL_COMPILE and CMAKE_JOB_POOL_LINK to any CMake based external project project if their are set.
You could then configure the top-level project using -DCMAKE_JOB_POOLS:STRING=compile=5;link=2.
This would mean that each instance of ninja would be limited to 5 compile jobs and 2 linker job.
Would that be helpful ? It may be worth exploring what would be sensible setup for Slicer.
Do you know of other projects using job pools + externalproject? From my understanding, I don’t think it would help because each EP spawns a separate ninja instance, and ninja/cmake don’t have any way to communicate a rate-limit across sub-projects (from what I can tell job pool is designed for rate-limit when a single top-level ninja is in control). I guess fixing this at the superbuild level would need job server support in ninja.
Support for the CMAKE_JOB_* variables was added to Slicer in r27117
That alone should help reduce the relative number of link job vs build job, considering that link job are require more memory, it is sensible to set a lower number.
Now, the good news is that I created a project to distribute the version of ninja with Jobserver support, see
I already have binary for Linux available, the macOS and Windows still need to be generated.
Cc: @thewtex
Did you happen to test if it works with the cmake job pool variable already?
Look like it is working as expected.
~/Software/cmake-3.11.0-Linux-x86_64/bin/cmake \
-GNinja \
-DQt5_DIR:PATH=$Qt5_DIR \
-DCMAKE_BUILD_TYPE:STRING=Release \
'-DCMAKE_JOB_POOLS:STRING=compile=4;link=3' \
-DCMAKE_JOB_POOL_COMPILE:STRING=compile \
-DCMAKE_JOB_POOL_LINK:STRING=link \
../Slicer-Qt5-VTK9
Make sure you add the extra quote around -DCMAKE_JOB_POOLS:STRING=compile=4;link=3
All of that should enable us to fixup CircleCI, and also release Slicer on our Linux server using https://github.com/Slicer/SlicerBuildEnvironment/
As a related note, if you use Archlinux, I just uploaded ninja-kitware to the AUR. It is my second package ever, so please, report any issue if you see one.