When I look at the dashboard for extension and sort them by time, there is a clear pattern. Extensions are build sequentially by the operating system.
Is there are reason we are doing that? Can’t they be parallelized? Like Linux is uploaded over 12h ago and Mac extension builds hasn’t even finished, and already half of the workday is gone.
Are you asking why aren’t we building them in parallel per os - we should probably work on this.
Are you asking if we are finishing the Linux and Windows extensions and only then building macos extensions? - that is not what is happening. The macos build/test process is just taking forever, for macos reasons, mainly that a single crash tested can trigger the crash reporter, causing tests to go to the timeout.
In general, the linux app build finishes first, then Windows and later macos, just based on their own build times
The former, why aren’t we parallelazing the build per os. They are independent anyways.
I understand macos builds takes longer. But from my reading of the time line, the Macos extension build starts after the windows and Linux is done, hence my suggestion of making them go in parallel.
They are not independent, (extensions can have dependencies on other extensions). We would need to redo the extension build system to addresss this.
They are already in parallel. They are running on three separate machines with no timing interaction with each other. The macos extension build starts once the macos app build/test is done, the same as the other two platforms. The reason why macos starts “after” the other two is because the mac build takes so much longer
Looking at this thread, I am wondering if we are misunderstanding each other on what (1) and (2) are.
1 - Building SlicerIGT and SlicerRt (for example) in parallel on one OS - we would need to work on the build system a bit for this
2 - MacOS, Linux, and Windows app/extension builds are already running completely independently from each other.
So the just mac build itself takes about 11h+ then? (Dashboard shows started 16h ago, and uploaded 5h ago). And then another 5h for the MacOS extensions to build?
Build time on macOS is very long due to packaging, which includes updating shared library paths in thousands of binary files.
Extension builds could be probably made somewhat faster by building them in parallel, but we should really move towards building each extension immediately in github actions. That would allow developers to create and test packages any time in a few minutes and publish them on the extensions index without waiting for the next day. It would also mean less load on Kitware build servers and less worrying about security issues associated with building constantly changing third-party code.
Packaging of Python-only extensions already works, you only need to add a very small (8 lines) script file to your extension. See more details here: ENH: Add script to package native Python extensions by lassoan · Pull Request #9118 · Slicer/Slicer · GitHub
There seems to be a shortage of intel macOS github runners, so macOS packages may be delayed, but this will be solved when we switch to native ARM builds.
The next step will be streamlining of extensions submission/update process. We can get the list of all the available extensions on github (via the 3d-slicer-extension github topic) and we could use that to populate an index dynamically. But it still makes sense to have a central repository of extension packages for archiving and performance reasons; and the tier system (1=experimental; 3=stable; 5=supported by Slicer core team) also needs centralized curation process.
This looks interesting and useful. But I guess unless we switch to pure ARM build for maOS, it is not going to be much of help shortening the build time. Will the next stable 5.12 be native macos application or still use Rosetta?
Per most recent Slicer hangout meeting notes, the next stable 5.12 will still require Rosetta for Apple Silicon Macs. There are likely additional build dependencies and build updates to make native arm builds possible. An Apple Silicon Mac will also need to be identified that will do the Slicer factory building, packaging and testing. Of course part of this work is making sure various parts extensions (primarily the compiled ones) are also still working on arm architecture. There are arm systems running Windows, Linux and macOS so it could be for all platforms.
Apple will discontinue support for Rosetta 2 sometime in 2027, if I understand it correctly.
This email is intended for non-work related messages
Yes Rosetta 2 is planned to not be present in macOS 28 expected release in October 2027, so we technically have some time where using Rosetta builds is still possible for the most recent version of macOS. Apple supports 3 given versions at any one time so macOS 27 with Rosetta 2 will still be receiving security updates likely up until macOS 30 (fall of 2029). Of course we will aim to provide native arm builds sooner especially with certain Python packages no longer providing pre-built whls for Intel Mac.
