@Andythe_great, @luarpy we celebrated PW38 last week and we got good progress in the SystoleOS project. Most of this progress can be utilized for breaking the superbuild. I could support this initiative by transferring this and any future progress in the SystoleOS (and more, if needed).
@Andythe_great @RafaelPalomar where would you like to maintain the project progress and comunication?
I have been thinking about how we might approach this case. It seems to me that while we try to solve the superbuild issue in Slicer, which doesnāt work, we could adapt the configuration files and download the dependencies through several scripts launched from a main one.
This way we do not force changes in the original project files and make only the right changes for this particular case. The changes should be no more than indicating where the dependencies are located with respect to the packaging path.
Then we would only have to apply the packaging via flatpak and everything should be fine (Iām sure bugs will come up during the process).
I have created this repo and given @luarpy full access
As there are still some uncertainties related to the development and the maintenance of this project I think it is prudent to have it as a separate project. We can discuss with the Slicer core developers if at some point this could be integrated in the Slicer organization. @pieper, @lassoan, @jcfr, do you think this is a good plan?
I have quickly scanned the Flatpak documentation and found this info about bundled dependencies (Dependencies ā Flatpak documentation):
As will be seen, bundled dependencies can be automatically
downloaded as part of the build process. It is also possible
to apply patches and perform other transformations.
Having bundled dependencies directly linked to the repositories used by the Slicer superbuild and the SystoleOS patches available in the SystoleOS/gentoo-overlay repository is one possibility. In the SystoleOS project we are trying to contribute all these patches to 3D Slicer, but some of them might take time and some others might not even be accepted.
Yesterday I was preparing some of the main files: manifest.yaml (also the one made by @Andythe_great), icon image and *.destkop file. All according to flatpak format. So I can do a push to start with something.
I also have a script I was preparing to automate the configuration process prior to packaging.
On the subject of what you indicate, I also read it but I did not understand how or what is meant by these libraries. I would have liked to find an example showing how to indicate the libraries to download.
1 Like
@luarpy, this is great.
Here is a ChatGPT-generated manifest bundling an ITK and a VTK. Does it look plausible? Keep in mind that we need to use Slicer/VTK and Slicer/ITK instead of vanilla ITK and VTK:
{
"app-id": "org.example.app",
"runtime": "org.freedesktop.Platform",
"runtime-version": "20.08",
"sdk": "org.freedesktop.Sdk",
"command": "app",
"finish-args": [
"--socket=x11",
"--device=dri",
"--filesystem=host"
],
"modules": [
{
"name": "ITK",
"buildsystem": "simple",
"build-commands": [
"wget https://github.com/InsightSoftwareConsortium/ITK/releases/download/v5.1.2/InsightToolkit-5.1.2.tar.gz",
"tar -xzf InsightToolkit-5.1.2.tar.gz",
"cd InsightToolkit-5.1.2",
"mkdir build",
"cd build",
"cmake ..",
"make",
"make install DESTDIR=${FLATPAK_DEST}"
],
"sources": [
{
"type": "file",
"path": "InsightToolkit-5.1.2.tar.gz"
}
]
},
{
"name": "VTK",
"buildsystem": "simple",
"build-commands": [
"wget https://github.com/Kitware/VTK/releases/download/v10.3.1/VTK-10.3.1.tar.gz",
"tar -xzf VTK-10.3.1.tar.gz",
"cd VTK-10.3.1",
"mkdir build",
"cd build",
"cmake ..",
"make",
"make install DESTDIR=${FLATPAK_DEST}"
],
"sources": [
{
"type": "file",
"path": "VTK-10.3.1.tar.gz"
}
]
}
]
}
Getting this to work would be a good starting point. I hope we can rely on the stock Flatpak runtime system for everything below VTK. This is de dependency graph for Slicer on SystoleOS (no python support yet):
* dependency graph for sci-medical/Slicer-9999
`-- sci-medical/Slicer-9999
`-- sci-medical/ctk-0.1.0 (sci-medical/ctk) [python]
`-- sci-libs/vtkAddon-0.1.0 (sci-libs/vtkAddon) [python]
`-- dev-qt/qtcore-5.15.8-r1 (dev-qt/qtcore) amd64
`-- dev-qt/linguist-tools-5.15.8 (dev-qt/linguist-tools) amd64
`-- dev-qt/qtmultimedia-5.15.8 (dev-qt/qtmultimedia) amd64 [widgets]
`-- dev-qt/qtopengl-5.15.8 (dev-qt/qtopengl) amd64
`-- dev-qt/qtsql-5.15.8 (dev-qt/qtsql) amd64
`-- dev-qt/qtxmlpatterns-5.15.8 (dev-qt/qtxmlpatterns) amd64
`-- dev-qt/qtx11extras-5.15.8 (dev-qt/qtx11extras) amd64
`-- dev-qt/qtsvg-5.15.8 (dev-qt/qtsvg) amd64
`-- dev-qt/qtwebengine-5.15.8_p20230112 (dev-qt/qtwebengine) amd64
`-- dev-qt/qtwebchannel-5.15.8 (dev-qt/qtwebchannel) amd64
`-- dev-qt/qtscript-5.15.8 (dev-qt/qtscript) amd64
`-- dev-qt/designer-5.15.8 (dev-qt/designer) amd64
`-- dev-libs/rapidjson-1.1.0-r3 (dev-libs/rapidjson) amd64
`-- dev-libs/jsoncpp-1.9.5 (dev-libs/jsoncpp) amd64
`-- dev-libs/qRestAPI-9999 (dev-libs/qRestAPI) ~amd64
`-- sci-medical/CTKAppLauncherLib-1.0.0 (sci-medical/CTKAppLauncherLib) ~amd64
`-- sci-medical/teem-1.12.0 (sci-medical/teem) ~amd64
`-- Slicer-CLI/SlicerExecutionModel-1.0.0 (Slicer-CLI/SlicerExecutionModel)
`-- sci-libs/itk-5.3.0 (sci-libs/itk) [vtkglue deprecated]
`-- sci-libs/vtk-9.1.0 (>=sci-libs/vtk-9.1.0) [qt5 rendering gl2ps]
`-- dev-util/cmake-3.24.3 (>=dev-util/cmake-3.23.1) amd64
`-- dev-util/ninja-1.11.1-r2 (>=dev-util/ninja-1.8.2) amd64
`-- dev-vcs/git-2.39.1 (>=dev-vcs/git-1.8.2.1) amd64 [curl]
It is the most unexpected answer possible hahahahahaha 
. It seems to be fine but needs some changes. I wrote it in YAML because it allows to add comments to the document.
This is what I wrote based in the documentation of flatpak and using the Weasis Flatpak project as an example:
app-id: org.slicer.Slicer
runtime: org.freedesktop.Platform
runtime: '22.08'
sdk: org.freedesktop.Sdk
base-version: '22.08'
command: slicer
finish-args:
# X11 + XShm access
- --share=ipc
- --socker=x11
# Wayland access
- --socket=wayland
# Needs to talk to the network:
- --share=network
# Needs to save files locally
- --filesystem=xdg-documents
- --filesystem=xdg-download
modules:
- name: VTK
buildsystem: cmake
builddir: true
config-opts:
- -DVTK_BUILD_ALL_MODULES=OFF
- -DBUILD_SHARED_LIBS=YES
- -DOpenGL_GL_PREFERENCE=LEGACY
- -DVTK_WRAP_JAVA=OFF
- -DVTK_WRAP_PYTHON=YES
- -DVTK_PYTHON_VERSION:STRING=3
- -DVTK_GROUP_ENABLE_Qt=YES
- -DVTK_GROUP_ENABLE_Views=YES
- -DVTK_MODULE_ENABLE_VTK_GUISupportQt=YES
- -DVTK_MODULE_ENABLE_VTK_GUISupportQtOpenGL=YES
- -DVTK_MODULE_ENABLE_VTK_RenderingContextOpenGL2=YES
- -DVTK_MODULE_ENABLE_VTK_RenderingFreeTypeFontConfig=YES
- -DVTK_MODULE_ENABLE_VTK_RenderingQt=YES
- -DVTK_MODULE_ENABLE_VTK_GUISupportQtSQL=YES
- -DVTK_MODULE_ENABLE_VTK_FiltersExtraction=YES
- -DVTK_MODULE_ENABLE_VTK_FiltersFlowPaths=YES
- -DVTK_MODULE_ENABLE_VTK_FiltersGeometry=YES
- -DVTK_MODULE_ENABLE_VTK_FiltersParallel=YES
- -DVTK_MODULE_ENABLE_VTK_IOExport=YES
- -DVTK_MODULE_ENABLE_VTK_IOImage=YES
- -DVTK_MODULE_ENABLE_VTK_IOLegacy=YES
- -DVTK_MODULE_ENABLE_VTK_IOPLY=YES
- -DVTK_MODULE_ENABLE_VTK_IOXML=YES
- -DVTK_MODULE_ENABLE_VTK_ImagingMath=YES
- -DVTK_MODULE_ENABLE_VTK_ImagingMorphological=YES
- -DVTK_MODULE_ENABLE_VTK_ImagingStatistics=YES
- -DVTK_MODULE_ENABLE_VTK_ImagingStencil=YES
- -DVTK_MODULE_ENABLE_VTK_InteractionImage=YES
- -DVTK_MODULE_ENABLE_VTK_RenderingVolumeOpenGL2=YES
- -DVTK_MODULE_ENABLE_VTK_TestingRendering=YES
- -DVTK_MODULE_ENABLE_VTK_ViewsQt=YES
- -DVTK_MODULE_ENABLE_VTK_zlib=YES
#cleanup:
#- /bin
sources:
- type: archive
url: https://www.vtk.org/files/release/9.0/VTK-9.0.3.tar.gz
sha256: bc3eb9625b2b8dbfecb6052a2ab091fc91405de4333b0ec68f3323815154ed8a
- name: CTK
buildsystem: cmake
builddir: true
config-opts:
- -DCMAKE_BUILD_TYPE:STRING=Release
- -DCTK_QT_VERSION=5
- -DCTK_BUILD_SHARED_LIBS=ON
- -DBUILD_TESTING=OFF
- -DCTK_SUPERBUILD=OFF
- -DCTK_ENABLE_DICOM=OFF
- -DCTK_ENABLE_PluginFramework=ON
- -DCTK_ENABLE_Widgets=ON
sources:
- type: git
url: git://github.com/commontk/CTK.git
commit: ec9fcee532fd9cdff74f96479197f626d8c71fe5
- name: CTK-PythonQt
buildsystem: cmake
config-opts:
- -DPythonQt_Wrap_QtAll=ON
sources:
- type: git
url: git://github.com/commontk/PythonQt.git
branch: patched-9
- name: Slicer
buildsystem: cmake
builddir: true
config-opts:
- -DCMAKE_BUILD_TYPE:STRING=Release
- -DSlicer_VTK_VERSION_MAJOR=9
build-options:
build-args:
build-commands:
- mkdir -p build
- cd build
- cmake ../Slicer/Slicer
- make
- mv Slicer-build/* ../app/
- install -D slicer.png /app/share/icons/hicolor/400x400/apps/org.slicer.Slicer.png
- install -D -t /app/share/applications/ org.slicer.Slicer.desktop
sources:
- type: file
path: org.slicer.Slicer.desktop
- type: file
path: org.slicer.Slicer.png
- type: git
url: git://github.com/Slicer/Slicer.git
tag: main
A good mix of the two is a good start. Would you mind to make it a PR so I can give it a shot? As I see the problem now we can go a long way by translating the SystoleOS packages to this yaml representation.
UPDATE: Based on the discussion held on 2023.02.07 Weekly Meeting with the Slicer core devs, it seems reasonable to try approaching the Slicer flatpak under a superbuild approach rather than a non-superbuild as I initially proposed. Reasons: (1) some of the SystoleOS patches may take time to land on the Slicer code base, some might not even make it; (2) the distributed Slicer should be the same (or very similar) regardless of whether this is a flatpak or a redistributable linux binary, as of today this a superbuild-based.
@luarpy and @RafaelPalomar have started putting together a Flatpak manifest generator for Slicer (this is needed to set the right versions for the dependencies in the manifest) in here: GitHub - RafaelPalomar/Slicer-Flatpak: Flatpak for 3D Slicer. The final manifest will go on its own repository (so far, GitHub - RafaelPalomar/org.slicer.Slicer: Flatpak repository for Slicer). It is still a non-usable work-in-progress, where we try to reproduce the suggestions above by @jcfr.
It is great to see all the progress with the build. I can help with runtime questions. More specifically: what folders can be read-only and what temporary and persistent writable folders we need for application settings, extensions, Python packages, DICOM database, cached downloads, etc.
This is great @lassoan, thank you. For sure we will need help to deal with the runtime configuration. Weāll keep updating on the progress of this.
1 Like
I leave two interesting pieces of information I found recently:
- I donāt think it would affect this development, but Ubuntu is planning to drop Flatpak integration by default (users will still be able to install the support manually as it is for many other distributions). More information on this post:
- Also found an interesting FOSDEM 2023 talk about Flatpaks, Snaps and AppImages from an OpenSUSE developer:
Whatās the TL;DW for this?
From worse to better:
- AppImages: āfail in everything they set out to doā ā Donāt use them.
- Linux apps running everywhere ā They donāt
- Easy to install apps as in Windows or Mac ā They are not.
- Provide applications without the need to āget into a distributionā or ābuild for a different distributionsā ā Practically, you need to learn every distro and build one.
- Snaps ā Practically confined to Canonical ecosystems.
-
Still very limited to Ubuntu ecosystem (wonāt deliver confinement in most Linux distributions).
-
Very dependent of Canonical. There is a pricing model for serious delivery of apps and there is no open-source delivery option (open format, but not open delivery).
- Flatpak ā The only option ready for primetime
-
Reasonable isolation model. The client is easy to integrate in any Linux distribution
-
Translates licensing and dependencies freedom and responsibility to the packagers
1 Like
What is required to resurrect the idea of Flatpak builds? Being able to discover it and install using package managers can be handy.
As I understand it, there a few problems.
- 3D Slicer wants optimized binary, thatās why all libs are statically compiled with different options than come from Linux distributions
- Linux distros prefer offline builds, and 3D Slicer fetches dependencies dynamically during build.
- Not clear how to compile, discover and install extension, which should be built with the same options.
@abitrolly, I have participated deeply on the efforts to bring 3D Slicer to its distribution based on flatpaks, and even managed to produce a working 3D Slicer flatpak. After that process, I came up with the conclusion that these efforts wonāt be sustainable in the current context, but maybe is worth re-visiting the topic in the future.
TLDR: In my view, a sustainable flatpak offer would require 3D Slicer community to assume the equivalent effort of maintaining 3D Slicer for an additional operating system (Windows+Linux+MacOS+Linux-Flatpak).
Long answer:
- 3D Slicer is not just a desktop application but a complex software stack. It deploys components ranging from application (high level) to fundamental libraries (low level). The motivation to bring all these components along is to ensure a consistent ecosystem across platforms (Windows, GNU/Linux and MacOS). Flatpak builds are based on its own community-driven SDKs, which provide already a build ecosystem and base libraries. Adapting the development logic of 3D Slicer to one of these SDKs (1) wonāt guarantee 3D Slicer runs on GNU/Linux without flatpak (which arguably many users would still want); (2) may (will) create divergences with other OS; and (3) may (will) impose a pace of changes to keep up the compatibility with the SDK.
- 3D Slicer has over 100 extensions, which will also need to be built on the premise of a flatpak SDK. In my experience, many extensions distributed through the Extensions Manager, were not compatible with flatpak 3D Slicer (ABI incompatibility). Migration of all of these to a flatpak SDK may require the developers of some extensions to assume an adaptation (superbuild extensions will be particularly painful).
- Flatpak does not support file downloading during the build process and 3D Slicer largely builds as Superbuild. There are workarounds for this and the 3D Slicer CMake infrastructure has some support for inputting offline dependencies. My approach to flatpak was generating a script that pulls Slicer, configures and parses dependencies to automatically generate the flatpak manifest. This approach was very fragile. A sustainable approach would require the CMake infrastructure of 3D Slicer (and not only, as other dependencies are also a Superbuilds e.g., CTK) to generate the flatpak manifest (e.g.,
make flatpak-manifest). This is not a small undertaking. - Even when a Slicer flatpak could be built, there are some additional changes required for Slicer to run in a flatpak environment. The flatpak underlying system is immutable at runtime. Currently Slicer relies on write access to directories inside its own Slicer root directory (both for new pip-installed python packages and for extensions). These can tecnichally move to user directories, but there is some work to do and considerations to what happens when a new Slicer version comes up, etc.
3 Likes
Thanks for the detailed review @RafaelPalomar I agree this sounds like creating a fourth package, and will likely to create a quite a bit of extra burden on the maintainers. We are already quite stretched maintaining consistent functionality across three major OSes.
I am predominantly a linux user myself, and while I see the appeal of being able to install Slicer from a package manager, I think the download numbers are not going to justify to invest the time into this. I also donāt think the Linux usage will go off the roof after introducing flatpak version.
Having said that, there might be minor (hopefully) tweaks to improve the user experience in Linux:
- Perhaps we can embed a Slicer.desktop file to make Slicer part of the application list
- If we can detect and return the list of missing base libraries necessary to run Slicer on that specific Linux environment after the first run. (As opposed to user going to the downloads page and try to troubleshoot that).
Do you think these would be possible or worthwhile, without adding too many layers of complexity?
1 Like
One thing Iād like to see, for all platforms, would be a lightweight Slicer installation manager. This would be a small program, ideally with a CLI and GUI interface that would do the download and install steps for you and check for new versions of Slicer and extensions (possibly as a background service). Maybe also clean up old versions and other tasks that would be hard to do in the application itself.
It could make sense to implement the functionality in slicerio so itās easy to get from pip, and then package up an application in flatpack, and the various OS app stores.
Is that script/cmake public? Iām particularly interested in building in a offline sandbox, as this would greatly aid efforts to build and run 3D Slicer using more rigorously controlled SBOM using Nix. E.g:
Extensions are locked to a matching release version due to ABI compatibility, no? Wouldnāt that simply imply user appdata folders be similarly version namespaced, mimicking portable installs?
Not sure Iām the best to add a pitch for Nix, but it does address quite a few of those points:
- If Nix can source build the package, there is no extra effort to maintain broader linux support
- Nix is more strict at dependency management, but re-uses the same build systems
- Nix natively supports both MacOS and any Linux distro (not just NixOS) with binary caching
2. Developing for a nix package re-uses much the same setup as installing a nix package - Nix has been around for two decades, with the language syntax being quite stable
3. At the same time, nixpkgs is also quite fresh, enabling use of more up-to-date libraries
