Interest to create Flatpak for 3D Slicer, have issue with GUISupportQtOpenGL not found

Hello,

I want to create Flatpak for 3D Slicer.
Flatpak is a packaging and distributing desktop software on Linux.
https://www.flatpak.org/

I would like it to be submit to Flathub, an app store for Flatpak apps. Many Linux distros supported Flatpak. So I think it would be beneficial to package large software like 3D Slicer into Flatpak, it will make installing 3D Slicer quick and easy.
https://flathub.org/home

This is what I have been working on so far.

I build VTK 9.0.1 with the following flags,

> - -DVTK_BUILD_ALL_MODULES=OFF 
> - -DBUILD_SHARED_LIBS=YES
> - -DOpenGL_GL_PREFERENCE=GLVND
> - -DVTK_WRAP_JAVA=OFF
> - -DVTK_WRAP_PYTHON=YES
> - -DCMAKE_BUILD_TYPE=Release
> - -DVTK_PYTHON_VERSION:STRING=3
> - -DVTK_GROUP_ENABLE_Qt=YES
> - -DVTK_GROUP_ENABLE_Views=YES
> - -DVTK_MODULE_ENABLE_VTK_GUISupportQt=YES
> - -DVTK_MODULE_ENABLE_VTK_RenderingContextOpenGL2=YES
> - -DVTK_MODULE_ENABLE_VTK_RenderingFreeTypeFontConfig=YES
> - -DVTK_MODULE_ENABLE_VTK_RenderingQt=YES
> - -DVTK_MODULE_ENABLE_VTK_GUISupportQtSQL=YES
> - -DVTK_MODULE_ENABLE_VTK_FiltersExtraction=YES
> - -DVTK_MODULE_ENABLE_VTK_FiltersFlowPaths=YES
> - -DVTK_MODULE_ENABLE_VTK_FiltersGeometry=YES
> - -DVTK_MODULE_ENABLE_VTK_FiltersParallel=YES
> - -DVTK_MODULE_ENABLE_VTK_IOExport=YES
> - -DVTK_MODULE_ENABLE_VTK_IOImage=YES
> - -DVTK_MODULE_ENABLE_VTK_IOLegacy=YES
> - -DVTK_MODULE_ENABLE_VTK_IOPLY=YES
> - -DVTK_MODULE_ENABLE_VTK_IOXML=YES
> - -DVTK_MODULE_ENABLE_VTK_ImagingMath=YES
> - -DVTK_MODULE_ENABLE_VTK_ImagingMorphological=YES
> - -DVTK_MODULE_ENABLE_VTK_ImagingStatistics=YES
> - -DVTK_MODULE_ENABLE_VTK_ImagingStencil=YES
> - -DVTK_MODULE_ENABLE_VTK_InteractionImage=YES
> - -DVTK_MODULE_ENABLE_VTK_RenderingVolumeOpenGL2=YES
> - -DVTK_MODULE_ENABLE_VTK_TestingRendering=YES
> - -DVTK_MODULE_ENABLE_VTK_ViewsQt=YES
> - -DVTK_MODULE_ENABLE_VTK_zlib=YES

And Slicer with this,
>   - -DCMAKE_BUILD_TYPE:STRING=Release
>   - -DSlicer_SUPERBUILD:BOOL=OFF

When building 3D Slicer, I got this error.
> -- Setting C++ standard
> -- Setting C++ standard - C++11
> -- The C compiler identification is GNU 10.2.0
> -- The CXX compiler identification is GNU 10.2.0
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Check for working C compiler: /usr/bin/cc - skipped
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Check for working CXX compiler: /usr/bin/c++ - skipped
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- Checking if --no-as-needed linker flag is required
> -- Checking if --no-as-needed linker flag is required - yes
> -- Check if the system is big endian
> -- Searching 16 bit integer
> -- Looking for sys/types.h
> -- Looking for sys/types.h - found
> -- Looking for stdint.h
> -- Looking for stdint.h - found
> -- Looking for stddef.h
> -- Looking for stddef.h - found
> -- Check size of unsigned short
> -- Check size of unsigned short - done
> -- Searching 16 bit integer - Using unsigned short
> -- Check if the system is big endian - little endian
> -- Could NOT find Subversion (missing: Subversion_SVN_EXECUTABLE) 
> -- Found Git: /usr/bin/git  
> -- Found Patch: /usr/bin/patch  
> -- Configuring Slicer organization domain [www.na-mic.org]
> -- Configuring Slicer organization name [NA-MIC]
> -- Configuring Slicer default home module [Welcome]
> -- Configuring Slicer default favorite modules [Data, Volumes, Models, Transforms, Markups, SegmentEditor]
> -- Configuring Slicer text of disclaimer at startup [Thank you for using %1!<br><br>This software is not intended for clinical use.]
> -- Configuring Slicer release type [Experimental]
> CMake Warning (dev) at CMake/SlicerMacroExtractRepositoryInfo.cmake:94 (message):
>   Skipping repository info extraction: directory [/run/build/Slicer] is not a
>   GIT or SVN checkout
> Call Stack (most recent call first):
>   CMake/SlicerVersion.cmake:55 (SlicerMacroExtractRepositoryInfo)
>   CMakeLists.txt:183 (include)
> This warning is for project developers.  Use -Wno-dev to suppress it.
> 
> -- Configuring Slicer version [4.11.20210226-0000-00-00]
> -- Configuring Slicer revision [0]
> CMake Warning (dev) at CMake/SlicerMacroExtractRepositoryInfo.cmake:94 (message):
>   Skipping repository info extraction: directory [/run/build/Slicer] is not a
>   GIT or SVN checkout
> Call Stack (most recent call first):
>   CMake/SlicerVersion.cmake:99 (SlicerMacroExtractRepositoryInfo)
>   CMakeLists.txt:183 (include)
> This warning is for project developers.  Use -Wno-dev to suppress it.
> 
> -- Checking to see if CXX compiler accepts flag -fdiagnostics-show-option
> -- Checking to see if CXX compiler accepts flag -fdiagnostics-show-option - yes
> -- Checking to see if CXX compiler accepts flag -Wl,--no-undefined
> -- Checking to see if CXX compiler accepts flag -Wl,--no-undefined - yes
> -- Checking to see if CXX compiler accepts flag -fstack-protector-all
> -- Checking to see if CXX compiler accepts flag -fstack-protector-all - yes
> -- Configuring VTK
> --   Slicer_VTK_RENDERING_BACKEND is OpenGL2
> --   Slicer_VTK_SMP_IMPLEMENTATION_TYPE is Sequential
> --   Slicer_VTK_VERSION_MAJOR is 8
> -- Looking for pthread.h
> -- Looking for pthread.h - found
> -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
> -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
> -- Looking for pthread_create in pthreads
> -- Looking for pthread_create in pthreads - not found
> -- Looking for pthread_create in pthread
> -- Looking for pthread_create in pthread - found
> -- Found Threads: TRUE  
> -- Configuring Slicer with Qt 5.15.2 (using modules: Core, Widgets, Multimedia, Network, OpenGL, PrintSupport, UiTools, Xml, XmlPatterns, Svg, Sql, WebEngine, WebEngineWidgets, WebChannel, Script, Test, )
> -- Setting QT_PLUGINS_DIR: /usr/lib/plugins
> -- Setting QT_BINARY_DIR: /usr/lib/bin
> -- 
> -- Forcing Slicer_USE_SYSTEM_QT to ON (Qt5_DIR [/usr/lib/x86_64-linux-gnu/cmake/Qt5] associated with a system location: /usr/lib/)
> -- 
> -- Setting ExternalData_OBJECT_STORES to '/run/build/Slicer/_flatpak_build/ExternalData/Objects'
> -- Configuring Slicer for [linux-amd64]
> -- Using system QT
> -- Found X11: /usr/include   
> -- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so
> -- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so - found
> -- Looking for gethostbyname
> -- Looking for gethostbyname - found
> -- Looking for connect
> -- Looking for connect - found
> -- Looking for remove
> -- Looking for remove - found
> -- Looking for shmat
> -- Looking for shmat - found
> -- Looking for IceConnectionNumber in ICE
> -- Looking for IceConnectionNumber in ICE - found
> -- Found OpenGL: /usr/lib/x86_64-linux-gnu/libOpenGL.so  found components: OpenGL GLX 
> -- Found Python3: /usr/bin/python3.8 (found suitable version "3.8.7", minimum required is "3.8") found components: Interpreter Development.Module Development.Embed 
> -- Found FontConfig: /usr/lib/x86_64-linux-gnu/libfontconfig.so  
> CMake Error at CMakeLists.txt:813 (find_package):
>   Found package configuration file:
> 
>     /app/lib/cmake/vtk-9.0/vtk-config.cmake
> 
>   but it set VTK_FOUND to FALSE so package "VTK" is considered to be NOT
>   FOUND.  Reason given by package:
> 
>   Could not find the VTK package with the following required components:
>   GUISupportQtOpenGL.
> 
> 
> 
> -- Configuring incomplete, errors occurred!

It does not found GUISupportQtOpenGL. I assume that there are patch that were applied to when superbuild is enable to VTK that fix the issue?
For Flatpak, superbuild are difficult to package, since superbuild will try to download packages, it will fail because Flathub does not allow that.

Are there patch file (.patch) that I can use? I can manually patch it.

Relate? GUISupportQtOpenGL is deprecated in VTK9 · Issue #5339 · Slicer/Slicer · GitHub

Thanks
Andy

Thank you for working on this, it will be awesome.

Why don’t you use Slicer’s superbuild to take care of downloading and building all dependencies?

Flathub (Flatpak store) does not allow network connection while building for the package to be reproducible. I can only download all package before hand.

OK, then we need to download git repositories of all projects in advance and then pass their path to Slicer so that it uses those instead of downloading them. It may be enough to just specify Slicer_…_GIT_REPOSITORY CMake variable to point to the local git repository for each project (VTK, ITK, … all 40 projects in the SuperBuild subfolder; and for the few projects listed in SuperBuild.cmake) and build Slicer normally.

I think that might work, will this be in the next 3D Slicer version?

I actually try to Flatpak Telesculptor too, but fail to build because superbuild method that need network. I need to hack around but unsuccessful.

Just successfully Flatpak Paraview, Paraview is easy because it does not use superbuild, pretty much self contain.

I do have graphical glitch in Paraview, this happen to Tomviz as well. Do you havesome clue?
Imgur
left flatpak, right tomviz binary
Imgur

Slicer superbuild does not need network connection if you download the git repositories in advance.

The rendering error is probably an issue in VTK (or maybe due to an error in your graphics driver). Report it to ParaView developers.

Where must I put the pre-download dependencies? Must it be unpack or kept as archive?

I agree with @lassoan. This can be really good for Linux users!

It might make sense to try to break the superbuild for this (Slicer_SUPERBUILD=OFF). I don’t know much about flatpacks, but if they provide a sandboxed environment, maybe the Slicer app launcher could be disabled in favor of the sandboxed environment.

I would not recommend to deviate too much from the standard build and runtime environment, because even if somebody finds the time to implement the changes once, it will be very hard to maintain it.

The main problem of requiring network access during superbuild can be resolved by downloading the git repositories in advance and passing them as build options. This does not require any change in Slicer. I would expect that we’ll need to figure out something for bundled Python packages, too (is it OK to download pre-built wheels?).

It is ok to download prebuild wheels as well.
Could you clarify where must the pre-download deps be? Must it be archive or unpack? What should their archive or folder name be?

To move forward, the following need to be done:

  1. support local download git repository
  2. support download the applauncher
  3. support local download and use of python wheels
  4. update remaining external project to support passing local directory
  5. support local download of remote dependencies download using Slicer_Remote_Add. (see here)
  6. configure Slicer with relevant variables

For (1) and (2), scripts are provided below.

For (3):

For (4), these external files need to be updated to allow passing local file path:

External_python.cmake
External_OpenSSL.cmake
External_PCRE.cmake
External_tbb.cmake
External_Swig.cmake
External_CTKResEdit.cmake  # Only used on windows

For (5), it should be easy to instrument the relevant CMake functions.

(1) download git repositories

Note: this is a partial list, we will likely have to update the build system to be more exhaustive

I suggest the following:

  • Checkout dependencies in local directory (script below should help)
  • Then, configure Slicer passing the following variables:
-DSlicer_zlib_GIT_REPOSITORY=file:///tmp/zlib
-DSlicer_curl_GIT_REPOSITORY=file:///tmp/curl
-DSlicer_CTKAppLauncherLib_GIT_REPOSITORY=file:///tmp/CTKAppLauncherLib
-DSlicer_bzip2_GIT_REPOSITORY=file:///tmp/bzip2
-DSlicer_LZMA_GIT_REPOSITORY=file:///tmp/LZMA
-DSlicer_sqlite_GIT_REPOSITORY=file:///tmp/sqlite
-DSlicer_python_GIT_REPOSITORY=file:///tmp/python
-DSlicer_VTK_GIT_REPOSITORY=file:///tmp/VTK
-DSlicer_teem_GIT_REPOSITORY=file:///tmp/teem
-DSlicer_DCMTK_GIT_REPOSITORY=file:///tmp/DCMTK
-DSlicer_ITK_GIT_REPOSITORY=file:///tmp/ITK
-DSlicer_CTK_GIT_REPOSITORY=file:///tmp/CTK
-DSlicer_LibArchive_GIT_REPOSITORY=file:///tmp/LibArchive
-DSlicer_RapidJSON_GIT_REPOSITORY=file:///tmp/RapidJSON
-DSlicer_SimpleITK_GIT_REPOSITORY=file:///tmp/SimpleITK
-DSlicer_JsonCpp_GIT_REPOSITORY=file:///tmp/JsonCpp
-DSlicer_ParameterSerializer_GIT_REPOSITORY=file:///tmp/ParameterSerializer
-DSlicer_SlicerExecutionModel_GIT_REPOSITORY=file:///tmp/SlicerExecutionModel
-DSlicer_qRestAPI_GIT_REPOSITORY=file:///tmp/qRestAPI

Here is a short script to help download dependencies:

#!/bin/bash

set -eo pipefail

# See https://discourse.slicer.org/t/interest-to-create-flatpak-for-3d-slicer-have-issue-with-guisupportqtopengl-not-found/16532

PROG=$(basename $0)

SLICER_DEPENDENCIES_DIR=/tmp

#
# Dependencies associated with Slicer d234b7ee9 from 2021.03.17
# See https://github.com/Slicer/Slicer/commit/d234b7ee9dd6f7ab7ed7d141c08db05f9e20a3d9
#

slicer_git_dependencies_file=${SLICER_DEPENDENCIES_DIR}/slicer_git_dependencies
cat << EOF > ${slicer_git_dependencies_file}
Slicer_zlib_GIT_REPOSITORY=git://github.com/commontk/zlib.git
Slicer_zlib_GIT_TAG=66a753054b356da85e1838a081aa94287226823e
Slicer_curl_GIT_REPOSITORY=git://github.com/Slicer/curl.git
Slicer_curl_GIT_TAG=ca5fe8e63df7faea0bfb988ef3fe58f538e6950b
Slicer_CTKAppLauncherLib_GIT_REPOSITORY=git://github.com/commontk/AppLauncher.git
Slicer_CTKAppLauncherLib_GIT_TAG=1367de4c6efde0c11e87835fb7245ea2b05074aa
Slicer_bzip2_GIT_REPOSITORY=git://github.com/commontk/bzip2.git
Slicer_bzip2_GIT_TAG=0e735f23032ececcf52ed49b27928390fff28e50
Slicer_LZMA_GIT_REPOSITORY=git://github.com/Slicer/lib_lzma.git
Slicer_LZMA_GIT_TAG=v5.2.2
Slicer_sqlite_GIT_REPOSITORY=git://github.com/azadkuh/sqlite-amalgamation.git
Slicer_sqlite_GIT_TAG=3.30.1
Slicer_python_GIT_REPOSITORY=git://github.com/python-cmake-buildsystem/python-cmake-buildsystem.git
Slicer_python_GIT_TAG=dca8bee81e29b452560bd969d67e7d08237e23d6
Slicer_VTK_GIT_REPOSITORY=git://github.com/slicer/VTK.git
Slicer_VTK_GIT_TAG=1076d8bc00f12c49af4e35a3e0e58beae6ad361b
Slicer_teem_GIT_REPOSITORY=git://github.com/Slicer/teem
Slicer_teem_GIT_TAG=e4746083c0e1dc0c137124c41eca5d23adf73bfa
Slicer_DCMTK_GIT_REPOSITORY=git://github.com/commontk/DCMTK.git
Slicer_DCMTK_GIT_TAG=patched-DCMTK-3.6.6_20210115
Slicer_ITK_GIT_REPOSITORY=git://github.com/InsightSoftwareConsortium/ITK
Slicer_ITK_GIT_TAG=2b34388159c20c0a6334d9b174fc6c230853988c
Slicer_CTK_GIT_REPOSITORY=git://github.com/commontk/CTK.git
Slicer_CTK_GIT_TAG=9a9573ec4e0653ee96fe02823ac7ee66b40d3b44
Slicer_LibArchive_GIT_REPOSITORY=git://github.com/libarchive/libarchive.git
Slicer_LibArchive_GIT_TAG=34940ef6ea0b21d77cb501d235164ad88f19d40c
Slicer_RapidJSON_GIT_REPOSITORY=git://github.com/miloyip/rapidjson.git
Slicer_RapidJSON_GIT_TAG=v1.1.0
Slicer_SimpleITK_GIT_REPOSITORY=git://github.com/SimpleITK/SimpleITK.git
Slicer_SimpleITK_GIT_TAG=460f9c1553621b649f376bb1c07c94d4bdf6f055
Slicer_JsonCpp_GIT_REPOSITORY=git://github.com/Slicer/jsoncpp.git
Slicer_JsonCpp_GIT_TAG=73b8e172d6615251ef851d883ef02f163e7075b2
Slicer_ParameterSerializer_GIT_REPOSITORY=git://github.com/Slicer/ParameterSerializer.git
Slicer_ParameterSerializer_GIT_TAG=70e95f1cdee52cc49dfc3375e956a8f5958240c7
Slicer_SlicerExecutionModel_GIT_REPOSITORY=git://github.com/Slicer/SlicerExecutionModel.git
Slicer_SlicerExecutionModel_GIT_TAG=f19d6e88a94ba8f31ddafcff4adf185fe90d7e72
Slicer_qRestAPI_GIT_REPOSITORY=git://github.com/commontk/qRestAPI.git
Slicer_qRestAPI_GIT_TAG=ddc0cfcc220d0ccd02b4afdd699d1e780dac3fa3

EOF

#
# List of variable obtained by locally modifying "ExternalProject_SetIfNotDefined"
#
# Patch:
#
# diff --git a/CMake/ExternalProjectDependency.cmake b/CMake/ExternalProjectDependency.cmake
# index 0f775d3ff0..c9a77856de 100644
# --- a/CMake/ExternalProjectDependency.cmake
# +++ b/CMake/ExternalProjectDependency.cmake
# @@ -1055,6 +1055,7 @@ macro(ExternalProject_SetIfNotDefined var defaultvalue)
#      endif()
#      set(${var} "${defaultvalue}")
#    endif()
# +  message(STATUS "${var}=${${var}}")
#  endmacro()
# 
#  #.rst:

# Collect project names
for entry in $(cat "${slicer_git_dependencies_file}" | grep "GIT_REPOSITORY"); do
  proj=$(echo ${entry} | cut -d= -f1 | cut -d_ -f2)
  repo=$(echo ${entry} | cut -d= -f2)
  sha=$(cat "${slicer_git_dependencies_file}" | grep ${proj}_GIT_TAG | cut -d= -f2)
  if [[ ! -d "${SLICER_DEPENDENCIES_DIR}/${proj}" ]]; then
    git clone ${repo} ${proj}
  fi
  (cd ${proj}; git fetch --tags origin; git reset --hard HEAD; git checkout ${sha})
  break
done

# List of options to configure Slicer
for entry in $(cat "${slicer_git_dependencies_file}" | grep "GIT_REPOSITORY"); do
  varname=$(echo ${entry} | cut -d= -f1)
  proj=$(echo ${entry} | cut -d= -f1 | cut -d_ -f2)
  local_path=${SLICER_DEPENDENCIES_DIR}/${proj}
  echo "-D${varname}=file://${local_path}"
done

(2) download launcher

Here are the steps:

  1. download launcher using script below
  2. configure Slicer passing option CTKAppLauncher_DIR

Option to pass (as of Slicer d234b7ee9 from 2021.03.17):

-DCTKAppLauncher_DIR:PATH=/tmp/CTKAppLauncher-0.1.29-linux-amd64

Script:


SLICER_DEPENDENCIES_DIR=/tmp

# Dependencies associated with Slicer d234b7ee9 from 2021.03.17
# See https://github.com/Slicer/Slicer/commit/d234b7ee9dd6f7ab7ed7d141c08db05f9e20a3d9

launcher_version=0.1.29

name=CTKAppLauncher-${launcher_version}-linux-amd64
curl -L# https://github.com/commontk/AppLauncher/releases/download/v${launcher_version}/${name}.tar.gz -o  ${name}.tar.gz
tar -xzvf ${name}.tar.gz

echo "-DCTKAppLauncher_DIR:PATH=${SLICER_DEPENDENCIES_DIR}/${name}"

launcher_version can be discovered looking at External_CTKAPPLAUNCHER.cmake#L24-L46

1 Like

Thanks for the help. It looks good, I have nothing to add, let me know if it ready to be test.

Has there been any progress? Do you need help?

1 Like

There hasn’t been much progress on this as far as I know. There also hasn’t been a large number of users in the community asking for it either. The Slicer community has primarily accepted going to https://download.slicer.org/ to get the latest Stable release and latest Preview builds of Slicer.

If you’re interested in the project, feel free to help out! You will likely have to become the maintainer of this moving forward as well since sometimes things like this get working for one version, but then never maintained for future versions.

Regarding networkless builds I suggest you first read the conversation at Plans for a new release? · Issue #1047 · commontk/CTK · GitHub which has some recent thoughts on the topic of Slicer using custom versions of various dependencies rather than an OS level set of dependencies.

There is also a Slicer GitHub issue detailing the networkless support:

Thank you very much for the information. And it is a problem of modern software to have dependencies in so many projects, especially if there is a different version per linux distribution.

To be able to maintain it, I think I would have to be a direct participant of the Slicer.org project because now the Flatpak platform or at least the flathub.org repository allows only to publish the packaged application for the authors of the original software.

Nevertheless, I think it would be an important step forward to continue with the task of transporting it to a framework such as flatpak to facilitate access to different users for this application.

1 Like

While having a stable flatpak makes sense, there is no point for preview versions, which change nightly. The other issue, is it even possible to install Slicer in a single read/only location (e.g., /usr/local/Slicer) and have the user files (extensions, pip packages) etc go in a local folder?

I would like to see this happening and I definitely could contribute to this. I’m not acquainted with Flatkpaks, but in general terms, there are two issues that complicate the integration of 3D Slicer with package systems:

  • One of the issues (as pointed out earlier in the thread and in the reference provided by @jamesobutler) is the superbuild approach, as many packaging systems require network isolation. In the SystoleOS project, we have the same issue and have been advancing on building 3D Slicer without the superbuild (so far, Slicer without python support is available). I hope for further advances during PW38.

  • Another issue is the divergence of some of the underlying dependencies from the official projects (most notably VTK, and to a minor degree, ITK). This makes difficult the co-existence of Slicer and other projects without some isolation mechanisms.

It may be possible to hack our way through (i.e., repo injection, etc) to make a Flatpack with the superbuild (@jcfr posted some snippets earlier in this thread). In my view, it would be better to invest some effort to make 3D Slicer possible to build without the superbuild as this will improve the overall quality of the 3D Slicer project and facilitate other packaging approaches beyond Flatpak and GNU/Linux systems.

There is not much progress since last time because Slicer relies on super build. I think when Slicer can build without network, it would be much easier to build and maintain Slicer flatpak.

Hi Andy,
I would like to be able to package Slicer inside with flatpak. Could you share the progress you made or comment on how far you got to go from there?
Thanks a lot