Failures due to build order of Slicer CAT with superbuild remote modules

For a Slicer CAT (Custom application) that specifies additional remote extension in the main CMakeLists.txt and these remote modules are superbuild type extensions, how should I specify appropriate build dependencies for these extensions?

Currently some of the remote extensions I’ve been playing around with require VTK, ITK, and Python projects from the main Slicer core to be built first. This is not a problem when building the extension after the main Slicer core build finishes because all of these dependencies are already complete. However when building these superbuild remote extensions as part of a Slicer CAT with the /MP (multiple processes) flag on, these superbuild projects of these remote extensions are attempted and the build fails because it has not yet completed the Slicer core projects of VTK, ITK, and Python. Using the /MP flag is helpful on Windows and works well with Slicer core to reduce the total build time so I would like to continue to use this build optimization.

One specific example of a remote superbuild extension is SlicerOpenIGTLink. It has superbuild projects such as OpenIGTLinkIO which depends on VTK and YASM which depends on Python. I have observed YASM starting too early in the build process and then failing.

Another example is remote superbuild extension SlicerOpenCV. It has a superbuild project of ITKVideoBridgeOpenCV which depends on the main ITK project. I have observed ITKVideoBridgeOpenCV starting too early in the build process and then failing.

1 Like

I have done this for various projects, but always in a hack/workaround fashion by adding the dependency info myself.

In general what is needed is that the External_Project files in the extension should be able to detect when they are being included in an application superbuild, and then add their depedencies.

I.e, at this line:

ITK etc should be included if we are doing a custom app. As a workaround hack, you can create a fork/branch of the extension that has the additional dependency info, ex:

I found the following example which may be the way to update extensions source without having to maintain a fork with the hack.

Also here was my original post on this topic from a couple years back. SlicerOpenIGTLink causing CustomApp build to fail

Exactly, extensions having external project dependencies that are required only when “bundled” should be specified using this pattern:

if(DEFINED Slicer_SOURCE_DIR)
  list(APPEND ${proj}_DEPENDS
    NameOfExternalProject1
    NameOfExternalProject2
    ...
    )
endif()

Then, when bundled it is sometime useful to exclude all the external projects or only consider a subset. This could be done setting variables like:
${extension_name}_EXTERNAL_PROJECT_EXCLUDE_ALL or ${extension_name}_EXTERNAL_PROJECT_DEPENDENCIES

Bundling of “SuperBuild-type” extension

It also sensible to read the following:

Example of use of ${extension_name}_EXTERNAL_PROJECT_EXCLUDE_ALL

Use of the SuperBuildPrerequisites.cmake pattern

More recently, we introduced a new pattern consisting in adding a file called SuperBuildPrerequisites.cmake in the extension, and including it in custom application at configuration time.

This is useful to avoid duplicating requirements.

SlicerSMTK

_while bundling case has been tested :white_check_mark: , the standalone build is still a work in progress :hourglass: _

The SlicerMSTK/SuperBuildPrerequisites.cmake file:

… and how it is integrated in the custom application:

SlicerIMSTK

References:

Worth noting that SuperBuildPrerequisites.cmake defines SlicerIMSTK_EXTERNAL_PROJECT_DEPENDENCIES

Could we put this pattern inside ExternalProject_Include_Dependencies? To me it looks like a bug in ExternalProject_Include_Dependencies that it always enforces existence of External_ITK.cmake even if it is not needed. I think we could solve the issues by adding a check in ExternalProject_Include_Dependencies that would prevent the function from looking for External_ITK.cmake if ITK_FOUND is already set, similarly how it is done here:

I guess by introducing a new SuperBuildPrerequisites.cmake file we could do more, but things are already so complex that we should really try to avoid adding a new mechanism to make things any more powerful, nicer, and generic and instead we should striving to make this simpler, less flexible, less configurable if at all possible.

If we want to make the Windows build faster I would not bother with the mere 20-30% speedup that /MP can do, because it rarely, if ever makes a difference - build takes hours either way. However, if we can reach the 30-minute full build from scratch (as on Linux) then it would worth investing time into developing and supporting that build option. Making Windows builds significantly faster using Ninja came up a few years ago, and interestingly nested external project builds caused the problems with that, too. If making things more complicated is inevitable then at least we should check if we can get other benefits out of it, such as faster builds.

I have had some success upon using the following changes in these PRs. I’m still having issues with SlicerOpenCV in the ITKVideoBridgeOpenCV project.

Yes, it seems that the same check is needed in every superbuild-type extensions.

@jcfr is there a specific reason that you recommended changing every External_*.cmake in every superbuild-type extension instead of fixing SuperBuildPrerequisites.cmake?

Following 2022.02.08 Weekly Meeting discussion, we considered the following options:

  1. Add a “virtual” project called Slicer Dependencies that would depend on all Slicer external project and ensure external projects from bundled SuperBuild extensions systematically depend on it.

  2. Update ExternalProjectDependency.cmake to support customizing dependency of any external project.

We will move forward with approach (2)

ExternalProject_Add_Dependencies

The good news is that the ExternalProjectDependency API already provide a function to do so:

ExternalProject_Add_Dependencies

This means that custom application integrator should be able to address the integration of the issue specfic to SlicerOpenCV by adding:

ExternalProject_Add_Dependencies(ITKVideoBridgeOpenCV
  DEPENDS
    ITK
  )

Path forward

This means that for now we have the flexibility of either updating extension or customizing the external project dependency graph when bundling extension in custom application.

Thank you @jcfr for working on this. Just one question:

If there is a project has dependencies that are provided in the Slicer core (ITK, VTK, DCMTK, etc.) then we need to add them using ExternalProject_Add_Dependencies(${proj} DEPENDS ...); and add other libraries (zmq, pybind11, xeus, etc.) using SET(${proj}_DEPENDS ...)?

Adding lines like the following appear to work to avoid having to update my custom application to now use the latest versions of extensions with my recently pushed updates.

ExternalProject_Add_Dependencies(OpenIGTLink
  DEPENDS
    ITK
  )

The ITKVideoBridgeOpenCV project is still failing for me because of a numpy location check that is in the initial configuration. I observed the CMake message numpy package not found in python distribution. OpenCV python bindings won't be generated. at time of configuring my custom application. Since this check for numpy location is done at time of configure rather than at time of the project beginning after ITK/python/python-numpy projects have finished, then it fails to appropriately build OpenCV as ITKVideoBridgeOpenCV needs.

46>CMake Warning at C:/SApp/OpenCV-install/OpenCVConfig.cmake:166 (message):
46>  Found OpenCV Windows Pack but it has no binaries compatible with your
46>  configuration.
46>
46>  You should manually point CMake variable OpenCV_DIR to your build of OpenCV
46>  library.
46>Call Stack (most recent call first):
46>  itk-module-init.cmake:4 (find_package)
46>  CMakeLists.txt:14 (include)
46>
46>
46>CMake Error at itk-module-init.cmake:4 (find_package):
46>  Found package configuration file:
46>
46>    C:/SApp/OpenCV-install/OpenCVConfig.cmake
46>
46>  but it set OpenCV_FOUND to FALSE so package "OpenCV" is considered to be
46>  NOT FOUND.
46>Call Stack (most recent call first):
46>  CMakeLists.txt:14 (include)
46>
46>
46>-- Configuring incomplete, errors occurred!
46>See also "C:/SApp/ITKVideoBridgeOpenCV-build/CMakeFiles/CMakeOutput.log".
46>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-mkdir.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-download.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-update.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-patch.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-configure.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-build.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-install.rule;C:\SApp\CMakeFiles\d0f4a9b23d08a9ea69be4fa4694697b0\ITKVideoBridgeOpenCV-complete.rule;C:\SApp\CMakeFiles\daf1effa4b5847d53d90fb6a3d2334b3\ITKVideoBridgeOpenCV.rule;C:\SApp\slicersources-src\CMakeLists.txt' exited with code 1.
46>Done building project "ITKVideoBridgeOpenCV.vcxproj" -- FAILED.

SlicerJupyter use-case custom application bundling use-case

There is no need to specify additional set(${proj}_DEPENDS ..)

To make things concrete, bundling SlicerJupyter would only require the following in the custom application top-level CMakeLists.txt:

# SlicerJupyter
set(extension_name "SlicerJupyter")
set(${extension_name}_SOURCE_DIR "${CMAKE_BINARY_DIR}/${extension_name}")
FetchContent_Populate(${extension_name}
  SOURCE_DIR     ${${extension_name}_SOURCE_DIR}
  GIT_REPOSITORY ${EP_GIT_PROTOCOL}://github.com/Slicer/SlicerJupyter.git
  GIT_TAG        b282a2c819da1e3e4fe1ab6c419e2299f976720d
  GIT_PROGRESS   1
  QUIET
  )
list(APPEND Slicer_EXTENSION_SOURCE_DIRS ${${extension_name}_SOURCE_DIR})

ExternalProject_Add_Dependencies(pybind11
  DEPENDS
    python
  )

ExternalProject_Add_Dependencies(python-packages
  DEPENDS
    python
    python-pip
    python-setuptools
  )

The configuration output effectively indicates the dependencies are considered:

$  cmake   -DCMAKE_BUILD_TYPE:STRING=Release   -DQt5_DIR:PATH=$Qt5_DIR  ../SlicerGalaxy
[...]
-- SuperBuild -   python-packages => Requires python[INCLUDED], python-pip[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -   python-packages[OK]
[...]
-- SuperBuild -   pybind11 => Requires python[INCLUDED], 
-- SuperBuild -   pybind11[OK]
[...]

Suggested SlicerJupyter updates

  • There are some hard-coded instance of the string lib/python3.6/site-packages that should be removed. Instead the variable PYTHON_SITE_PACKAGES_SUBDIRshould be used. This pull request should take care of this: https://github.com/Slicer/SlicerJupyter/pull/67

  • Considering python-packages project name is fairly generic and could conflict with other extension install python packages, we should rename it to python-slicerjupyter-requirements. This will be done after the previous pull-request has been integrated. It is tracked by this issue: https://github.com/Slicer/SlicerJupyter/issues/68

  • To provide some more flexibility, we should also look into adding
    Slicer_REQUIRED_PYTHON_VERSION_DOT recently introduced in Slicer@e32fa814f to SlicerConfig.cmake.in. This pull request takes care of this: https://github.com/Slicer/Slicer/pull/6169

SlicerOpenCV custom application bundling use-case

The following pull-request simplify and fix the support for custom application bundling: https://github.com/Slicer/SlicerOpenCV/pull/69

In a nutshell, the introspection leading to the error you experienced should be done at build time. And it turns out this was already the case. An additional OpenCV patch was needed to make sure no attempt at finding Python2 was done (see ).

To illustrate the following should allows to bundle SlicerOpenCV:

:warning: The example below currently references a fork because the pull request referenced above has not yet been integrated. Please, do not implement any production code relying a personal fork.


# SlicerOpenCV
set(extension_name "SlicerOpenCV")
set(${extension_name}_SOURCE_DIR "${CMAKE_BINARY_DIR}/${extension_name}")
FetchContent_Populate(${extension_name}
  SOURCE_DIR     ${${extension_name}_SOURCE_DIR}
  GIT_REPOSITORY ${EP_GIT_PROTOCOL}://github.com/jcfr/SlicerOpenCV.git
  GIT_TAG        e2a86986482ba0da4f7b5ca2abe8e158a0f2a991
  GIT_PROGRESS   1
  QUIET
  )
list(APPEND Slicer_EXTENSION_SOURCE_DIRS ${${extension_name}_SOURCE_DIR})

# SlicerOpenCV: Add OpenCV dependencies specific to Slicer
ExternalProject_Add_Dependencies(OpenCV
  DEPENDS
    python-numpy
  )

# SlicerOpenCV: Overide list of dependencies to avoid attempt to build ITKVideoBridgeOpenCV
set(${extension_name}_EXTERNAL_PROJECT_DEPENDENCIES
  OpenCV
  )

# SlicerOpenCV: Enable ITKVideoBridgeOpenCV module
set(Module_ITKVideoBridgeOpenCV ON)
mark_as_superbuild(
  VARS
    Module_ITKVideoBridgeOpenCV:BOOL
    OpenCV_DIR:PATH # Set in External_OpenCV.cmake
  PROJECTS
    ITK
  )
ExternalProject_Add_Dependencies(ITK
  DEPENDS
    OpenCV
  )

You may also want to specify VTK since OpenCV is built with the option:

ExternalProject_Add_Dependencies(OpenCV
  DEPENDS
    python-numpy
+   VTK
  )