For a Slicer CAT (Custom application) that specifies additional remote extension in the main CMakeLists.txt and these remote modules are superbuild type extensions, how should I specify appropriate build dependencies for these extensions?
Currently some of the remote extensions I’ve been playing around with require VTK, ITK, and Python projects from the main Slicer core to be built first. This is not a problem when building the extension after the main Slicer core build finishes because all of these dependencies are already complete. However when building these superbuild remote extensions as part of a Slicer CAT with the /MP (multiple processes) flag on, these superbuild projects of these remote extensions are attempted and the build fails because it has not yet completed the Slicer core projects of VTK, ITK, and Python. Using the /MP flag is helpful on Windows and works well with Slicer core to reduce the total build time so I would like to continue to use this build optimization.
One specific example of a remote superbuild extension is SlicerOpenIGTLink. It has superbuild projects such as OpenIGTLinkIO which depends on VTK and YASM which depends on Python. I have observed YASM starting too early in the build process and then failing.
Another example is remote superbuild extension SlicerOpenCV. It has a superbuild project of ITKVideoBridgeOpenCV which depends on the main ITK project. I have observed ITKVideoBridgeOpenCV starting too early in the build process and then failing.
I have done this for various projects, but always in a hack/workaround fashion by adding the dependency info myself.
In general what is needed is that the External_Project files in the extension should be able to detect when they are being included in an application superbuild, and then add their depedencies.
I.e, at this line:
ITK etc should be included if we are doing a custom app. As a workaround hack, you can create a fork/branch of the extension that has the additional dependency info, ex:
Then, when bundled it is sometime useful to exclude all the external projects or only consider a subset. This could be done setting variables like: ${extension_name}_EXTERNAL_PROJECT_EXCLUDE_ALL or ${extension_name}_EXTERNAL_PROJECT_DEPENDENCIES
Bundling of “SuperBuild-type” extension
It also sensible to read the following:
Example of use of ${extension_name}_EXTERNAL_PROJECT_EXCLUDE_ALL
Use of the SuperBuildPrerequisites.cmake pattern
More recently, we introduced a new pattern consisting in adding a file called SuperBuildPrerequisites.cmake in the extension, and including it in custom application at configuration time.
This is useful to avoid duplicating requirements.
SlicerSMTK
_while bundling case has been tested , the standalone build is still a work in progress _
The SlicerMSTK/SuperBuildPrerequisites.cmake file:
Could we put this pattern inside ExternalProject_Include_Dependencies? To me it looks like a bug in ExternalProject_Include_Dependencies that it always enforces existence of External_ITK.cmake even if it is not needed. I think we could solve the issues by adding a check in ExternalProject_Include_Dependencies that would prevent the function from looking for External_ITK.cmake if ITK_FOUND is already set, similarly how it is done here:
I guess by introducing a new SuperBuildPrerequisites.cmake file we could do more, but things are already so complex that we should really try to avoid adding a new mechanism to make things any more powerful, nicer, and generic and instead we should striving to make this simpler, less flexible, less configurable if at all possible.
If we want to make the Windows build faster I would not bother with the mere 20-30% speedup that /MP can do, because it rarely, if ever makes a difference - build takes hours either way. However, if we can reach the 30-minute full build from scratch (as on Linux) then it would worth investing time into developing and supporting that build option. Making Windows builds significantly faster using Ninja came up a few years ago, and interestingly nested external project builds caused the problems with that, too. If making things more complicated is inevitable then at least we should check if we can get other benefits out of it, such as faster builds.
Yes, it seems that the same check is needed in every superbuild-type extensions.
@jcfr is there a specific reason that you recommended changing every External_*.cmake in every superbuild-type extension instead of fixing SuperBuildPrerequisites.cmake?
Add a “virtual” project called Slicer Dependencies that would depend on all Slicer external project and ensure external projects from bundled SuperBuild extensions systematically depend on it.
Update ExternalProjectDependency.cmake to support customizing dependency of any external project.
We will move forward with approach (2)
ExternalProject_Add_Dependencies
The good news is that the ExternalProjectDependency API already provide a function to do so:
This means that for now we have the flexibility of either updating extension or customizing the external project dependency graph when bundling extension in custom application.
Thank you @jcfr for working on this. Just one question:
If there is a project has dependencies that are provided in the Slicer core (ITK, VTK, DCMTK, etc.) then we need to add them using ExternalProject_Add_Dependencies(${proj} DEPENDS ...); and add other libraries (zmq, pybind11, xeus, etc.) using SET(${proj}_DEPENDS ...)?
Adding lines like the following appear to work to avoid having to update my custom application to now use the latest versions of extensions with my recently pushed updates.
The ITKVideoBridgeOpenCV project is still failing for me because of a numpy location check that is in the initial configuration. I observed the CMake message numpy package not found in python distribution. OpenCV python bindings won't be generated. at time of configuring my custom application. Since this check for numpy location is done at time of configure rather than at time of the project beginning after ITK/python/python-numpy projects have finished, then it fails to appropriately build OpenCV as ITKVideoBridgeOpenCV needs.
46>CMake Warning at C:/SApp/OpenCV-install/OpenCVConfig.cmake:166 (message):
46> Found OpenCV Windows Pack but it has no binaries compatible with your
46> configuration.
46>
46> You should manually point CMake variable OpenCV_DIR to your build of OpenCV
46> library.
46>Call Stack (most recent call first):
46> itk-module-init.cmake:4 (find_package)
46> CMakeLists.txt:14 (include)
46>
46>
46>CMake Error at itk-module-init.cmake:4 (find_package):
46> Found package configuration file:
46>
46> C:/SApp/OpenCV-install/OpenCVConfig.cmake
46>
46> but it set OpenCV_FOUND to FALSE so package "OpenCV" is considered to be
46> NOT FOUND.
46>Call Stack (most recent call first):
46> CMakeLists.txt:14 (include)
46>
46>
46>-- Configuring incomplete, errors occurred!
46>See also "C:/SApp/ITKVideoBridgeOpenCV-build/CMakeFiles/CMakeOutput.log".
46>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-mkdir.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-download.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-update.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-patch.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-configure.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-build.rule;C:\SApp\CMakeFiles\320ab9882e45cf6a908302c3985d3202\ITKVideoBridgeOpenCV-install.rule;C:\SApp\CMakeFiles\d0f4a9b23d08a9ea69be4fa4694697b0\ITKVideoBridgeOpenCV-complete.rule;C:\SApp\CMakeFiles\daf1effa4b5847d53d90fb6a3d2334b3\ITKVideoBridgeOpenCV.rule;C:\SApp\slicersources-src\CMakeLists.txt' exited with code 1.
46>Done building project "ITKVideoBridgeOpenCV.vcxproj" -- FAILED.
In a nutshell, the introspection leading to the error you experienced should be done at build time. And it turns out this was already the case. An additional OpenCV patch was needed to make sure no attempt at finding Python2 was done (see ).
To illustrate the following should allows to bundle SlicerOpenCV:
The example below currently references a fork because the pull request referenced above has not yet been integrated. Please, do not implement any production code relying a personal fork.
# SlicerOpenCV
set(extension_name "SlicerOpenCV")
set(${extension_name}_SOURCE_DIR "${CMAKE_BINARY_DIR}/${extension_name}")
FetchContent_Populate(${extension_name}
SOURCE_DIR ${${extension_name}_SOURCE_DIR}
GIT_REPOSITORY ${EP_GIT_PROTOCOL}://github.com/jcfr/SlicerOpenCV.git
GIT_TAG e2a86986482ba0da4f7b5ca2abe8e158a0f2a991
GIT_PROGRESS 1
QUIET
)
list(APPEND Slicer_EXTENSION_SOURCE_DIRS ${${extension_name}_SOURCE_DIR})
# SlicerOpenCV: Add OpenCV dependencies specific to Slicer
ExternalProject_Add_Dependencies(OpenCV
DEPENDS
python-numpy
)
# SlicerOpenCV: Overide list of dependencies to avoid attempt to build ITKVideoBridgeOpenCV
set(${extension_name}_EXTERNAL_PROJECT_DEPENDENCIES
OpenCV
)
# SlicerOpenCV: Enable ITKVideoBridgeOpenCV module
set(Module_ITKVideoBridgeOpenCV ON)
mark_as_superbuild(
VARS
Module_ITKVideoBridgeOpenCV:BOOL
OpenCV_DIR:PATH # Set in External_OpenCV.cmake
PROJECTS
ITK
)
ExternalProject_Add_Dependencies(ITK
DEPENDS
OpenCV
)
, the standalone build is still a work in progress 
_
The example below currently references a fork because the pull request referenced above has not yet been integrated. Please, do not implement any production code relying a personal fork.