Nightly build TESTS failing for extensions with dependencies to other scriptable modules

Update: Same thing after applying the fix

Is there a way to simulate the build server locally on my system, so that I can just test that locally? Making some tiny changes and waiting 24hours to it failing again doesn’t sound very productive :wink:

1 Like

Yes. It’s very simple to set up extension build on your computer. See instructions here: https://www.slicer.org/wiki/Documentation/Nightly/Developers/Build_ExtensionsIndex

Ok same thing is happening when I run the extension build locally. (see http://slicer.cdash.org/testDetails.php?test=8443330&build=1135723)

I managed to run the tests with multiple dependencies by adding the dependencies to the tests, see


and

If the Xyz_DIR variables are correctly set (you can check it on your local factory build), then the tests should run if you add these too.

This error

/Users/christian/sources/cpp/Builds/Slicer/Release/Slicer-build/lib/Slicer-4.9/qt-loadable-modules/./libvtkSlicerMarkupsModuleMRMLDisplayableManagerPythonD.dylib: malformed mach-o: load commands size (32840) > 32768

seems to be due to the limit on total size of all the load commands in the dynamic library in the MacOS loader.

I guess using a shorter build directory path would solve the issue. For example, instead of /Users/christian/sources/cpp/Builds/Slicer/Release, you could try /Users/christian/D/S4R.

@lassoan so does this mean that the way things work, developers cannot have python modules that are independent from Slicer? Python modules must have Slicer widget/logic in order to establish module class instantiation?

By “module” I mean Slicer scripted module. You can use any Python packages that are independent from Slicer, in any way you want. You just have to place them in a subdirectory of any Slicer module directory so that Python can find it.

I think in the case of @che85, the issue comes from the dependency on a Slicer extension, SlicerDevelopmentToolbox, that contains a set of python packages to reuse from multiple Slicer modules, and it is not loaded when it is needed. It sounds like we would need to add “fake” widget/logic to the classes from that extension to make it loaded in order. Christian, please correct me if I am wrong.

It doesn’t need to be an actual module, the only thing that needs to be ensured is that the python files are deployed to the proper location, which is recognized by the extension. An example can be DICOMLib or Modules/Loadable/Segmentations/EditorEffects/Python.

If the files are deployed to the right place, the dependencies need to be added (at least this is how I managed to get it to work). See the git commits I referenced above.

@cpinter It would be great if that works. In my opinion Slicer should set those in a generic manner instead of having those attributes set within the extensions CMakeLists. I think otherwise it would be very error prone and a lot of overhead.

No idea if that can be done in Slicer itself.

1 Like

Hi,

You can build the project Slicer/extensions/Cmake.

Look for “build extensionindex” on the devel FAQ.

Hth
Jc

@jcfr That’s what I did and those are the results: http://slicer.cdash.org/testDetails.php?test=8443330&build=11357232

@che85 you can confirm this with otool -l LIBRARYNAME.

There is a hack we could use, essentially to daisy-chain dependencies, but at that level of effort we’re better off reducing the number of shared libraries.

(last time I looked at the slow startup issue on macOS – especially debug build – I think there were over 600 shared libraries loaded, and majority of them were ours. IIRC there were something like 30 million strcmps during startup while sorting symbols).

FWIW, a patch is being integrated into VTK to get ride of the PythonD
library and remove the number of libraries.

We could backport it.

Jc

1 Like

@ihnorton Here is the output of otool

Update: Those path additions that enabled the tests to run locally didn’t do the trick on the factory.
http://slicer.cdash.org/testDetails.php?test=8445508&build=1136268
The only reason I can see is that the *_DIR paths are not correct. I’ll check on our factory machine.

Hi guys,

are there any updates/ideas for solving this issue?

Considering the switch to Qt5 and other dramatic changes in the core, it is unfortunate that we don’t have a reliable way to test extensions to identify regressions associated with that process.

To clarify, extensions are still build and tested nightly. It means you will be able to look at the build report every day.