Extension tests fail on CDash, why?

mau_igna_06 · December 4, 2022, 9:40pm

For community interest this issue is duplicated from BoneReconstructionPlanner Github

BRP is working correctly and the tests are succeded on my laptop.

Do you have some instruction on how to solve these tests failure ?

mau_igna_06 · December 4, 2022, 9:41pm

Do you have some instruction on how to solve these test failure ?

According to this test results of the Plots module:
RunTest and all test_* matching functions are run in parallel on test-machine

It appears I should then use only RunTest function and use section_ keyword

mau_igna_06 · December 5, 2022, 3:07pm

CDash is still failing for BoneReconstructionPlanner… Core-devs do you know how to fix this?

jamesobutler · December 5, 2022, 5:31pm

I see the following test failures on the Windows platform for BoneReconstructionPlanner (SlicerPreview : Test Details). Are you referring to these test failures?

======================================================================
FAIL: runTest (BoneReconstructionPlanner.BoneReconstructionPlannerTest)
Run as few or as many tests as needed here.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/D/P/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/qt-scripted-modules/BoneReconstructionPlanner.py", line 3466, in runTest
    self.section_LoadFinishedPlanSampleData()
  File "D:/D/P/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/qt-scripted-modules/BoneReconstructionPlanner.py", line 3528, in section_LoadFinishedPlanSampleData
    self.assertEqual(
AssertionError: 6 != 2

----------------------------------------------------------------------
Ran 2 tests in 87.301s

FAILED (failures=1)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\D\P\S-0-build\Slicer-build\bin\Python\slicer\testing.py", line 26, in runUnitTest
    exitFailure()
  File "D:\D\P\S-0-build\Slicer-build\bin\Python\slicer\testing.py", line 10, in exitFailure
    raise Exception(message)
Exception
Switch to module:  ""
Warning: In D:\D\P\S-0\Libs\MRML\Core\vtkMRMLSubjectHierarchyNode.cxx, line 3009
vtkMRMLSubjectHierarchyNode (0000026324DF87A0): GetItemByName: Multiple subject hierarchy item found with name 'Mandibular planes. Returning first


double __cdecl qSlicerSubjectHierarchyMarkupsPlugin::canOwnSubjectHierarchyItem(__int64) const : could not get the Markups module logic.
void __cdecl qMRMLSubjectHierarchyModel::updateItemDataFromSubjectHierarchyItem(class QStandardItem *,__int64,int) : Terminologies module is not found
double __cdecl qSlicerSubjectHierarchyMarkupsPlugin::canOwnSubjectHierarchyItem(__int64) const : could not get the Markups module logic.
void __cdecl qMRMLSubjectHierarchyModel::updateItemDataFromSubjectHierarchyItem(class QStandardItem *,__int64,int) : Terminologies module is not found
Traceback (most recent call last):
  File "D:/D/P/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/qt-scripted-modules/BoneReconstructionPlanner.py", line 356, in exit
    mandibularPlanesList[i].RemoveObserver(self.logic.mandiblePlaneObserversAndNodeIDList[i][0])
IndexError: list index out of range

mau_igna_06 · December 5, 2022, 6:09pm

Yes. Thank you, I forgot to put the link to the failing-test…

Here is the relevant code:

github.com

SlicerIGT/SlicerBoneReconstructionPlanner/blob/830a793367b28f156b9a27947be5bcd6bf3b13dc/BoneReconstructionPlanner/BoneReconstructionPlanner.py#L3459-L3476


      
          def runTest(self):

            """Run as few or as many tests as needed here.

            """

            self.setUp()

            self.section_EnterBRP()

            self.section_GetWidget()

            self.section_GetLogic()

            self.section_LoadFinishedPlanSampleData()

          

            self.setUp()

            self.section_EnterBRP()

            self.section_GetWidget()

            self.section_GetLogic()

            self.section_LoadSampleData()

            self.section_MakeModels()

            self.section_AddMandibularCurve()

            self.section_AddMandiblePlanes()

            self.section_AddFibulaLineAndCenterIt()

I would like runTest to be executed only once and not in parallel but sequentially

mau_igna_06 · December 6, 2022, 10:47pm

This is my research on the topic

Next is the command line to execute BRP tests on Slicer:

/work/Preview/Slicer-0-build/Slicer-build/Slicer "--no-splash" "--testing" "--launcher-additional-settings" "/work/Preview/S-0-E-b/BoneReconstructionPlanner-build/AdditionalLauncherSettings.ini" "--additional-module-paths" "/work/Preview/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/qt-scripted-modules" "/work/Preview/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/cli-modules" "/work/Preview/S-0-E-b/BoneReconstructionPlanner-build/lib/Slicer-5.3/qt-loadable-modules" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['/work/Preview/S-0-E-b/BoneReconstructionPlanner-build/BoneReconstructionPlanner', '/work/Preview/S-0-E-b/BoneReconstructionPlanner/BoneReconstructionPlanner'], 'BoneReconstructionPlanner')"

Which comes from this cmake file:

github.com

Slicer/Slicer/blob/a5f75351073ef62fd6198d9480d86c0009d70f9b/CMake/SlicerMacroPythonTesting.cmake#L50-L71


      
          macro(slicer_add_python_unittest)
            set(options)
            set(oneValueArgs TESTNAME_PREFIX SCRIPT)
            set(multiValueArgs SLICER_ARGS)
            cmake_parse_arguments(MY "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
            get_filename_component(test_name ${MY_SCRIPT} NAME_WE)
            get_filename_component(_script_source_dir ${MY_SCRIPT} PATH)
            if("${_script_source_dir}" STREQUAL "")
              set(_script_source_dir ${CMAKE_CURRENT_SOURCE_DIR})
            endif()
            ExternalData_add_test(${Slicer_ExternalData_DATA_MANAGEMENT_TARGET}
              NAME py_${MY_TESTNAME_PREFIX}${test_name}
              COMMAND ${Slicer_LAUNCHER_EXECUTABLE}
              --no-splash
              --testing
              ${Slicer_ADDITIONAL_LAUNCHER_SETTINGS}
              ${MY_SLICER_ARGS}
              ${UNITTEST_LIB_PATHS}
              --python-code "import slicer.testing\\; slicer.testing.runUnitTest(['${CMAKE_CURRENT_BINARY_DIR}', '${_script_source_dir}'], '${test_name}')"
              )

This file has been truncated. show original

Here is the runUnitTest definition:

github.com

Slicer/Slicer/blob/a5f75351073ef62fd6198d9480d86c0009d70f9b/Base/Python/slicer/testing.py#L13-L26


      
          def runUnitTest(path, testname):
              import sys
              import unittest
              if isinstance(path, str):
                  sys.path.append(path)
              else:
                  sys.path.extend(path)
              print("-------------------------------------------")
              print(f"path: {path}\ntestname: {testname}")
              print("-------------------------------------------")
              suite = unittest.TestLoader().loadTestsFromName(testname)
              result = unittest.TextTestRunner(verbosity=2).run(suite)
              if not result.wasSuccessful():
                  exitFailure()

If I run it on Slicer I get:

import unittest
suite = unittest.TestLoader().loadTestsFromName('BoneReconstructionPlanner')
print(suite)

<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<BoneReconstructionPlanner.BoneReconstructionPlannerTest testMethod=runTest>]>, <unittest.suite.TestSuite tests=[<slicer.ScriptedLoadableModule.ScriptedLoadableModuleTest testMethod=runTest>]>]>

There are two tests listed above, so that explains the logs from CDash:

runTest (BoneReconstructionPlanner.BoneReconstructionPlannerTest)
…
Run as few or as many tests as needed here. … FAIL
runTest (slicer.ScriptedLoadableModule.ScriptedLoadableModuleTest)
Run a default selection of tests here. … ok

The code sources are linked in the above quote.

This code piece also looks relevant:

github.com

Slicer/Slicer/blob/8a1c2e03931ab06ca44c7a61fc3f15766c2b4ac1/Base/Python/slicer/ScriptedLoadableModule.py#L43-L50


      
          # Add this test to the SelfTest module's list for discovery when the module
          # is created.  Since this module may be discovered before SelfTests itself,
          # create the list if it doesn't already exist.
          try:
              slicer.selfTests
          except AttributeError:
              slicer.selfTests = {}
          slicer.selfTests[self.moduleName] = self.runTest

The main question I have right now is why my test data is loaded 3 times instead of 1 (as I intended on the test here and this line)

First load:

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_092933_484/TestPlanBRP/Data/Fibula.nrrd. Dimensions: 512x512x1213. Number of components: 1. Pixel type: int.

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_092933_484/TestPlanBRP/Data/Skull.nrrd. Dimensions: 512x512x207. Number of components: 1. Pixel type: int.

Second load:

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_092959_291/TestPlanBRP/Data/Fibula.nrrd. Dimensions: 512x512x1213. Number of components: 1. Pixel type: int.

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_092959_291/TestPlanBRP/Data/Skull.nrrd. Dimensions: 512x512x207. Number of components: 1. Pixel type: int.

Third load:

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_093041_333/TestPlanBRP/Data/Fibula.nrrd. Dimensions: 512x512x1213. Number of components: 1. Pixel type: int.

Loaded volume from file: /home/kitware/.cache/NA-MIC/Slicer-tmp/Slicer-tmpIO/__BundleLoadTemp-2022-12-06_093041_333/TestPlanBRP/Data/Skull.nrrd. Dimensions: 512x512x207. Number of components: 1. Pixel type: int.

mau_igna_06 · December 9, 2022, 3:33pm

Dear @lassoan, @pieper, @jamesobutler, @jcfr, @Sam_Horvath

It would be really nice if you could help me solve this test-break

lassoan · December 11, 2022, 3:38pm

You can print the stack trace in the loading method to see what is executing the method:

import traceback
traceback.print_stack()

In this case, loading of the scene is attempted 3 times because an error occurred during loading. SampleData module tries to download and load a data set 3 times before giving up. So, you need to check what causes the scene loading to return with failure and if you fix it then the scene will only be loaded once and probably all the other errors will go away.

If it is too difficult to find/fix the root cause of the scene loading error then you can use a lower-level SampleData API, where you can set number of attempts to 1:

def downloadFromSource(self, source, maximumAttemptsCount=3):

mau_igna_06 · December 11, 2022, 4:55pm

Thanks for the answer Andras

So I guess I’ll add that code to the test after this line so it gets executed by the Kitware-build-machine as the test doesn’t fail on my laptop

lassoan · December 11, 2022, 5:00pm

It fails the same on my computer, too. I’ve simply downloaded the ExtensionsIndex repository, kept only those s4ext files that are relevant for BRP extension, and then built the extensions. It reproduced the same error as you can see on the dashboard. Probably some extension is not fully loaded and that causes the scene load with errors.

@RafaelPalomar had some issues with testing extensions that depend on other extensions - he might have some tips for you.

mau_igna_06 · December 11, 2022, 6:32pm

Thank you Andras. I could replicate the bug by installing BRP on Slicer and then disabling the MarkupsToModel extension it depends on.

These are the Extensions that depend on MarkupsToModel

According to the CMakeLists of the PathReconstruction Extension

The next lines searching the dependencies are missing on BoneReconstructionPlanner CMake file:

find_package(SurfaceWrapSolidify REQUIRED)
find_package(MarkupsToModel REQUIRED)
find_package(Sandbox REQUIRED)
find_package(SlicerRT REQUIRED)

I hope it works, we’ll see tomorrow

Thank you

RafaelPalomar · December 12, 2022, 2:06pm

@mau_igna_06, @lassoan I’ll have a closer look to this and get back to you

RafaelPalomar · December 13, 2022, 3:44pm

@mau_igna_06 we were discussing this topic a few weeks ago in the devs meeting. It seems that tests requiring external module dependencies fail because the dependencies are not loaded in fresh environments. In a development setup, where you manually build the dependencies and set the paths everything should work. However, the testing environment for the extensions tests the modules in an independent manner without considering its dependencies (in the Slicer-Liver, we even have dependencies included in the same extension that are not loaded in some tests, making these fails.

I think this issue should be solved as it will improve the testing dashboards and the quaility of the tests, but solving this problem will require some CMake plumbing work. @jcfr, @pieper do you agree with this assessment? I could open an issue on this and even try to come up with some ideas if that would be helpful. Unfortunately, today I won’t be able to join the meeteing, but we can have this as a discussion point in upcoming meetings.

pieper · December 13, 2022, 8:09pm

We talked about this on today’s call and yes, I think what you describe was the conclusion. I believe Jc has an idea what needs to be done.

mau_igna_06 · December 20, 2022, 8:52pm

Hi

As I see an effort to improve testing documentation here, I would like to point out that I still cannot solve the issue of the testing data not loaded properly on BoneReconstructionPlanner test. The test data can be loaded properly from the SampleData module GUI

Andras explained the problem below

lassoan · December 20, 2022, 9:22pm

The problem is that loading of the scene requires SicerIGT extension, which is not loaded when the automatic test is used. If SlicerIGT modules are not present then Slicer does not know about some of the node types that SlicerIGT defines, therefore the loading ends with error code, therefore the SampleData module makes repeated attempts to download and load the scene file (3 times in total) until it gives up.

The fix would be to add all module paths from required extensions to additional module paths when running the tests.

Topic		Replies	Views
Cdash build test failing for Windows without error log Development	16	856	March 24, 2018
Test errors in CDash Development extensions-manager , testing	2	430	October 8, 2020
Fixing extension testing failures Development testing	2	165	November 6, 2023
Nightly build TESTS failing for extensions with dependencies to other scriptable modules Development	61	2977	February 16, 2018
SlicerPathReconstruction not being built nightly? Development extensions-manager	2	441	October 23, 2018

Extension tests fail on CDash, why?

Related Topics