How to generate a test coverage report for scripted module?

Development
1

The title says it all.

Is there a way to generate a test coverage report based on the contents of the ScriptedLoadableModuleTest class contents in a scripted module?

1 Like
2

I’m not aware that anyone has tried that, but if I’m not mistaken the scripted module tests are simply pytest invoked from cmake. So if that’s right you could probably use the pytest coverage tools. If you come up with a recipe be sure to share it back.

1 Like
3

I have tried that, it doesn’t work. See https://gitlab.com/opendose/opendose3d/-/blob/develop/.gitlab-ci.yml

The relevant pytest lines are commented out because they prevent the module to be built inside CDASH

1 Like
4

Thanks for the info. I’ll look into what coverage package can provide and if there can be an easy hack to make it work

5

Thanks everyone🖖

Yesterday at the developer call @jcfr steered me towards looking into the API of the coverage package and trying to run execution through the Slicer environment.

The coverage package is pip_installable into Slicer and has a pretty simple API.
I came up with this python script that can generate selfTests coverage report for a scriptable module:

import os
from coverage import Coverage

# Setup test execution recording. An sqlite database file will be created here
cov = Coverage(data_file=PATH_TO_COVERAGE_DB)

# Record what lines of code are executed during self test run
cov.start()
slicer.selfTests[MY_MODULE_NAME]()
cov.stop()

# List python files that should be mentioned in the report
extensionPythonFiles = []
for root, dirNames, fileNames in os.walk(PATH_TO_MY_MODULE_FOLDER):
    fileNames = [f for f in fileNames if f.endswith('.py') and not f.startswith('__')]
    for f in fileNames:
        extensionPythonFiles.append(os.path.join(root, f))

# Generate report to a text file
with open(PATH_TO_REPORT_FILE, 'w') as rf:
    cov.report(morfs=extensionPythonFiles, file=rf, show_missing=True)
1 Like
6

I think this doesn’t solve the problem. Although it generates the report inside slicer itself, it does not generate the report on build. The build is made using cmake and the tests are done through ctest. This last procedure is not generating reports.

7

I might have not described my use case. At the moment I work only with scriptable modules that I run against the latest slicer so I am not doing builds on the CI. I wanted to have a look at my coverage on the python side and I didn’t look at how to integrate this into the build process.

1 Like
8

for now I am creating the reports as txt inside slicer, upload them in git pull inside my repo and loading the report after the ctest, UGLY but is the only thing that’s working for me ATM

1 Like