The good news is that CDash support email notification and now support specifying more than one repository.
Can someone help me figure out how to do this? My guess was that this should be configured in project subscription, but I don’t see how this can be done. There are no labels to select from.
Hi @fedorov ,
When using ctest and CDash to build a single project, this work our of the box.
The challenge here is that we conveniently centralize the configure/build/test of more than hundred of projects.
You will find below comments and idea I came up with while looking at the current system
… the extension build system is doing the following:
The external project is added here.
The idea is to have for every extension build, repository information send along the configure/build/test restults. This should allow to associate together new configure/build/test error or warning with a given revision of the code.
To support this, I think we need to:
Say differently, how does CTest/CDash find out the diff associated with two consecutive configure/build/test ?
Note 1: the diff is important to know as it allows to find out the list of authors who potentially introduced warnings/errors/failures/… and later notify them.
Note 2: the extension sources are checked out cleanly every night, we are currently not keep source trees around. The only source tree we currently keep around it the one associated with the ExtensionsIndex
The answer to this question will help us move forward and decide what are the steps.
This means that CDash can compute the diff on its his own by only using the build name and repository name.
In that case, it may be as simple as adding the ctest_update call between these two calls:
intuition: I doubt this is the case, I will confirm this with the CDash development team.
This means that the diff is generated locally by using the call to ctest_update
In that case, I think more changes would be involved:
We need to:
extension source checkouts outside of the extension index build tree
ctest_update to update the checkout instead of teaching ExternalProject how to do it. The code used to setup the download command is here
Setting either CTEST_GIT_UPDATE_OPTIONS or CTEST_GIT_UPDATE_CUSTOM should help checking a specific branch or commit.
i also suggest we transition all remaining extensions from svn to git, and only support git. That would help simplify the system.
@jcfr thank you for this thorough analysis. But, in a nutshell, this basically means we cannot monitor status of the individual extensions at the moment.
I am fully aware about the extremely limited resources, and unfortunately I have no resources to contribute to solving this issue. So I guess we will wait for better times to revisit this…
I will follow up shortly, I have meeting with the CDash core developer in few minutes.
So, as I was thinking about this, conceptually at least, this looks like a very manageable problem to solve (“this” == automatic notification about failing extensions based on the dashboard status).
A while ago, when I ran into this, I wrote this script to parse the dashboard HTML. I would not be surprised if this is broken by now, but I am also confident this can be done for the current incarnation of the the dashboard.
If we forget about the peculiarities of CDash/current system, isn’t it as simple as having a cron job running that kind of script and sending emails? Am I missing something?
What if we have the following setup:
Instead of parsing html, you could use the API.
e.g.: http://slicer.cdash.org/api/v1/index.php?project=SlicerPreview&filtercombine=or&filtercount=1&showfilters=1&field1=label&compare1=63&value1=radiomics (for corresponding html page)
CDash almost does everything that we need and would offer more then the proposed custom scraping/query-based solution (can take into account repository modifications, has convenient web GUI for notification configuration, etc).
Setting up a notification system once may be less work than improving CDash, but in the long term it would probably cost more time to continuously maintain and support a custom system.
Indeed, but in absence of a perfect solution, there is no notifications mechanism for years now, and it happens again and again (at least for me!) that extensions get broken and that remains unnoticed.
The good part is that the solution I proposed, in its most simple implementation using a locally running cron job, has 0 dependencies and is easy to implement with minimum resources (unless I am missing something, which may well be). I will update this thread when I get around to do it.
Yes, JC mentioned that in the past as well, but I had troubles locating documentation. I agree, I should search again.
thanks to @fedorov and @ihnorton, we have now set up simple GitHub actions to check the status of the extension’s build.
Thank you @vkt1414 - this addresses the immediate problem for me! Most definitely, the proper way to address this would be via some kind of a notification or status reporting mechanism from CDash, but clearly it is either not easy, not a priority, or both.
We should go ahead and implement this for all of the extensions we maintain! 
Great. Github actions is turning out to be an excellent tool to do all the chores.
any other extensions to keep track of, than below?
DCMQI
QuantitativeReporting
QIICR SDT
SlicerMRunner (mhub)
mpReview
@vkt1414 can you provide some more documentation about this feature? Specifically where does it post the results and how do people set it up for their own extensions?
Hi @pieper !
Absolutely!
I wrote a simple GitHub actions script that runs a python script that calls the cdash API to get the build info. It extracts the error counts from all categories (configuration, compilation, and test). If the sum of errors is greater than zero, I simply triggered an exit code 1 to fail the GitHub actions, and it should then send a notification.
To adapt this workflow to any extension, copy the .github/ folder to any repo, and then just need to modify the variable ‘slicerExtensionName’ in the get_build_status.py, and that’s it. Happy to submit a pull request if you would like me to submit one for any extension!
SlicerIDCBrowser/.github at main · ImagingDataCommons/SlicerIDCBrowser
We are not posting the results anywhere, the script just prints the counts of errors to the console.
One big picture idea I had was, if we want to implement for all extensions as part of slicer, we can use an action like this Send email · Actions · GitHub Marketplace to send notifications to only developers of the extension, as I believe we can get contributors email addresses from github api. We will need to setup some official slicer email address I guess. Just my two cents.
This is a great idea 
What do you think of generalizing this to all extensions ?
Possible workflow:
ExtensionName [Preview] or ExtensionName [Stable X.Y] created in the Slicer/ExtensionsIndex GitHub project.Some pointers:
ExtensionsIndex associating each .s4ext with relevant developersslicerbot GitHub userRelated actions:
Sounds like a great idea! I have not used the CODEOWNERS file before but the only gripe I have is, that someone will have to maintain the CODEOWNERS file or you make it a requirement, whenever a new extension is added I guess.
I’ll be happy to work on this and report back!
Sounds great @vkt1414 This should be really helpful.
Hi @jcfr, @pieper, and @fedorov,
I’ve worked on this and summarized my progress in the attached figure.
I’ve been conducting my experiments on a fork, without actually tagging anyone (for now). You can view the issues that GitHub actions bot has created/closed here link.
The GitHub Actions workflow I’ve been using can be found here, and the CODEOWNERS file is located here.
I would appreciate any feedback or suggestions for improvement. One area that could use refinement is the CODEOWNERS file. Currently, I check contributors and select the individual who merged the most recent pull request. If more people should be notified, we can edit the file to include them.
This workflow does not interfere if an extension already has a point of contact. However, it will automatically check for new extensions that do not have a CODEOWNER. If you wish to edit it, feel free to do so once I submit the PR. Whether you choose to commit the updated CODEOWNERS back to the repo when an extension lacks a CODEOWNER is entirely up to you. I can add another step to the workflow to accommodate this.
Another thing I would like to know is if there is a github issue for an extension already, I do not update it. Should we? I do not if some extensions are maintained anymore, so we will have a long thread if errors or warnings are not addressed promptly.
Please note that a limitation of this workflow is that non-GitHub repositories cannot be notified.
Looking forward to your feedback.
@vkt1414 I think this is excellent! Having all issues opened under ExtensionsIndex is very good from the user perspective, I think, since it helps with having a single point to check the status.
If we had a vote, I would definitely support adding this to ExtensionIndex - it’s such a huge improvement over what we have right now (which isn’t much!) in terms of helping developers monitor the dashboard, and informing users about the de facto status of the extensions based on nightly tests.
