How can I subscribe to monitor individual extensions on Slicer dashboard?

@jcfr mentioned

The good news is that CDash support email notification and now support specifying more than one repository.

Can someone help me figure out how to do this? My guess was that this should be configured in project subscription, but I don’t see how this can be done. There are no labels to select from.

image

Hi @fedorov ,

When using ctest and CDash to build a single project, this work our of the box.

The challenge here is that we conveniently centralize the configure/build/test of more than hundred of projects.

You will find below comments and idea I came up with while looking at the current system

in a nutshell

… the extension build system is doing the following:

  1. Discover and parse all extension description files
  2. Sort extension topologically based on dependency information
  3. Add one external project per extension with associated dependency information (this allow to support parallel build): This is used to download the source code and build the project.

The external project is added here.

way forward

The idea is to have for every extension build, repository information send along the configure/build/test restults. This should allow to associate together new configure/build/test error or warning with a given revision of the code.

To support this, I think we need to:

  1. keep around the extension source directory between successive builds
  2. ensure that the command ctest_update is called.

Do we need to keep extension source checkout around between build ?

Say differently, how does CTest/CDash find out the diff associated with two consecutive configure/build/test ?

Note 1: the diff is important to know as it allows to find out the list of authors who potentially introduced warnings/errors/failures/… and later notify them.

Note 2: the extension sources are checked out cleanly every night, we are currently not keep source trees around. The only source tree we currently keep around it the one associated with the ExtensionsIndex

The answer to this question will help us move forward and decide what are the steps.

case 1: if the answer is no

This means that CDash can compute the diff on its his own by only using the build name and repository name.

In that case, it may be as simple as adding the ctest_update call between these two calls:

intuition: I doubt this is the case, I will confirm this with the CDash development team.

case 2: if the answer is yes

This means that the diff is generated locally by using the call to ctest_update

In that case, I think more changes would be involved:

We need to:

  1. update the system to keep extension source checkouts outside of the extension index build tree
  2. use ctest_update to update the checkout instead of teaching ExternalProject how to do it. The code used to setup the download command is here

Setting either CTEST_GIT_UPDATE_OPTIONS or CTEST_GIT_UPDATE_CUSTOM should help checking a specific branch or commit.

i also suggest we transition all remaining extensions from svn to git, and only support git. That would help simplify the system.

1 Like

@jcfr thank you for this thorough analysis. But, in a nutshell, this basically means we cannot monitor status of the individual extensions at the moment.

I am fully aware about the extremely limited resources, and unfortunately I have no resources to contribute to solving this issue. So I guess we will wait for better times to revisit this…

I will follow up shortly, I have meeting with the CDash core developer in few minutes.

1 Like

So, as I was thinking about this, conceptually at least, this looks like a very manageable problem to solve (“this” == automatic notification about failing extensions based on the dashboard status).

A while ago, when I ran into this, I wrote this script to parse the dashboard HTML. I would not be surprised if this is broken by now, but I am also confident this can be done for the current incarnation of the the dashboard.

If we forget about the peculiarities of CDash/current system, isn’t it as simple as having a cron job running that kind of script and sending emails? Am I missing something?

What if we have the following setup:

  1. upon successful completion of all builds on the dashboard, some webhook is called to initiate a CI build (I think this webhook is the only piece missing).
  2. CI script downloads the dashboard HTML page, parses it, and figures out the status of all extensions.
  3. For projects hosted on GitHub we can automatically get the list of contributors, and send them automatic notifications if dashboard problems with their extension are identified.
  4. To keep things simple, we could hard-code the list of extensions for which we would send notifications to not bother with the subscribe/unsubscribe functionality.

Instead of parsing html, you could use the API.

e.g.: http://slicer.cdash.org/api/v1/index.php?project=SlicerPreview&filtercombine=or&filtercount=1&showfilters=1&field1=label&compare1=63&value1=radiomics (for corresponding html page)

CDash almost does everything that we need and would offer more then the proposed custom scraping/query-based solution (can take into account repository modifications, has convenient web GUI for notification configuration, etc).

Setting up a notification system once may be less work than improving CDash, but in the long term it would probably cost more time to continuously maintain and support a custom system.

Indeed, but in absence of a perfect solution, there is no notifications mechanism for years now, and it happens again and again (at least for me!) that extensions get broken and that remains unnoticed.

The good part is that the solution I proposed, in its most simple implementation using a locally running cron job, has 0 dependencies and is easy to implement with minimum resources (unless I am missing something, which may well be). I will update this thread when I get around to do it.

1 Like

Yes, JC mentioned that in the past as well, but I had troubles locating documentation. I agree, I should search again.