CircleCI build failures

Development
1

A few weeks ago all our circleCI builds on pull requests (such as this) started to fail because the build is not completed within 60 minutes.

I’ve checked our usage and it seems that we would need to pay at least $100 per month (CircleCI) to cover our needs, and I’m not sure if the maximum build timeout can be adjusted even then. If it can be then we may potentially need to pay more. Another risk of the “pay-as-you-go” pricing of circleCI is that if we mess up something and builds don’t end for many hours and we don’t notice that in time then the charges can go up a lot.

Should we try to speed up the build? For example we could try creating the docker image after Slicer build is completed and do an incremental build instead of a full build for each pull request. This would potentially miss some rare build errors (that only occur when building from scratch), but we would get results much faster, so developers could fix the issues more quickly and circleCI would charge less (or could remain free).

Or, should we switch to GitHub actions?

Or set up runners on our computers? It would be no problem to allocate a few computers in the PerkLab for continuous builds, if there is someone to help me with the setup.

@jcfr @Sam_Horvath @pieper

2

We (VASST, Robarts) can also contribute some runners if that’s the route we take.

3

I’d be happy to contribute runner machines, but in general we’d get a lot more value from breaking up the build into manageable sets of dependencies that could be rebuilt intelligently. I think gh actions could be a better infrastructure for this since we can set up a build matrix. But it will require someone working through all the details. Runners might be an easier short-term solution.

4

We will look into why the build is going over time. For the CircleCI builds we are using a Docker image with the dependencies already built.

We can look at doing an incremental build of Slicer for CircleCI

5

This appears to be the first commit where the timeout failure becomes permanent: BUG: Fix window/level presets in Volumes module · Slicer/Slicer@409c7f0 · GitHub

However, looking at older commits that passed, it seems that the timeout has changed? Older commits are going as much as 2 hrs in some cases and passing. We would need to trim quite a bit off the build to fix that it seems, so an incremental build may be necessary.

6

Yes, it seems that circleCI has changed its policy. I saw that a few weeks ago ITK maintainers disabled circleCI builds, too, for the same reason.

7

This should be addressed by the following pull request:

1 Like
8

It would be great to have a faster initial build after a new Slicer fork / branch.

What does make the S4R or S4D directory so extremely big?
Windows: Could we not just provide the outer build files (DLLs and lib files) as they are and let them only compile new if dependencies change?