Introduction of "tiers" for Slicer extensions

Development
1

The definition of tiers was revisited during July 16th weekly meeting[1] and this post was updated accordingly.

During our last Slicer developer hangout of March 13th 2024, we discussed the issue that review and approval of new extensions often takes very long time. We identified that the main difficulty was that extensions had to meet many requirements to be admitted to the Extensions Catalog. Reducing the requirements could make extensions approval easier but it could result in having lower quality extensions (harder to use, not well documented or tested, etc.).

To address this problem, we propose organization of Slicer extensions into “tiers”. This helps users understand how mature, well tested, and well supported an extension is and allows extension developers to choose the amount of effort that they are comfortable with investing into the extension.

By default, the Extensions Manager would only show extensions that are in Tier >=3 and users would need to click somewhere in the Extensions Manager to see all extensions. This allows new users to more easily find relevant extensions and less likely to come across experimental extensions that may be hard to use or not well documented.

The proposed tiers are the following:

Tier 5

Critically important extensions, supported by Slicer core developers.

  • Slicer core developers accept the responsibility of fixing any issues caused by Slicer core changes; at least one Slicer core developer (anyone who has commit right to Slicer core) must be granted commit right to the extension’s repository
  • Automated tests for all critical features
  • Maintainers respond to questions related to the extension on the Slicer Forum
  • All requirements of tiers < 5

Tier 3

Community-supported extensions.

  • Documentation, tutorial, and test data are provided for most modules. A tutorial provides step-by-step description of at least the most typical use case, include a few screenshots. Any sample data sets that is used in tutorials must be registered with the Sample Data module to provide easy access to the user.
  • Follows programming and user interface conventions of 3D Slicer (e.g., GUI and logic are separated, usage of popups is minimized, no unnecessary custom GUI styling, etc.)
  • Maintainers respond to issues and pull request submitted to the extension’s repository
  • Maintainers respond to questions directly addressed to him/her via @mention on the Slicer Forum
  • Permissive license is used for the main functions of the extension (recommended Apache or MIT). The extension can provide additional functionality in optional components that are distributed with non-permissive license, but the user has to explicitly approve those before using them (e.g., a pop-up can be displayed that explains the licensing terms and the user has to acknowledge them to proceed).
  • All requirements of tiers < 3

Tier 1

Experimental extensions.

  • Extension has a reasonable name (not too general, not too narrow, suggests what the extension is for)
  • Repository name is Slicer+ExtensionName
  • Repository is associated with 3d-slicer-extension GitHub topic so that it is listed here. To edit topics, click the settings icon in the right side of “About” section header and enter 3d-slicer-extension in “Topics” and click “Save changes”. To learn more about topics, read Classifying your repository with topics - GitHub Docs
  • Extension description summarizes in 1-2 sentences what the extension is usable (should be understandable for non-experts)
  • Any known related patents must be mentioned in the extension description.
  • LICENSE.txt is present in the repository root. MIT (MIT License | Choose a License) or Apache (Apache License 2.0 | Choose a License) license is recommended. If source code license is more restrictive for users than MIT, BSD, Apache, or 3D Slicer license then the name of the used license must be mentioned in the extension description.
  • Extension URL and revision (scmurl, scmrevision) is correct, consider using a branch name (main, release, …) instead of a specific git hash to avoid re-submitting pull request whenever the extension is updated
  • Extension icon URL is correct (do not use the icon’s webpage but the raw data download URL that you get from the download button - it should look something like this: https://raw.githubusercontent.com/user/repo/main/SomeIcon.png)
  • Screenshot URLs (screenshoturls) are correct, contains at least one
  • Homepage URL points to valid webpage containing the following:
    • Extension name
    • Short description: 1-2 sentences, which summarizes what the extension is usable for
    • At least one nice, informative image, that illustrates what the extension can do. It may be a screenshot.
    • Description of contained modules: at one sentence for each module
    • Publication: link to publication and/or to PubMed reference (if available)
    • License: We suggest you use a permissive license that includes patent and contribution clauses. This will help protect developers and ensure the code remains freely available. We suggest you use the Slicer License or the Apache 2.0. Always mention in your README file the license you have chosen. If you choose a different license, explain why to the extension maintainers. Depending on the license we may not be able to host your work. Read here to learn more about licenses.
  • Hide unused github features (such as Wiki, Projects, and Discussions, Releases, Packages) in the repository to reduce noise/irrelevant information:
    • Click Settings and in repository settings uncheck Wiki, Projects, and Discussions (if they are currently not used).
    • Click the settings icon next to About in the top-right corner of the repository main page and uncheck Releases and Packages (if they are currently not used)
  • The extension is safe
    • Does not include or download binaries from unreliable sources
    • Does not send any information anywhere without user consent (explicit opt-in is required)

Extension tier 2 and 4 are reserved for future use, to allow defining intermediate steps between the initially defined tiers without reclassifying every extension.

Extensions catalog entry file

We also propose to switch from the “Extension description (.s4ext)” file format to a simpler, standard JSON format that only contain necessary information. This file will be called Extensions Catalog entry json file. The filename is the exension’s name (e.g., SlicerOpenIGTLink.json) and it contains the following content:

{
    "scmurl": "https://github.com/openigtlink/SlicerOpenIGTLink.git",
    "scmrevision": "master",
    "depends": [],
    "category": "IGT"
    "maintainers": ["https://discourse.slicer.org/u/lassoan", "https://discourse.slicer.org/u/sunderlandkyl"]
    "tier": 5
}

Any comments and suggestions are welcome.

@pieper @jcfr @Sam_Horvath @cpinter @jamesobutler @Sunderlandkyl @RafaelPalomar @mau_igna_06 @muratmaga @rbumm @fedorov

  1. 2024.07.16 Weekly Meeting ↩︎

8 Likes
2

This is great.

It will help to filter the extension manager for older or unsupported extensions.
Concerning the opt-in for sending information from extensions it would be nice to have demo source code available including the don’t ask again feature, maybe in the script repo.

3

For reference, corresponding changes are being implemented in the following pull requests:

1 Like
4

Since we will be able to decouple built-time from runtime dependencies, I suggest we rename depends to build_depends

The “depends” metadata currently set in the extension CMakeLists would then correspond to the runtime dependencies (aka extension expected to be installed after clicking on the “install” button)

5

Renaming sounds good. I would slightly prefer using full words, such as build_dependencies.

In the long term, both build and runtime dependencies should be specified in the repository, in a separate json file so that the both the extension build script can easily read it (to determine the order of build) and CMake files in the extension can access it (so that it can check that all required directories are set, etc.). At that point, the dependencies can be removed from the extensions catalog entry file.

1 Like
6

Does this concern extensions already in the extension repository?

7

Yes, we’ll need to assign every extension to a tier. I would expect that most extensions that are currently actively maintained will be in Tier 3.

8

Will setting tiers for all the existing extensions and changing the extensions review process to be more flexible for lower tier extensions happening soon? It appears that the above mentioned PRs regarding switch to the new json format have been completed.

9

^ These requirements in Tier 3 probably exclude any new developer or a new extension from starting out being published at Tier 3. This is because there isn’t a record yet if the maintainer will respond to questions on the forum and also unclear if the maintainer responds to issues and PR with a brand new extension that hasn’t been used yet.

10

We could accept a promise from newcomers to be responsive (and downgrade the extension if the developer turns out not to be committed enough).

But I would also find it reasonable to limit accessible tiers for extensions that don’t have any established community member among its developers (and the extension would be upgraded over time if the developer team proves to be responsive).

11

Will setting tiers for all the existing extensions and changing the extensions review process to be more flexible for lower tier extensions happening soon?

Now that we just finalized the update of all opened pull-requests so that they include a json file instead of a s4ext file, this will happen next :rocket:

In the meantime, you can also see here the list of pull requests consolidating extension metadata based on the corresponding s4ext that was contributed.

12

Update: The definition of tiers was revisited during July 16th weekly meeting and this post was updated accordingly.

1 Like
13

A pull request has been submitted that assigns tier value for each extension. By default, tier 3 is used for all extensions - except extensions that are not maintained or has other issues are assigned tier 1, and highly important extensions are assigned tier 5. It is expected that tiers will be finalized within about a week. Please review and if you have any comments (e.g., suggest changing tier of a specific extension) then submit it to the pull request:

14

Just like to put my grain of sand here. For me the extension tiers would mean:
5 → Updated documentation and works well on stable release
4 → Old documentation but works well on stable release
3 → Works well on earlier release and has old documentation
2 → Works well on earlier release but no documentation
1 → Does not work

15

1 → Does not work

If an extension does not work then it is archived (extension description is moved to the ARCHIVE folder.

All these will be mapped to the same tier (Tier 3) because it would be very expensive (thousands of dollars) to test all 200+ extensions or check their documentation after each release. Even if we had the funding to do this now, it would not be sustainable and it would not scale. We already have trouble keeping up with just reviewing new extensions. In contrast, we can easily assess available support: we can see when users ask questions or report problems and developers do not respond. If this happens then we can downgrade an extension’s tier. If a developer requests it and the requirements are met then we can raise an extension’s tier.

Tier 5 is distinguished from tier 3 by their maintainers (same people as Slicer core), long term availability, and quality (additional quality requirements in addition to tier 3). These could be independent attributes. However, it is simpler for users if we can map all properties to a single linear scale. We can revisit in the future if we find that is too much of an oversimplification.

2 Likes