2024.04.16 Weekly Meeting

Sam_Horvath · April 15, 2024, 8:35pm

Tomorrow, we will be having our next weekly hangout at 10:00 AM ET until 11:00 AM ET .

Anyone is welcome to join at this link: https://bit.ly/slicer-googlemeet-hosted-by-kitware

Agenda:

Please post to this thread to put a topic on the agenda! We will try to prioritize agenda items during the meeting.

Thanks
Sam and J-Christophe

pieper · April 15, 2024, 9:38pm

I’d like to discuss updating ITK to get the fix for ITK #4109.

allemangd · April 16, 2024, 1:59pm

I’d like to discuss my requirements.txt proposal for scripted modules

github.com/Slicer/Slicer

Allow scripted modules to declare pip requirements and wrap pip dependencies.

opened 02:35PM - 15 Apr 24 UTC

allemangD

make-simple-things-simple Domain: build-system

This is tightly related to #7171 and #6913, but since this is more an RFC for a …particular solution, so I'm creating a separate issue for discussion specifically on this proposal. --- Install scripted modules python dependencies during module discovery to support new use cases: Safely `import` pip-installed dependencies at the global level without `catch ImportError: slicer.util.pip_install(...)` wrappers. Create scripted CLI modules which wrap python package [entrypoints](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). Supersede `slicer.util.pip_install` usage in scripted loadable modules to help prevent/debug environment breakage from incompatible dependencies. Create scripted CLI modules implemented in multiple Python source files. # Proposed solution When scripted modules are discovered (ie. `$NAME.py` or `$NAME.xml` is discovered) check for a corresponding `$NAME-requirements.txt`. When the module is initialized (lazily), check the dependencies and install. Show a summary and confirmation dialog to the user, similar to `pip install` when `-y` is not given. If installation is rejected, do not load the module. For an extension to install `$NAME-requirements.txt`, they should use the `RESOURCES` argument to `SlicerMacroBuildScriptedModule`. This requires the `$NAME-requirements.txt` appear, with that name, in the source tree so that "installing" a module from source via module search path will still discover the file. `SlicerMacroBuildScriptedCLI` does not currently have a `RESOURCES` argument, so additionally: add `SCRIPTS` and `RESOURCES` arguments and update it to use `ctkMacroCompilePythonScript` in the same way as `SlicerMacroBuildScriptedModule`. If neither `RESOURCES` nor `SCRIPTS` are provided, automatically add `$NAME.py` and `$NAME.xml` to each, respectively, for backwards compatibility. Note - this also allows splitting CLI modules into multiple files and multiple scripts. Only the script with a matching `$NAME.xml` is discovered at runtime. Add a convenience argument `ENTRYPOINT` to `slicerMacroBuildScriptedCLI` (with the same semantics as [setuptools console script entrypoints](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#console-scripts)) which is mutually exclusive with `SCRIPTS`. If provided, at configure time, generate a `$NAME.py` which invokes the function identified by `ENTRYPOINT`. ## Examples The current behavior would be unchanged: ```cmake slicerMacroBuildScriptedCLI( NAME Foo ) slicerMacroBuildScriptedModule( NAME Bar SCRIPTS ... RESOURCES ... ) ``` Now it would be possible to explicitly list the script and xml for `ScriptedCLI`, or to declare additional scripts/resources: ```cmake slicerMacroBuildScriptedCLI( NAME Foo SCRIPTS Foo.py ... RESOURCES Foo.xml ... ) ``` Each module could declare a `requirements.txt` to be checked just before module initialization: ```cmake slicerMacroBuildScriptedCLI( NAME Foo SCRIPTS Foo.py ... RESOURCES Foo.xml Foo-requirements.txt ... ) slicerMacroBuildScriptedModule( NAME Bar SCRIPTS Bar.py ... RESOURCES Bar-requirements.txt ... ) ``` And the CLI module could wrap a pip-installed entrypoint: ```cmake slicerMacroBuildScriptedCLI( NAME Baz ENTRYPOINT libbaz.run:main RESOURCES Baz.xml Baz-requirements.txt # installs `libbaz` ) ``` # Alternatives considered - Dependencies declared per-extension, rather than per-module. - Dependencies declared in a `requirements.txt` which is renamed to `$NAME-requirements.txt`. - Dependencies declared directly in CMake. - `REQUIREMENTS_FILE` or `PIP_DEPENDENCIES` or similar dedicated CMake macro argument. All these suffer issues when developers "install" modules by adding the source tree to Slicer module path. Placing the dependencies in a `$NAME-requirements.txt` forces them to be picked up at module discovery and ensures they are updated as the file is edited. Even if an extension developer does not build the project with CMake, the requirements are sure to be handled for users when the project is built and published via CI. If the developer forgets to declare the requirements file in `RESOURCES`, there would be a clear import error in user logs that the required packages are missing. - Include `pip_install` calls in the generated `$NAME.py` file for `ENTRYPOINT`. This only supports the `ENTRYPOINT` mechanism. This invokes dependency resolution on every CLI _invocation_, not just on CLI instantiation. It is incompatible with installing modules from source via module path. # Future Planning ## CLI module virtual environments In the future if CLI modules get their own virtual environment, module initialization must create the virtual environment and install `$NAME-requirements.txt` to _that_ virtual environment, then launch `$NAME.py` in the same environment. Behavior would be the same with or without the `requirements.txt`. ## Slicer Constraints File If a Slicer constraints file is provided to lock versions of built-in packages like `numpy`, `vtk`, `SimpleITK`, etc. it may be provided with `pip -r $NAME-requirements.txt -c path-to-slicer-constraints.txt`. Should not be done for CLI modules if they have their own virtual environments. ## Dependency resolution errors Discover all _loadable_ module requirements.txt togethre and invoke pip once with all the requirements listed together? This would detect incompatibilities between modules and mitigate startup time cost - especially if using `uv`. Should not be done for _CLI_ modules if they have their own virtual environments. e.g. `uv install -r Foo-requirements.txt -r Bar-requirements.txt -c path-to-slicer-constraints.txt`; might detect an incompatibility between `Foo` and `Bar` dependencies. How should this error be handled? The user should have a means to downgrade or remove one or both of the infringing extensions. ## Deprecate `slicer.util.pip_install` Once `requirements.txt` is deemed mature, we should show a deprecation warning on `slicer.util.pip_install` to encourage extension developers to migrate to this mechanism; else dependency resolution will not be useful. # Questions `ENTRYPOINT` would not be supported by install-from-source, since it depends on CMake configure time. Is this a problem? Scripted Loadable modules are initialized during startup - this means _all_ scripted modules dependencies would be resolved at startup. Is this overhead significant? We could mitigate this by: - using `uv` instead of `pip` - collecting all requirements and installing them together with multiple `-r` arguments. (This would also identify incompatibilities between extensions' dependencies). - doing a fast-check with `pip freeze` to avoid dependency resolution in the happy path when everything is already installed. Should pip installation be done at _extension_ installation time? How would from-source modules be handled? How would extension installation time module discovery be handled? Should `pyproject.toml` be involved in any way? How do we handle from-source installation and module discovery? `tomllib` is available to Python in 3.11, or [tomli](https://pypi.org/project/tomli/) for prior versions. Setuptools supports [dynamic dependencies](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata), but this is a _beta_ feature. This technically allows installing multiple `.xml` files in a single `slicerMacroBuildScriptedCLI` call. Should this be explictly forbidden? Should this be explicitly allowed? Should multiple `ENTRYPOINT` arguments be allowed? --- I am working on a draft implementation of these changes but it is not yet complete. I invite comments and concerns in the meantime. cc @jcfr @ebrahimebrahim

Topic		Replies	Views
2024.07.16 Weekly Meeting Weekly meetings	4	37	July 16, 2024
2024.04.23 Weekly Meeting Weekly meetings	5	200	April 23, 2024
2024.04.02 Weekly Meeting Weekly meetings	4	166	April 2, 2024
2024.03.19 Weekly Meeting Weekly meetings	3	138	March 19, 2024
2024.04.09 Weekly Meeting Weekly meetings	1	102	April 9, 2024

2024.04.16 Weekly Meeting

Related topics