Sam_Horvath
April 22, 2024, 8:31pm
1
Tomorrow, we will be having our next weekly hangout at 10:00 AM ET until 11:00 AM ET .
Anyone is welcome to join at this link: https://bit.ly/slicer-googlemeet-hosted-by-kitware
Agenda:
Please post to this thread to put a topic on the agenda! We will try to prioritize agenda items during the meeting.
Thanks
evaherbst
April 23, 2024, 11:10am
2
I would like to discuss 1) modifying the TotalSegmentator widget as suggested in this thread and 2) fixing my issue with TotalSegmentator license not being saved
jcfr
April 23, 2024, 1:20pm
3
Discuss and finalize these two pull requests:
Notes:
Related posts:
allemangd
April 23, 2024, 1:31pm
4
Discuss revision of proposal for scripted modules requirements.txt support following last week’s feedback.
opened 08:15PM - 22 Apr 24 UTC
make-simple-things-simple
Continuation of #7697; tightly coupled to #7171 and #6913.
After discussion a… nd feedback at the weekly hangout Apr 16 (see https://github.com/Slicer/Slicer/issues/7697#issuecomment-2059271196 and https://github.com/Slicer/Slicer/issues/7697#issuecomment-2059273077) I think my first issue conflated three separate tasks:
- Declare and install dependencies
- Run modules in an isolated subprocess (CLI modules)
- Run modules in an isolated environment (venv, conda, poetry, etc)
I'm interested in tackling only the first of these, and since CLI have questionable value anyway it doesn't make sense to require CLI usage to get the benefit here. Really, what we want is a way for scripted module's logic to depend on pypi packages, and only install those packages when they are needed. Doing this purely in the Python layer allows all scripted modules to benefit.
## Draft implementation and features
I have a Python 3.9 draft implementation of a `LazyImportGroup` context manager, based on the [`LazyLoader`](https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader) and [`lazy_import` recipe](https://docs.python.org/3/library/importlib.html#implementing-lazy-imports) in the standard library `importlib` documentation.
https://github.com/allemangD/lazyloader - see the doctests for `LazyImportGroup` for more details.
The important principles here are:
- Consolidate. Keep all `import` statements at top-level, all dependencies in `requirements.txt`.
- Lazy. Do not install or import expensive libraries until they are used.
- Reliable. Do not import libraries unless they are guaranteed to be available in a user's environment.
These were not mentioned during the discussion last week, but I think they are important:
- Support static analysis.
- Be explicit about when "magic" occurs.
---
Usage requires the extension developer place a `requirements.txt` in a python package, installed via `RESOURCES` argument of the existing CMake macros. The [anchor and path](https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files) for that `requirements.txt` resource are given as an argument to `LazyImportGroup` context manager, wrapping top-level imports.
```py
with LazyImportGroup('lib:requirements.txt'):
import lib.logic
import itk
```
These imports are real import statements that are detected by static analysis, and run-time magic only occurs with imports within the context manager. These imports produce proxy modules that defer install and real import until first usage:
```py
def onApplyButton(self):
lib.logic.do_the_task(...)
# ^ attribute access triggers install and import
def onComputeButton(self):
itk.Fpfh....
# ^ attribute access triggers install and import
```
---
Modules imported within a `LazyImportGroup` can safely have bare top-level imports of their dependencies. This is probably recommended usage for this tool:
```
qt-scripted-modules
|- libCompute
| |- __init__.py
| |- requirements.txt
| \- logic.py
\- Compute.py
```
Above, `__init__.py` and `train` can safely include their dependencies at top level, as long as `Compute.py` imports them in a lazy group:
```
# requirements.txt
A
B
```
```py
# train.py
import A
```
```py
# __init__.py
import B
from .logic import ...
```
```py
# Compute.py
with LazyImportGroup('libCompute:requirements.txt'):
import libCompute
import libCompute.logic
```
---
Only the first attribute access of a module in a group will trigger the installation for all imports in that group. There may be multiple groups with different requirements to denote multiple feature sets, and only the used features will be installed. Each group is installed at most a single time.
```py
with LazyImportGroup('lib:training-requirements.txt'):
import X
import Y
with LazyImportGroup('libinference:requirements.txt'):
import Z
```
The first attribute access of X or Y will install `training-requirements.txt`, and the first attribute access of Z will install `inference-requirements.txt`.
A fast `pip install --no-deps --dry-run --report` check is first used to ensure all the items in `requirements.txt` are satisfied. If so, no action is taken, the module is imported, and the proxy module is updated. If not, the requirements are first installed with `pip install -r`.
The dry-run report is used to produce a summary of packages to be installed. The Slicer implementation should show this to the user with `slicer.util.confirmOkCancelDisplay`.
Not implemented in the draft, but the report and install steps could accept a `-c constraints.txt` argument to pin Slicer libraries like `numpy`, `SimpleITK`, etc. Also the requirements of other groups could be inspected to detect incompatibilities with messages like: "Module X wants to install package Y which is incompatible with module Z. Proceed?"
---
I've tested these proxy objects on plain python packages, loadable packages, and even `itk`'s complex lazy-loaded metapackage. It works on all of these, although there are some caveats on object identity and safe import order. See the doctests for details.
## Outstanding questions
- Where to place this? Suggest `slicer.LazyImportGroup`.
- Bikeshed the name? Signature? It also does pip installations.
- Best practices on confirmation dialog?
- Should constraints files or compatibility checks between modules be part of this initial feature? Or should they be delayed till we see usage in practice.
---
From prior discussions: cc @jcfr @pieper @lassoan @ebrahimebrahim
pieper
April 23, 2024, 5:53pm
6
jcfr:
This portal is really nice. Be sure to check out this Quick Guide document. It nicely summarizes the steps in a workflow and each slicer icon is a deep link into a demo video illustrating the use of the tool. It’s designed for biology samples but it’s a great resource for anyone doing segmentation and analysis:
http://www.graysvertebrateanatomy.com/__static/c0e61a322fc1f771762f9a4d60fbf7ee/gray-3d-slicer-quick-guide.pdf?dl=1
2 Likes
