How to start with monailabel for new models

[rbumm (Rudolf Bumm)]

@diazandr3s

I meant the radiologist/clinician using MONAI Label

@diazandr3s or that use case: How would the annotator benefit from monailabel and the monailabel model during segmenting without labels? Which buttons could he press or which functions would he use to create AI-assisted segmentation easier/faster than in the normal segment editor?

[muratmaga]

@rbumm

How would the annotator benefit from monailabel and the monailabel model during segmenting without labels?

MonaiLabel can be used both to infer and train. If your goal is to facilitate segmentation, first you train a model in say a small number of samples with labelmaps. You might even get something with 10-12 samples (not good but ok).

For inference you have two options, you can either use a dataset loaded into Slicer scene and can ask MonaiLabel to segment it. Or you can deploy the MonaiLabel server such that there are unsegmented volumes on the server side and that’s displayed in the dropdown menu in the monailabel extension and you can just load one after other and run inference on them.

Then there is even an option to push those segmentations back to the server and restart training with them (in addition to the first 10-12 samples you have trained the model on).

The UI is a bit complex, and as I mentioned to the @diazandr3s the inference and training option probably should be decoupled (mostly to avoid accidentally running the training). This might be done perhaps at the server side (e.g., a flag to deploy the server only for inference, or some sort of authentication for training)

[diazandr3s (Andres Diaz-Pinto)]

Thanks for the detailed explanation, @muratmaga

@rbumm please let us know if there is anything else I can help with

All your feedback is valuable to making MONAI Label better and easier to use.

[rbumm (Rudolf Bumm)]

@muratmaga

MonaiLabel can be used both to infer and train. If your goal is to facilitate segmentation, first you train a model in say a small number of samples with labelmaps. You might even get something with 10-12 samples (not good but ok).

Thank you. Hope it is ok for the community to push that thread up again, if not, I could also DM @diazandr3s or @muratmaga.

Probably best to try the train option. Is it correct that I would need to

• segment each of let’s say 12 of my 65 lung datasets and create “right lung”, “left lung” and “airways”
• am I allowed to do this segmentation in a different segmentation program?
• after segmentation of each dataset → switch back to monailabel extension and “submit Label”
• After each “Submit” or after a few “Submits” do a “Train”

Now comes the part which I do not understand:

At patient 12/65 I would like to check the performance of the trained monailabel model.
So I load patient dataset 13.
Expect to press a button in monailabel which makes the extension populate the labels “right lung”, left lung", “airways” in dataset 13 from what it has learned so far in patients 1-12. I could then improve and “Submit” again.

• Is there such a button / menu entry / python command in monailabel?
• Do I have to place any markups/seeds for this?

[pieper (Steve Pieper (Isomics, Inc.))]

@rbumm

Thank you. Hope it is ok for the community to push that thread up again, if not, I could also DM @diazandr3s or @muratmaga.

Let’s do keep this public - I find it very valuable and I hope everyone else does too! It’s really important to learn, get feedback, and as needed improve the process or interface.

Your description of the workflow sounds right to me, and your expectation that 12 or 65 datasets should give a first pass model sounds consistent with expectations I have heard expressed.

@rbumm

• Is there such a button / menu entry / python command in monailabel?
• Do I have to place any markups/seeds for this?

For me, once I had the model trained to 75-80% I was able to use the Run. button to apply to a new case (either one of the training set or a new similar volume loaded in Slicer). No new seeds required.

[muratmaga]

I will list my experience and hopefully @diazandr3s can chime in with more technical aspects.

1. You can do the segmentations in any program, provided that you can export a labelmap and that labelmap and associated volume line up correctly in Slicer.
2. Sounds like you want to do multi-label segmentation. If that’s the case you really want to pay a lot of attention that each structure is label with identical label in each of the labels. If Left Lung is 2, and Right Lung is 3 in some segments and vice versa your training will fail.
3. I haven’t used the Active Learning paradigm yet, but what you describe sounds reasonable (meaning you push newly segmented volumes and associated labelmaps back to the MonaiServer, and then when you have a bunch more, retrain the model).
4. For how to parse the data automatically, you need to look at the data organization on the MonaiLabel server side. As I recall, you will organize such that 65 volumes in one folder and 12 ground truth label maps in sub folder called labels/final underneath it. File names and formats of volumes and labels has to match. Then, if you use the next sample button it should download automatically the 13th one (and other unsegmented ones), then you can hit the Run deep-edit_seg button and run the model on them (inference). This is of course after you train a model.

[mikebind (Mike Bindschadler)]

Please do keep this public. I plan to try out MonaiLabel through Slicer in the next few weeks, and am trying to keep up with this discussion which I think will be very helpful when I get to starting!

[rbumm (Rudolf Bumm)]

@mikebind

Please do keep this public

Ok guys, then this may also be a timesaver when you try to set up the monailabel server with a more recent NVIDIA graphics card (NV is mandatory for CUDA):

First and most importantly, one has to upgrade to Windows 11 to run WSL2 and CUDA with recent cards.
Still ran into

GeForce RTX 3070 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75

shown in the monailabel server output and training sessions failed.
torch.version.cuda showed:

>>> import torch
>>> torch.version.cuda
'10.2'

So it appeared that two CUDA versions were on that computer simultaneously.
Got this only solved by

# remove all CUDAs
sudo apt-get --purge remove "*cublas*" "cuda*" "nsight*"
sudo apt-get --purge remove "*cublas*" "*cufft*" "*curand*" 
# install torch 1.9 and CUDA 11.1
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

After that

>>> import torch
>>> torch.version.cuda
'11.2'

and there was no more warning, monailabel training worked with that GTX 3070 Ti too.

[diazandr3s (Andres Diaz-Pinto)]

I think this is a great discussion. I’d keep it public

Thanks again for sharing your experience, @muratmaga

You accurately highlighted the most important things:

1/ labels can be created/modified in any program and then submitted to MONAI Label server,
2/ each structure should be consistently labeled with an identical label number, and
3/ data organization in MONAI Label matters (i.e. when starting with already created labels, both images and labels should have the same name …)

Thanks, everyone!!

[muratmaga]

@rbumm

shown in the monailabel server output and training sessions failed.

İn my experience, matching all the tools and base libraries and drivers is half the challenge of setting up a deep learning environment. You took an extra step in doing all this under wsl2. But sounds like you are making progress.

[rbumm (Rudolf Bumm)]

Would this part of the configuration be right for setting up a lung deepedit conf ?
How could we start a model from scratch when you can not download a pretrained model (see → # Download PreTrained Model section) ?

...
        # Single label
        #self.labels = {
        #    "spleen": 1,
        #    "background": 0,
        #}
        # Single label
        self.labels = {
            "right lung": 1,
            "left lung": 2,
            "airways": 3,
            "background": 0,
        }

        # Number of input channels - 4 for BRATS and 1 for spleen
        self.number_intensity_ch = 4

        network = self.conf.get("network", "dynunet")

        # Model Files
        self.path = [
            os.path.join(self.model_dir, f"pretrained_{self.name}_{network}.pt"),  # pretrained
            os.path.join(self.model_dir, f"{self.name}_{network}.pt"),  # published
        ]

        # Download PreTrained Model
        if strtobool(self.conf.get("use_pretrained_model", "true")):
            url = f"{self.PRE_TRAINED_PATH}/deepedit_{network}_singlelabel.pt"
            download_file(url, self.path[0])
...
``

[rbumm (Rudolf Bumm)]

Assuming we load a “Next sample” in 3DSlicers monailabel extension after starting the monailabel server with

monailabel start_server --app ./apps/radiology --studies ./datasets/Task06_Lung/imagesTr --conf models deepedit

in which standard segmentations of the above labels are created ok by monailabel:
Does preservation of segment color matter for the learning process if you switch to an external segmentation module and the back to monailabel ?

[rbumm (Rudolf Bumm)]

@rbumm

in which standard segmentations

I ment “in which empty standard segments are created ok”

[rbumm (Rudolf Bumm)]

There is very very slow progress in getting a monailabel lung CT segmentation server up and running.
What I did:

Changed segmentation.py in ./apps/radiology/lib/conf/

        # Labels
        #self.labels = {
        #    "spleen": 1,
        #    "right kidney": 2,
        #    "left kidney": 3,
        #    "gallbladder": 4,
        #    "esophagus": 5,
        #    "liver": 6,
        #    "stomach": 7,
        #    "aorta": 8,
        #    "inferior vena cava": 9,
        #    "portal vein and splenic vein": 10,
        #    "pancreas": 11,
        #    "right adrenal gland": 12,
        #    "left adrenal gland": 13,
        #}

        # Labels
        self.labels = {
            "right lung": 1,
            "left lung": 2,
            "airways": 3,
        }

Then

monailabel start_server --app ./apps/radiology --studies ./datasets/Task06_Lung/imagesTr --conf models segmentation

resulted in multiple error messages and it took ages to find out that this was caused by monailabel using the multilabel pretrained_segmentation.pt that obviously has much more labels than the three I need for lung segmentation and thus - fails.

It turned out that changing the server startup code to

monailabel start_server --app ./apps/radiology --studies ./datasets/Task06_Lung/imagesTr --conf models segmentation --conf use_pretrained_model false

avoids the errors and enables training.

[muratmaga]

@rbumm

avoids the errors and enables training.

Alternatively, you can rename your app to something, create a separate folder for it and pass that to the monailabel start. İ found that approach i bit cleaner when i was experimenting.

[muratmaga]

@rbumm

Does preservation of segment color matter for the learning process if you switch to an external segmentation module and the back to monailabel ?

I actually don’t know the answer to that. I suspect it does a lot. MonaiLabel works on labelmaps, not segmentation. So the issue is how those segments are converted to labelmaps. I think use of a strict terminology is necessary to for correct conversion. I am trying to learn terminology myself as you can see here

[mikebind (Mike Bindschadler)]

@rbumm, was there a reason you chose to install monailabel in WSL2 rather than in Windows?

[rbumm (Rudolf Bumm)]

This is a very good question @mikebind, for some reason and experience, I thought that this kind of server can only be set up on a Linux (sub)system, which is probably not the case. Generally not good news, because I spent days adjusting CUDA in WSL2 (see above).

The monailabel server (Windows 11, WSL2) now runs without error messages here, I can train it, but I do not get usable segmentation results in 3D Slicer yet and I decided to put this on hold until the Monailabel workshop in June or until there are new bright ideas.

Are you using monailabel directly in Windows from the powershell?

[jamesobutler (James Butler)]

Yes MonaiLabel is supported on Windows. GitHub - Project-MONAI/MONAILabel: MONAI Label is an intelligent open source image labeling and learning tool.

[rbumm (Rudolf Bumm)]

It can confirm that actually works in Windows 11 native too.

Needed to install Python 3.9, then

git clone https://github.com/Project-MONAI/MONAILabel
$Env:PATH += ";C:\Users\rudol\MONAILabel\monailabel\scripts"
monailabel apps --download --name radiology --output apps
# modify segmentation.py by hand, add the lung and airway labels
monailabel datasets --download --name Task06_Lung --output datasets
monailabel start_server --app ./apps/radiology --studies ./datasets/Task06_Lung/imagesTr --conf models segmentation --conf use_pretrained_model false

I train 7-8 datasets, but all I get from auto segmentation is something like

where only the first segment carries the displayed data.