How to start with monailabel for new models

Support
82

@diazandr3s Not yet :slight_smile: Have been trying this now on/off for more than two weeks and I would like to get this running before the workshop locally in order to be able to assist other people.

What I have done so far:
Set up Monailabel on two different systems (laptop 6 GB GPU memory, desktop 16 GB GPU memory) and now switched to the desktop completely due to inconsistent results and -9 server error messages on the laptop.
Set ML up in WSL Ubuntu, enabled CUDA, did a parallel setup in Windows 11 native, trained, infered, technically it seems to work in both worlds without error messages.

The goal is to train ML with the Lung CT Segmenter to detect ā€œright lungā€ in total, ā€œleft lungā€ in total, and kind of an airway. It never worked so far. Tried this with lung CT segmenter input as well as with manual segmentations using the ML predefined segments.

All I get is an ugly block segmentation when I use the ML auto segmentation upon ā€œNext sampleā€ā€¦
This is my start_server line:

monailabel start_server --app ./apps/radiology --studies ./datasets/Task06_Lung/imagesTr --conf models segmentation

Could you provide a complete and working application that I could start training with the Task06_Lung dataset before the MONAILabel workshop?

Kind regards
Rudolf

3 Likes
84

Thanks for your time on this, @rbumm and sorry for the inconvenience.

The first thing Iā€™d like you to check is the labels in the Task06_Lung dataset. What I see as labels in that dataset are actually lung tumors, not lungs and airway segmentation. See the following image:

sample-lung-dataset.PNG
sample-lung-dataset.PNG1917Ɨ1039 119 KB

I might be using a different version of that dataset.

Iā€™m happy to meet on Teams/Zoom early Monday if youā€™re available.

Please let me know.

2 Likes
85

Great, absolutely. Monday is fine.

1 Like
86

Noticed that too and thought that I had not used those labels, but will check again

1 Like
87

Superb! Just sent you the Teams link for next Monday at 13:30 UK time. Please confirm.

88

After great meetings with @diazandr3s we were able to set the MONAILabel lung segmentation up from scratch, train a model, and autosegment lungs and airways.

Windows: We recommend doing this in Windows 11 using Powershell.
Follow the instructions from the ML Github to install the prerequisites and ML itself.

All in the above link:

  • Test that CUDA is available.
  • Fork MONAILabel from Github
  • Download ML radiology app
  • Download lung dataset

Use Slicer 5.1 and ML extension
We found that the Deepedit model is best suited for our task.
We modified the deepedit.py in /apps/lib/conf as follows:

        # Single label
        self.labels = {
            "right lung-1": 1,
            "left lung-2": 2,
            "airways-3": 3,
            "background": 0,
        }

        # Number of input channels - 4 for BRATS and 1 for spleen
        self.number_intensity_ch = 1

        network = self.conf.get("network", "dynunet")

        # Model Files
        self.path = [
            os.path.join(self.model_dir, f"pretrained_{self.name}_{network}.pt"),  # pretrained
            os.path.join(self.model_dir, f"{self.name}_{network}.pt"),  # published
        ]

        # Download PreTrained Model
        if strtobool(self.conf.get("use_pretrained_model", "false")):
            url = f"{self.conf.get('pretrained_path', self.PRE_TRAINED_PATH)}/deepedit_{network}_singlelabel.pt"
            download_file(url, self.path[0])

        self.target_spacing = (1.0, 1.0, 1.0)  # target space for image
        self.spatial_size = (128, 128, 64)  # train input size

Then we started the ML server with:

monailabel start_server --app apps/radiology --studies datasets/Task06_Lung/imagesTr --conf models deepedit

In 3D Slicer, we connected to ML Server in the ML extension and pressed ā€œNext sampleā€.
Then we made a couple of lung and airway segmentation with Lung CT Segmenter and ā€œsubmit Labelā€ for each in ML.

We trained a test model with only 50 epochs, and did an autosegmentation on the ā€œnext sampleā€.
The result was still suboptimal, so I produced 9 more labels in Lung CT Segmenter, uploaded them via ā€œsubmit Labelā€ to ML one by one, then trained the model with CUDA on my GTX 3070 Ti for 1000 epochs (10 hours)
After that, autosegmentation was very fast and shows promising results:

image
image928Ɨ550 116 KB

5 Likes
89

Many thanks for the detailed message, @rbumm :slight_smile:

90

Currently, my goal is to use my pre-trained model in 3DSlicer for interactive segmentation. The discussions in the group seem to be in the same line and illuminating. I am looking for suggestions in my work. The procedure that I followed is -
I have generated a pre-trained model and added the path of the .pth file to the lib ā†’ conf ā†’ deepedit.py file. I edited the labels accordingly.

On starting the monailabel server and loading the model, I run the autosegment. I can see the same .pth file being loaded in the cmd window.

The model developed (.pth) has been tested to generate accurate results. On running in 3D Slicer it generates completely odd results.

My doubt is that my procedure to load the .pth file into the module is correct ? Do I need to do anything more ? Perhaps in the infer folder ?

Thanks

92

Hello, I have a similar problem with this ā€˜ā€˜CUDA errorā€™ā€™ when I am using the App radiology app and the spleen dataset which was used in an example in a tutorial.

Screenshot_10
Screenshot_101600Ɨ50 29.4 KB

93

This indicates that the GPU of your computer has insufficient dedicated video RAM - at least 6 GB is recommended to run the server. Please check your NVIDIA control panel ā†’ System Information.
You may consider upgrading your GPU to a more recent model or running the server in the cloud.

94

Hello and thank you very much for your answer !!

Actually,I checked all that when I was installing MONAI.

My GPU is GeForce MX450 and as I was searching it is compatible with CUDA 10.0 to 10.2.

What GPU in your opinion I should install so that MONAI can work ?

Ī£Ļ„Ī¹Ļ‚ Ī£Ī¬Ī², 11 Ī¦ĪµĪ² 2023, 13:23 Īæ Ļ‡ĻĪ®ĻƒĻ„Ī·Ļ‚ Rudolf Bumm via 3D Slicer Community <notifications@slicer.discoursemail.com> Ī­Ī³ĻĪ±ĻˆĪµ:

95

We would recommend a RTX 3060 (12 GB RAM) or RTX 3090 (24 GB) for current CUDA based deep learning purposes. Got a MONAILabel server running on a GTX 1060 (6 GB) but this is the very low limit.

96

Thank you a lot for your answers, you saved me valuable time for this issue.

Judging from the prices of these graphic cards,

Do you believe running MONAI in the cloud as you said would be a valid solution to this situation ?

97

If yes, which cloud do you suggest for AI procedures such as this?

98

The instructions here for AWS are convenient for windows users. And linux instructions are here for Google but similar techniques can be used on other cloud providers. The nice thing about these environments is that there are many GPU options and you only pay for the time you need.

1 Like
99

Thank you very much:) Kind of basic question, Docker for Windows cannot work in this case?

100

You can use docker on any of these platforms if you find it useful. Personally I find that for the cloud itā€™s more convenient just to use the native OS so thereā€™s only one layer of GPU drivers to worry about.

101

Hello again

I just created an account on AWS for window users. However, I cannot download the template you provided in a link to use it in the stackā€¦

Also, I have a question regarding the cost of the AWS. If the MONAILabel is under the free use of AWS or not.

Thank you a lot for your valuable time and help
Best
Spiros

102

My badā€¦I downloaded it.

103

Hello again

I followed the instructions for AWS and tried to create a cloud for MONAI using the already template.

However, I got this ROLLBACK_COMPLETE message:

Screenshot_20
Screenshot_201833Ɨ598 126 KB

What am I doing wrong here ?

Best and thank you all for your time,
Spiros.