# AI-assisted segmentation extension

**URL:** <https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536>\
**Category:** Announcements\
**Tags:** feature, deep-learning\
**Created:** [December 17, 2019, 9:03pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536 "2019-12-17T21:03:10Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 17, 2019, 9:03pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/1 "2019-12-17T21:03:10Z")

</div>

> **Important note: NVIDIA-AIAA extension is still available, but it is not supported and not recommended anymore. It is no longer necessary to set up a server for segmentation and not necessary to even have a GPU. Instead, there are several new extensions that allow segmenting images much more easily. For example, TotalSegmentator and MONAIAuto3DSeg extensions.**

We are excited to announce that [Nvidia AI-assisted segmentation extension](https://github.com/NVIDIA/ai-assisted-annotation-client/blob/master/slicer-plugin/README.md) is ready to use in latest Slicer Preview Release (rev28686 or later). The extension has been developed by Nvidia, with contributions from Slicer core developers. While there have been other AI-assisted segmentation modules in Slicer (such as [DeepInfer](http://www.deepinfer.org/), [TOMAAT](https://github.com/faustomilletari/TOMAAT-Slicer), [SlicerCIP](https://chestimagingplatform.org/workstation-slicer-cip)), this newest addition uses [Nvidia Clara](https://developer.nvidia.com/clara), a toolkit with significant industrial support and sufficient openness for researchers.

Few-minute overview video, showing guided MRI brain tumor and liver segmentation, and fully automatic liver, tumor, and spleen segmentation:

[![](https://img.youtube.com/vi/ucnvE16pkmI/maxresdefault.jpg "AI-assisted segmentation using free tools - 3D Slicer and Nvidia Clara") ](https://www.youtube.com/watch?v=ucnvE16pkmI)

Tutorial and detailed description: see [Nvidia AIAA extension documentation](https://github.com/NVIDIA/ai-assisted-annotation-client/blob/master/slicer-plugin/README.md).

**How does it work?** The input image (and input points - in case of guided segmentation) are sent to a computer equipped with an Nvidia GPU, running Linux operating system and [Nvidia Clara software](https://docs.nvidia.com/clara/aiaa/tlt-mi-ai-an-sdk-getting-started/). The server computes segmentation using the selected AI model and sends back the results to Slicer for display and further processing.

**We have set up a demonstration server at the PerkLab** (Queen’s University in Canada) to make it easier for Slicer users to get started without setting up their own processing computer. We uploaded a couple of AI models that Nvidia developed. We provide these models and the processing service as is, we don’t guarantee quality of this service (validity of segmentation results, speed, server uptime, etc.). No patient information is sent to the processing server and images and results are deleted from the server after processing, but users need to make sure they comply with their data management guidelines when using our server. If there are any confidentiality concerns then publicly available images may be used for testing: see Slicer’s Sample Data module or [TCIA Browser](https://www.slicer.org/wiki/Documentation/Nightly/Extensions/TCIABrowser) extension, or download from other websites, such as [Medical Decathlon](http://medicaldecathlon.com/).

**If anyone would like to share their AI models for segmentation** , let us know. As long as the model is compatible with Nvidia Clara, we should be able to install it on our server and make it available to the Slicer community.

Any questions and suggestions are welcome!

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [December 17, 2019, 9:24pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/2 "2019-12-17T21:24:51Z")

</div>

This looks great. I assume the existing trained models are for human clinical scans?  
Is it possible to add/train additional reference segmentation (e.g, mouse skulls?, embryos etc)

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 17, 2019, 9:32pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/3 "2019-12-17T21:32:43Z")

</div>

All NVidia models are based on human clinical data sets (you can find more information [here](https://docs.nvidia.com/clara/aiaa/tlt-mi-ai-an-sdk-getting-started/)). However, the methods are completely generic and you should be able to use [Clara Train](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v2.0/index.html) to build models from your data.

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 17, 2019, 9:55pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/4 "2019-12-17T21:55:22Z")

</div>

This is indeed great news !!! 🥳

However only segmentation spleen and segmentation liver and tumor is shown in mine.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 17, 2019, 10:15pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/5 "2019-12-17T22:15:35Z")

</div>

I’ve only installed liver+tumor and spleen models for fully automatic segmentation (others were listed to have rather low scores or just did not seem that interesting).

There are about 10 more models for boundary points based segmentation.

If you want us to install any other models listed [here](https://docs.nvidia.com/clara/aiaa/tlt-mi-ai-an-sdk-getting-started/) then let me know.

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 17, 2019, 10:38pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/6 "2019-12-17T22:38:08Z")

</div>

Dear Prof Lassoan,  
Well these AI trained modules at NVIDIA are not relevant to me i looked at that out of interest and i thought it was due to something wrong with the linux version as i downloaded the new build moments ago. I would have loved to experiment with the training on oral cancer but unfortunately reuqired specs are out of my reach.

Hardware Requirements

**Recommended**

- 1 GPU or more
- 16 GB GPU memory
- 8 core CPU
- 32 GB system RAM
- 80 GB free disk space

---

<div class="post-metadata">

**Author:** ![jdx-john](https://avatars.discourse-cdn.com/v4/letter/j/a6a055/32.png) [@jdx-john](https://discourse.slicer.org/u/jdx-john)\
**Post date:** [December 17, 2019, 11:15pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/7 "2019-12-17T23:15:18Z")

</div>

This sounds really cool. @lassoan I wondered if you recall our CT cardio project… Any ideas if this might work for automatic segmentation of heart datasets like we were dealing with?

John

---

<div class="post-metadata">

**Author:** ![keesh](https://avatars.discourse-cdn.com/v4/letter/k/e19adc/32.png) [@keesh](https://discourse.slicer.org/u/keesh)\
**Post date:** [December 17, 2019, 11:30pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/8 "2019-12-17T23:30:40Z")

</div>

The automatic liver/spleen segmentation was impressive. NVidia has come a long way fast since the manual point and click approach. I plan to compare their approach to a Caffe 2D UNet paradigm. My only concern is the low input resolution (128 x 128 x 128) required.

---

<div class="post-metadata">

**Author:** ![JanWitowski](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/janwitowski/32/4512_2.png) [@JanWitowski](https://discourse.slicer.org/u/JanWitowski)\
**Post date:** [December 18, 2019, 6:35am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/9 "2019-12-18T06:35:03Z")

</div>

Andras,  
this is amazing news. Thinking that we might have a true one-click segmentation very soon is so exciting.

I tested this extension, specifically liver segmentation models, on my local datasets and wanted to share some thoughts:

1. Liver segmentation from boundary points works flawlessly. It is by far more accurate and faster than any other approach available in Slicer (grow from seeds, marching…). In most cases it doesn’t require any postprocessing. For people like me, who work with liver parenchyma segmentation regularly, this is another major upgrade.
2. Automatic liver and tumor segmentation has some accuracy issues, though. It looks like model is a little overfitted on Medical Decathlon dataset. I tested a few examples from that set and it did pretty accurate segmentation. On my local dataset it had a lot of trouble and is not functional at the moment (though it has some strong moments and can detect small lesions). Also, it’s too time-consuming for the effects you’re getting. A few examples:  
 ![clara-01.PNG](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/b/3/b349980dc08a7f48419963f104851cf079bfffb4.jpeg) ![clara-02.PNG](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/d/b/db029dda8f2a60f0ff0d93c1bf51611bd61c17da.jpeg) ![clara-03.PNG](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/1/0/10efe69ae3aafd74d1404dc3e0b668866b55e4fc.jpeg)  
I will look at the Clara closer ASAP and try to figure out how to contribute models.

---

<div class="post-metadata">

**Author:** ![brhoom](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/brhoom/32/1228_2.png) [@brhoom](https://discourse.slicer.org/u/brhoom)\
**Post date:** [December 18, 2019, 7:58am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/10 "2019-12-18T07:58:50Z")

</div>

@lassoan

Great news, I just downloaded Slicer 4.11.0 for windows but this extension does not appear in the extension manager. Another question, do you know if there is an automatic spine segmentation available?

> [@JanWitowski](#):
>
> It looks like model is a little overfitted on Medical Decathlon dataset

I think it is possible to fine-tune the models with new datasets. It would be nice if this is available in the slicer module.

---

<div class="post-metadata">

**Author:** ![NoobForSlicer](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/noobforslicer/32/10900_2.png) [@NoobForSlicer](https://discourse.slicer.org/u/NoobForSlicer)\
**Post date:** [December 18, 2019, 10:18am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/11 "2019-12-18T10:18:35Z")

</div>

I have a question… Can this thing be used to segment whatever organ we want or just the ones they prepared the models of segmentation for?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 18, 2019, 2:01pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/12 "2019-12-18T14:01:34Z")

</div>

> [@JanWitowski](#):
>
> Liver segmentation from boundary points works flawlessly. It is by far more accurate and faster than any other approach available in Slicer (grow from seeds, marching…). In most cases it doesn’t require any postprocessing. For people like me, who work with liver parenchyma segmentation regularly, this is another major upgrade.

Thanks for the feedback. It’s great to hear that you find the extension useful.

> [@JanWitowski](#):
>
> Automatic liver and tumor segmentation has some accuracy issues, though

I think so, too. If the volume has different size or resolution then the model gets confused. The models are pretty good overall, considering that Nvidia provides these models as technology demonstrations, but if you spend some time with training, especially with your own data sets then you can expect much better results.

> [@JanWitowski](#):
>
> I will look at the Clara closer ASAP and try to figure out how to contribute models.

Awesome. If you can create models and willing to share them then we would be happy to upload them to our segmentation server.

> [@brhoom](#):
>
> Great news, I just downloaded Slicer 4.11.0 for windows but this extension does not appear in the extension manager.

There is a few-hour timeslot each day when the new nightly Slicer installer is already available but not all extensions are built yet. You either need to wait a few hours or download the preview release from the day before using this link: [Download 3D Slicer | 3D Slicer](https://download.slicer.org/?offset=-1)

> [@brhoom](#):
>
> I think it is possible to fine-tune the models with new datasets.

Yes, it is. It is not as easy as running a pre-trained model, so I’m not sure we can create a simple GUI for this in Slicer, but you can follow these [instructions](https://docs.nvidia.com/clara/tlt-mi/tlt-mi-getting-started/).

> [@manjula](#):
>
> unfortunately reuqired specs are out of my reach.

These specs are just recommendations and only relevant for training (for inference, any hardware with a CUDA-capable GPU will do). For training, you should be fine with 8GB GPU memory and 16GB RAM but probably even half of that is enough for many applications.

> [@NoobForSlicer](#):
>
> Can this thing be used to segment whatever organ we want or just the ones they prepared the models of segmentation for?

Each model can segment what it is trained for. You can fine-tune pre-trained models or train new models from scratch if you have enough data sets (for simple problems you might not need that much data and training parameter tuning) by following these [instructions](https://docs.nvidia.com/clara/tlt-mi/tlt-mi-getting-started/) or creating a any model and bringing it into [MMAR format](https://docs.nvidia.com/clara/tlt-mi/tlt-mi-getting-started/#medical_model_archive).

> [@jdx-john](#):
>
> This sounds really cool. @lassoan I wondered if you recall our CT cardio project… Any ideas if this might work for automatic segmentation of heart datasets like we were dealing with?

The training framework is quite generic and heart segmentation on contrasted CT is fundamentally not very hard, so training a model should be feasible if you have enough data sets. There could be some special considerations to handle large 4D periodic data sets.

---

<div class="post-metadata">

**Author:** ![e36alpine](https://avatars.discourse-cdn.com/v4/letter/e/a9a28c/32.png) [@e36alpine](https://discourse.slicer.org/u/e36alpine)\
**Post date:** [December 18, 2019, 5:06pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/13 "2019-12-18T17:06:26Z")

</div>

Very excited about these modules , however I must be doing something wrong. I keep getting an error when I try to fetch models. It says "Failed to fetch models from server. Make sure address is correct and retry. Not sure if my work PC is blocking me from accessing that server? I have 0.0.0.0 for the server address.  
Details:  
Traceback (most recent call last):  
File “C:/Users/en12283/AppData/Roaming/NA-MIC/Extensions-28690/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 164, in onClickFetchModels  
models = self.logic.list\_models(self.ui.modelFilterLabelLineEdit.text)  
File “C:/Users/en12283/AppData/Roaming/NA-MIC/Extensions-28690/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 692, in list\_models  
result = self.aiaaClient.model\_list(label)  
File “C:\Users\en12283\AppData\Roaming\NA-MIC\Extensions-28690\NvidiaAIAssistedAnnotation\lib\Slicer-4.11\qt-scripted-modules\NvidiaAIAAClientAPI\client\_api.py”, line 107, in model\_list  
response = AIAAUtils.http\_get\_method(self.server\_url, selector)  
File “C:\Users\en12283\AppData\Roaming\NA-MIC\Extensions-28690\NvidiaAIAssistedAnnotation\lib\Slicer-4.11\qt-scripted-modules\NvidiaAIAAClientAPI\client\_api.py”, line 402, in http\_get\_method  
conn = httplib.HTTPConnection(parsed.hostname, parsed.port)  
File “C:\Users\en12283\AppData\Local\NA-MIC\Slicer 4.11.0-2019-12-17\lib\Python\Lib\http\client.py”, line 849, in **init**  
(self.host, self.port) = self.\_get\_hostport(host, port)  
File “C:\Users\en12283\AppData\Local\NA-MIC\Slicer 4.11.0-2019-12-17\lib\Python\Lib\http\client.py”, line 881, in \_get\_hostport  
i = host.rfind(’:’)  
AttributeError: ‘NoneType’ object has no attribute ‘rfind’

I just did a Pancreatic tumor case and would love to see if the auto segment gets it close. Thanks!

---

<div class="post-metadata">

**Author:** ![JanWitowski](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/janwitowski/32/4512_2.png) [@JanWitowski](https://discourse.slicer.org/u/JanWitowski)\
**Post date:** [December 18, 2019, 5:15pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/15 "2019-12-18T17:15:58Z")

</div>

Are you running a local server? If you’re trying to access publicly available models hosted by PerkLab, leave the “NVidia AIAA server” field empty.

---

<div class="post-metadata">

**Author:** ![e36alpine](https://avatars.discourse-cdn.com/v4/letter/e/a9a28c/32.png) [@e36alpine](https://discourse.slicer.org/u/e36alpine)\
**Post date:** [December 19, 2019, 2:05pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/16 "2019-12-19T14:05:09Z")

</div>

I thought it was blank yesterday. I have no idea what I did different today but now it’s working!! Sorry for wasting your time! Thanks,Greg

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 19, 2019, 7:34pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/17 "2019-12-19T19:34:47Z")

</div>

Is it possible to train this to segment a area of bone defect (e.g Alveolar Cleft ) by looking at predefine set of standard landmarks ?

I am Zero in AI.

Can someone be able to do a small tutorial on how to train to segment out a structure ?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 19, 2019, 8:38pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/18 "2019-12-19T20:38:27Z")

</div>

The extension allows users to run trained AI models. Creating models is a non-trivial task, but usually it is not too hard, if you already have large training data set (hundreds of manually segmented cases). You can search specifically for “nvidia clara train tutorial” (or look for any deep learning image segmentation tutorial on the web and then set up Clara interface to it).

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 19, 2019, 8:48pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/19 "2019-12-19T20:48:41Z")

</div>

> [@lassoan](#):
>
> oo hard, if you already have la

i do have access to large sets of CT and CBCT of data of head and neck region. But how much of manual segmentation will be needed for e.g

1. to train on oral cancer detection in the Maxilla or Mandible ? (we get like 150-200 oral cancers per year)  
or
2. more simple segmentation like inferior alveolar nerve ? (big numbers of data)

Also is there a way to compensate for conditions with lack of data ? for example in cleft lip and palate we don’t get patients in number of hundreds… what we have is like 20 new patients per year. So in those cases is it possible to train ?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [December 19, 2019, 9:15pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/20 "2019-12-19T21:15:32Z")

</div>

For training models for segmentation, you need to have manually segmentations as input (it is not enough to have just the CT volume). The more data the better, but a few hundred should be enough. If you only have a few ten cases, it may be feasible, too, but it makes the training more difficult (you may need to tune network structure and learning parameters more carefully, do more sophisticated data augmentation, etc.).

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 19, 2019, 9:21pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/21 "2019-12-19T21:21:04Z")

</div>

Thank you. I will try with simple inferior alveolar nerve training… i think i should be able to segment about 200 alveolar nerves and try this as a experiment.

Is the segmenting is the usual process with Segment editor ?  
What format i need to save the segments/data in ?

Once i segment out 200 alveolar nerves i will try to set the clara interface to it.

[Next page](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536.md?page=2)