# AI-assisted segmentation extension

**URL:** <https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536>\
**Category:** Announcements\
**Tags:** feature, deep-learning\
**Created:** [December 17, 2019, 9:03pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536 "2019-12-17T21:03:10Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![JanWitowski](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/janwitowski/32/4512_2.png) [@JanWitowski](https://discourse.slicer.org/u/JanWitowski)\
**Post date:** [December 19, 2019, 9:38pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/22 "2019-12-19T21:38:50Z")

</div>

Before you start segmentation please take a look at tutorials / user guides. If you decide to use Nvidia Clara you can find more information e.g. here: [https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v1.1/index.html](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v1.1/index.html) In short: it depends on your approach.

Also, think about the structure that you want to segment. You might want to start with a larger structure than inferior alveolar nerve – small structures tend to be more difficult to understand by AI models.

> Is the segmenting is the usual process with Segment editor ?

Yes, the process of creating the “ground truth” segmentation can be done completely with Slicer tools. Usually, you’ll want to export your segmentation masks into binary label maps (through the Segmentations module).

---

<div class="post-metadata">

**Author:** ![manjula](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/manjula/32/80981_2.png) [@manjula](https://discourse.slicer.org/u/manjula)\
**Post date:** [December 19, 2019, 9:40pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/23 "2019-12-19T21:40:41Z")

</div>

Thanks much appreciated. Will experiment with this.

---

<div class="post-metadata">

**Author:** ![lzq3315](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lzq3315/32/5518_2.png) [@lzq3315](https://discourse.slicer.org/u/lzq3315)\
**Post date:** [January 4, 2020, 6:41am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/24 "2020-01-04T06:41:04Z")

</div>

Hi ! I want to make segment for ovarion tumor but I can not find the suitable model . Could you set up one model for ovarian tumor please? Thank you very much!

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [January 4, 2020, 5:33pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/25 "2020-01-04T17:33:55Z")

</div>

You can try other models (such as those developed for brain tumors). If they don’t work then you probably need to train your own model (see [reply above](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/20)).

---

<div class="post-metadata">

**Author:** ![tbuikr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/tbuikr/32/3899_2.png) [@tbuikr](https://discourse.slicer.org/u/tbuikr)\
**Post date:** [January 20, 2020, 3:10am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/26 "2020-01-20T03:10:47Z")

</div>

@lassoan: Thank your team and NVIDIA to provide an interesting project.

Currently, we are working on object detection (i.e. tumor detection…) using faster-rcnn and plan to incorporate it into 3D Slicer and Nvidia Clara. This task is a new task and does not include in the current version of Nvidia Clara. I hope we can contribute it. However, I am newbie of Slicer and I would like to ask you some questions:

My pipeline is : Load a json file --\> Draw a rectangular bounding box (4 points) --\> Modify the location of these points --\> Overwrite to the json file (Pascal VOC format)

Could you please suggest some tutorial to implement the pipeline in the Slicer with python extension? For write json file, I found a tutorial at [https://www.slicer.org/wiki/Documentation/Nightly/ScriptRepository#Write\_annotation\_ROI\_to\_JSON\_file](https://www.slicer.org/wiki/Documentation/Nightly/ScriptRepository#Write_annotation_ROI_to_JSON_file)

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [January 20, 2020, 9:39am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/27 "2020-01-20T09:39:37Z")

</div>

Since NVidiaAIAA segment editor effect has already implemented the Clara API, I would recommend to start with modifying that effect in your own fork.

> [@tbuikr](#):
>
> Draw a rectangular bounding box (4 points)

Probably the easiest is to use 4 markup fiducial points for this. Marking points is already implemented in the AIAA effect, you can connect them using a line as it is done in [SurfaceCut](https://github.com/lassoan/SlicerSegmentEditorExtraEffects/blob/master/SegmentEditorSurfaceCut/SegmentEditorSurfaceCut.py) effect.

> [@tbuikr](#):
>
> Load a json file

Save/load json file is very easy in Python. Let us know if you have any specific questions.

---

<div class="post-metadata">

**Author:** ![caioath](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/caioath/32/9688_2.png) [@caioath](https://discourse.slicer.org/u/caioath)\
**Post date:** [January 31, 2020, 6:19pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/28 "2020-01-31T18:19:08Z")

</div>

Thank you Prof Lasso and team for this extension. I’ve just trained a model with good results and I was able to deploy it right away in Slicer.  
I would like to complete a script to perform the autosegmentation on some datasets using this model ([Extract skin](https://gist.github.com/lassoan/1673b25d8e7913cbc245b4f09ed853f9)) with nvidia aiaa extension.  
I looked for the setMRMLDefault to find out the parameters, but I’m missing something. Also I miss the input for the AIAA server URL.

I appreciate any help. Thanks again

Caio

```
# NVIDIA auto segmentation

```

segmentEditorWidget.setActiveEffectByName(“NvidiaAIAA”)  
effect = segmentEditorWidget.activeEffect()  
effect.setParameter(“SegmentationModel”, “my\_model”)  
effect.self().onApply()

Traceback (most recent call last):  
File “”, line 1, in   
AttributeError: ‘NoneType’ object has no attribute ‘setParameter’

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [January 31, 2020, 6:48pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/29 "2020-01-31T18:48:27Z")

</div>

> [@caioath](#):
>
> Thank you Prof Lasso and team for this extension. I’ve just trained a model with good results and I was able to deploy it right away in Slicer.

Great!

> [@caioath](#):
>
> I would like to complete a script to perform the autosegmentation on some datasets using this model ([Extract skin](https://gist.github.com/lassoan/1673b25d8e7913cbc245b4f09ed853f9)) with nvidia aiaa extension

Would you like to run extract skin example on an NVidia AIAA server?

> [@caioath](#):
>
> I looked for the setMRMLDefault to find out the parameters, but I’m missing something.

What parameters of what effect you are trying to find parameters of?

> [@caioath](#):
>
> I miss the input for the AIAA server URL

To use the default Slicer AIAA server, leave the URL field empty.

---

<div class="post-metadata">

**Author:** ![caioath](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/caioath/32/9688_2.png) [@caioath](https://discourse.slicer.org/u/caioath)\
**Post date:** [January 31, 2020, 7:38pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/30 "2020-01-31T19:38:50Z")

</div>

> [@lassoan](#):
>
> Would you like to run extract skin example on an NVidia AIAA server?

I would like to run the autosegmentation by script (loading volumes, creating segmentations and using the nvidia aiaa-slicer extension by code). I used extract skin example to get the code for creating segmentation etc. I’m running a local AIAA server here with my model.  
Everything is working fine with my server and model, but I would like to skip the clicking parts of it executing a script.

> [@lassoan](#):
>
> What parameters of what effect you are trying to find parameters of?

Parameters of NVIDIA AIAA extension

> [@caioath](#):
>
> segmentEditorWidget.setActiveEffectByName(“NvidiaAIAA”)  
> effect = segmentEditorWidget.activeEffect()  
> effect.setParameter(“SegmentationModel”, “my\_model”)  
> effect.self().onApply()

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [February 2, 2020, 1:01am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/31 "2020-02-02T01:01:18Z")

</div>

I’ve added an example for boundary points based AI segmentation in batch mode (without GUI):

> <https://gist.github.com/lassoan/ef30bc27a22a648ead7f82243f5cc7d5>

---

<div class="post-metadata">

**Author:** ![caioath](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/caioath/32/9688_2.png) [@caioath](https://discourse.slicer.org/u/caioath)\
**Post date:** [February 2, 2020, 1:50pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/32 "2020-02-02T13:50:03Z")

</div>

Thanks so much. I’m learning coding as I go, and I was able to complete the script for the segmentation based on yours.

Auto segmentation  
segmentEditorWidget.setActiveEffectByName(“Nvidia AIAA”)  
effect = segmentEditorWidget.activeEffect()  
serverUrl = “http://AIAA\_SERVER\_ADDRESS:PORT/v1/models”  
effect.self().ui.serverComboBox.currentText = serverUrl  
effect.self().onClickFetchModels()  
aiaaModelName = “MY\_MODEL”  
effect.self().ui.segmentationModelSelector.currentText = aiaaModelName  
effect.self().onClickSegmentation()

---

<div class="post-metadata">

**Author:** ![Dunstan](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dunstan](https://discourse.slicer.org/u/Dunstan)\
**Post date:** [June 27, 2020, 3:34pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/33 "2020-06-27T15:34:04Z")

</div>

Hi, I’m also trying to automatically segment ovarian masses.  
I have a collection of over 200 manually segmented tumor lesions of both ovaries, several of them are bilateral and partly merged. How can I train my own model using this approach? Is there any tutorial available? Many thanks for your help!

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [June 27, 2020, 3:49pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/34 "2020-06-27T15:49:11Z")

</div>

There are lots of deep learning tutorials online.

You can follow any keras or pytorch tutorial and feed your segmentations into them. Most likely it will not immediately work, but if the segmentation problem is not too hard then by tuning network layers, learning parameters, data preprocessing, you will eventually get to something usable.

If you want to have a model that runs directly in NVidia AIAA server then you can try to train a model using NVidia Clara. It is a higher-level library, which implements some commonly needed features and hides some of the low-level details, which may save you time, but also make the learning a bit harder (it may be a bit less clear what exactly happens internally).

---

<div class="post-metadata">

**Author:** ![Fernando](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fernando/32/5640_2.png) [@Fernando](https://discourse.slicer.org/u/Fernando)\
**Post date:** [June 28, 2020, 10:21am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/35 "2020-06-28T10:21:41Z")

</div>

Hi @Dunstan

[Here’s a tutorial](https://colab.research.google.com/drive/112NTL8uJXzcMw4PQbUvMQN-WHlVwQS3i?usp=sharing) you can follow. It uses [PyTorch](https://pytorch.org/) and [TorchIO](https://torchio.readthedocs.io/) to train a 3D U-Net for segmentation. There will be a TorchIO extension available in Slicer soon. I hope this helps.

---

<div class="post-metadata">

**Author:** ![ssyip](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@ssyip](https://discourse.slicer.org/u/ssyip)\
**Post date:** [July 7, 2020, 10:43pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/36 "2020-07-07T22:43:12Z")

</div>

@lassoan - We tested the AI organ segmentation extension on a bunch of public data. The extension worked very well. Awesome. However, as our research group would like to also analyze hospital / patient data, we do not feel comfortable sending our input image (even the deidentified ones) to “an external” server. Therefore, my question is “Is there any way to run the AI organ segmentation using our own hardware (e.g. GPU) w/o sending our data to an different server?” Thank you!

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [July 8, 2020, 12:40pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/37 "2020-07-08T12:40:58Z")

</div>

You can set up your own server by following instructions on [NVidia Clara website](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/index.html).

---

<div class="post-metadata">

**Author:** ![gpillajo](https://avatars.discourse-cdn.com/v4/letter/g/ecb155/32.png) [@gpillajo](https://discourse.slicer.org/u/gpillajo)\
**Post date:** [August 20, 2020, 12:24am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/38 "2020-08-20T00:24:10Z")

</div>

Dear Prof Lasso:  
I have tried nvidia AIAA module in 3d slicer, there are my results:

1. DExtr3D works fine.
2. Auto-segmentation does not work:  
Traceback (most recent call last):  
File “C:/Users/Expert Pro R2/AppData/Roaming/NA-MIC/Extensions-29276/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 361, in createAiaaSessionIfNotExists  
in\_file, session\_id = self.logic.createSession(inputVolume)  
File “C:/Users/Expert Pro R2/AppData/Roaming/NA-MIC/Extensions-29276/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 999, in createSession  
response = aiaaClient.create\_session(in\_file)  
File “C:\Users\Expert Pro R2\AppData\Roaming\NA-MIC\Extensions-29276\NvidiaAIAssistedAnnotation\lib\Slicer-4.11\qt-scripted-modules\NvidiaAIAAClientAPI\client\_api.py”, line 107, in create\_session  
raise AIAAException(AIAAError.SERVER\_ERROR, ‘Status: {}; Response: {}’.format(status, response))  
NvidiaAIAAClientAPI.client\_api.AIAAException: (3, ‘Status: 404; Response: b’\n404 Not Found\n
# Not Found
\n

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

\n’’)

What would be the solution for this?  
Kind regards

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 20, 2020, 2:26am UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/39 "2020-08-20T02:26:16Z")

</div>

Please report this to NVidia engineers at [https://github.com/NVIDIA/ai-assisted-annotation-client/issues](https://github.com/NVIDIA/ai-assisted-annotation-client/issues). Maybe we would just need to upgrade the default Slicer server ([https://github.com/NVIDIA/ai-assisted-annotation-client/issues/62](https://github.com/NVIDIA/ai-assisted-annotation-client/issues/62)).

---

<div class="post-metadata">

**Author:** ![Rodrigo\_Visconti](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/rodrigo_visconti/32/3436_2.png) [@Rodrigo\_Visconti](https://discourse.slicer.org/u/Rodrigo_Visconti)\
**Post date:** [August 22, 2020, 4:19pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/40 "2020-08-22T16:19:01Z")

</div>

There’s a way to incorporate US into the AI assisted segmentation module?

---

<div class="post-metadata">

**Author:** ![ungi](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/ungi/32/78573_2.png) [@ungi](https://discourse.slicer.org/u/ungi)\
**Post date:** [August 22, 2020, 4:53pm UTC](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536/41 "2020-08-22T16:53:59Z")

</div>

Ultrasound is very different from CT and MRI. There is code for training and deploying ultrasound segmentation in this repository: [https://github.com/SlicerIGT/aigt](https://github.com/SlicerIGT/aigt)  
This repository is not very organized, but if you tell me more what you need to do, I can point you to more specific examples.

[Previous page](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536.md?page=1)

[Next page](https://discourse.slicer.org/t/ai-assisted-segmentation-extension/9536.md?page=3)