Does anyone know of a group working on Slicer extension for the recently released NVIDIA Clara Train SDK AI-Assisted Annotation? Our radiology department is eager to optimize our ground truth creation efforts and this looks like a promising approach. Looks like MITK already has implemented this. Would love to collaborate with anyone working on this on the Slicer side, possibly to integrate with Case Iterator.
We have not had a chance yet to evaluate Clara, but it is certainly a promising effort. It should not be too much work to put together a convenient workflow for training data generation for Clara (using Case iterator and Segment editor) and making Clara segmentation algorithm available as a Segment Editor effect. If you can spend some time with it, we would be happy to contribute.
You could start with trying to run the Clara examples from Slicer’s Python environment (just manual execution, scripting) and see if everything works as expected. We could discuss your findings in detail and set up a plan in an upcoming weekly hangout.
@fedorov mentioned that he saw a webinar about clara and that there was to be a video of it online somewhere. Maybe he or someone can post a link?
They were going to send the link to the materials at some point after the event, but they have not yet. At least I have not received it.
According to this NVIDIA page, the SDK is already integrated with Slicer 
“Clara Train SDK’s AI Assisted Annotation accelerates the annotation process by enabling application developers to integrate deep learning tools built into the SDK with existing medical imaging viewers , such as MITK, ITK-Snap, 3D Slicer or a custom-built application viewer”
The webinar actually had a slide with a diagram showing Slicer as a potential client for their AI-assisted annotation server. Apparently there’s a Slicer extension in development somewhere… That’s what prompted my reaching out 
For those interested in the webinar you can access the full recording via here. Search for “Clara”. Will have to register with NVIDIA.
Key Takeaways From The Webinar:
- Two SDK container-based, open source packages: Clara Train SDK and Clara Deploy SDK
- Introduces modular container-based approach for DL model training/inference/evaluation
- Clara Train SDK allows use of free pre-trained models based off the Medical Decathlon Challenge
- Open source API offered for creating a server-client infrastructure where annotations/segmentations are generated from the server and can be touched up in client (eg Slicer, MITK, Snap-ITK) then used for re-training/transfer learning of the pre-trained models
- Clara Deploy SDK attempts to allow end-to-end, DICOM compatible pipeline that can interface with a PACS, apply an inference and relevant post-processing then output DICOM images/renderings
I haven’t been able to test everything out but am eager to collaborate with others to use Slicer with this. I think there’s much to gain on both the Slicer and NVIDIA ends.
@alasso I’ll keep you posted on our efforts here, but will also reach out to NVIDIA to find out who their Slicer collaborators!
1 Like
Thanks for the link!
Based on the picture, this is the webcast I joined last week:
I reached out to NVIDIA and they shared the Slicer prototype extension they’re working on. Looks promising!
2 Likes
It is very promising indeed. I’ve sent a pull request to make the extension compatible with Slicer Preview Releases and submitted an issue for asking clarification about how to access/set up an annotation server.
Here’s the latest documentation for the AI-Assisted Server via NVIDIA’s Clara Train SDK.
I’ll also suggest to the NVIDIA team working on the extension to join in on the discussion here as I think they’d be happy provide as much detail as needed to facilitate this.
Thank for correcting and making it compatible
Some optimizations are possible specially related to image saving etc… before sending it to AIAA server.
But otherwise, the next step is to add Polygon Editing feature (similar to what exists in MITK Nvidia Plugin)
Curve nodes are added in recent Slicer Preview Releases that should be used for this. These new curve nodes provide interactive editing, various interpolation schemes, out-of-plane display, etc. You can also look at other segment editor effects, such as Surface cut and Draw tube to see how you can preserve editable curve points in segment tags. If you need any further help then we can continue the discussion in the extension’s repository (you can add issues for each item to discuss/develop).
This process is very involved and most potentially interested users cannot complete all the steps. Would it be possible to set up a test server that would allow people to try the extension?
We (NVIDIA) would like to… even if it is slow… but still helps user to understand about the feature.
We (as part of NVIDIA) need to clear few security checks to make it public… hence it’s delayed
2 Likes
@lassoan I spoke with one if my colleagues here and standing up an annotation server for development is something we can help with. May take a couple of days but will keep you posted. Will post both here and on repository with details.
Exciting turn around on this work!
2 Likes
Thank you… I have been referring the same extensions to start with.
One thing I couldn’t find: when user clicks label (selects label like Spleen) I would like to send a request to AIAA server and keep the list of models ready… instead of additional click to “Fetch Models”
Is there a way I can get an event that label is selected from the labelmap?
If you mean that you would like to know when the user changes the segment name (or double-clicks the color swatch and selects a terminology item) then you can add an observer to the SegmentModified event in the segmentation node. Whenever this event is invoked your observer can retrieve the segment’s name or terminology and request the appropriate model.
@lassoan I think the question is how to get the event when the segment is selected in the segment list.
@SachidanandAlle in the QuantitativeReporting module we had a similar situation where we wanted to configure actions based on segment selection. I might not remember it correctly, but I think Christian (@che85) who implemented that module didn’t find a way to get that event, and because of that had to customize Segment Editor in this class. I would’t be able to list all the reasons why that customization was needed, beyond this event handler.
To get a notification immediately when the user selects a different segment, you can use the currentSegmentIDChanged signal of the segment editor widget. If you want to check anytime what is the currently selected segment, you can get it from the segment using currentSegmentID method (or from the segment editor MRML node).
1 Like