New Module: Remote VISTA3D server for CT segmentation

Goal
Provide access to MONAI-based foundation models to users running 3D Slicer on low-end (e.g., no GPU) machines by launching those AI methods on freely available (albeit limited total usage) high-end servers.

Overview
I am proposing to offer a set of Slicer Extensions that will allow Slicer users to call AI methods built using MONAI and running on NVIDIA servers.

Access to the servers requires registration which is free, to get an API key with which 1,000 or more images can be processed for free. Research users can also register as developers (also for free) to get unlimited local access.

The first Extension will allow Slicer users (with a free API key) to process images using the MONAI VISTA-3D segmentation (remote) running on NVIDIA servers. Via this Extension, Slicer users will be able to perform large / fast CT image AI segmentations on low-end (e.g., no GPU) machines.

TLDR:
Try the VISTA-3D module running on NVIDIA’s GPU Cloud: vista-3d Model by NVIDIA | NVIDIA NIM

I am proposing to make that service callable using a simple Slicer extension, to enable remote VISTA-3D processing of data from within 3D Slicer.

Details
NVIDIA is defined NIMs (NVIDIA Inference Microservices) as optimized containers for portable, scalable AI. These are nominally offered on NV AI Enterprise / GPU Cloud servers as callable methods. Anyone can register for free and get 1,000 to 5,000 free credits (depending on email domain used for registration, with 1 credit used for each image processed). Additionally, anyone can register for free to become an NVIDIA Developer, and then they can download NIMs for free for research purposes - enabling unlimited data processing, albeit using local GPU resources. NIMs can also run on AWS and Azure servers.

NVIDIA AI Enterprise / GPU-Cloud servers use high-end GPUs (e.g., H100s) so via NIMS running on these servers it is possible to evaluate very large AI models and very large images very rapidly.

Initially I propose to developed a "MONAI VISTA-3D segmentation (remote) Extension. For details on VISTA-3D, see the online demo:

Required Inputs

  • Image currently loaded in Slicer that the user wants to process.
  • NVIDIA GPU Cloud API Key (available with free registration)

Output

  • Segmentation results as a labelmap using same categories as TotalSegmentator

Optional Inputs

  • Seeds indicating object and not-object locations for arbitrary object segmentation
  • List of specific anatomic structures to be segmented (of 120+ supported)
3 Likes

Sounds great @aylward - any chance you or others might want to work on this as part of Project Week?

Definitely! I will attend today’s Project Week meeting, and attend Project Week (probably virtually).

Open to suggestions and help as always!