transfer scene files from 3DSlicer to Unity3D

So let’s imagine I have a segmentation scene file and I want to put these segmentations in unity3D from slicer without changing the characteristics of the segmentations. So how do I do this? because concretely I think it is possible but I cannot find exactly what to do.

You can use use SlicerOpenAnatomy extension’s OpenAnatomy export module, which can export a segmentation as a set of low-poly models by a single click of a button in OBJ or glTF format.

Once you have the exported OBJ files, if you drop them into the Assets folder (or a subdirectory of the Assets folder if you want to keep things more organized) of your Unity project, Unity will automatically import them as assets. Then you can add them to your scene within Unity. It’s been a while since I’ve done it, but I recall the import process working smoothly. You need both the .obj file and the associated .mtl file to bring along the associated color you have from Slicer. The obj file holds the mesh and the mtl file holds the material properties, which includes color.

1 Like

About the scaling, the Slicer export has units of mm, and I think the Unity default might interpret numbers as meters, but scaling things in Unity is very easy, and if your objects are 1000x too big, just scale them down. For what I was doing, only relative sizes mattered and all meshes were coming from Slicer, so I didn’t end up having to change anything.

1 Like

If you want to avoid changing the segmentation to a mesh model and display the segmentation as a volume in unity - there are also several options to do this.

Basically speaking - you can export the segmentation as a sequence of 2D slices and import them into a 3D texture. The same way as volumetric clouds are rendered.

If you want to use DICOM you might need to write an import plugin that will read and convert the files to a texture sequence or a 3D texture, but if you just want simple visualisation you can export a sequence of images using the Screen Capture module and use the sequence with volume renderers like this one.

There were a couple of very nice looking prototypes to do volume rendering of medical images in UE4 that appeared to show better performance - so Unity is not the only game engine that can give you this functionality.

Edit: there is a Simple DICOM Loader in the Unity Asset Store.

Thank you everyone, you have helped me a lot.
I have another question. How do I PRECISELY transfer the volume rendering of the sections I have on 3DSlicer to Unity3D without changing the orientation, the scale etc … PLEASE ??

yes I have a complete Proyecto fist you need to create a 3D model and save it as .ply file and go to this website so you don’t need Blender to convert to fix the file
this is because is the format that unity needs.
https://products.aspose.app/3d/conversion/ply-to-fbx

after converting your 3d model format you can drag it in unity I guess you want to do an AR or something like it, if that’s the case feel free to ask me

thanks , Basically I have MRI sections that I transferred from 3Dslicer to unity3D and now I want to have the volume rendering of these images in unity3D, how do I do that?

image

they are in .PNG format

pleas open Slicer and see the name of your file you can see these by going to Data in Modules

after that pleas change the name of your file mi file name is aseg.presurf-IXI061-A
so try this:
You can open python code by crl+3

nodo='aseg.presurf-IXI061-A'

sliceLogic = slicer.app.layoutManager().sliceWidget('Red').sliceLogic()
compositeNode = sliceLogic.GetSliceCompositeNode()
compositeNode.SetLinkedControl(1)
#link the slice control
slicer.util.getNode('vtkMRMLSliceNodeRed').SetUseLabelOutline(1);
#Thickness line 
labelMapNode=getNode(nodo)
labelMapNode.GetDisplayNode().SetSliceIntersectionThickness(1)

#####3d

seg=slicer.mrmlScene.AddNewNodeByClass('vtkMRMLSegmentationNode')
slicer.modules.segmentations.logic().ImportLabelmapToSegmentationNode(getNode(nodo), seg)
seg.CreateClosedSurfaceRepresentation()

After this code you will have a 3d model
image

You will have to go to segment edit click segmentation
after click go to this menu
image
and you will have you 3d protect in stl file

Implementing medical image viewer in Unity is a huge pain. You essentially need to redevelop everything from scratch. You find a few very basic volume rendering packages that you should be able to set up a simple demo with it, but they are nowhere near to what real users would expect as a minimum requirement for a clinical software.

I would only recommend Unity for very special use cases, for example for building simulation-based training applications, where the emphasis is on fluid computer-game-like immersive experience and you don’t need clinical-grade visualization and tools.

1 Like

I have this error

>>> #####3d
>>> seg=slicer.mrmlScene.AddNewNodeByClass('vtkMRMLSegmentationNode')
>>>slicer.modules.segmentations.logic().ImportLabelmapToSegmentationNode(getNode(nodo), seg)
Traceback (most recent call last):
File "<console>", line 1, in <module>
TypeError: arguments do not match any overloaded methods
>>> seg.CreateClosedSurfaceRepresentation()
True

First argument of ImportLabelmapToSegmentationNode must be a labelmap volume. See more examples here.

Yes I know but I have to do this on unity3D even if the result is not perfect but I have to show something to my professor

Actually I already have my images in unity3D so now from unity3D I have to do the volume rendering, thanks to a plugin I guess

Or how use the surface rendering instead of volume rendering in unity3D ?

You would load the image into Slicer, use Segment Editor to segment all the structures that you are interested in, and export the segmentation to OBJ files (see step-by-step instructions here). You can import the OBJ files into Unity.

hum okey thanks a lot and if i just want to export the top right file to unity, what do i do?

It looks like that is showing a volume rendering, which is different than a segmentation. If you only care about capturing that outer visible surface, then the quickest approach will be to create a segment via thresholding. Open the segment editor module, create a new segment, and then choose the threshold tool. Choose a threshold minimum value which is larger than the voxel values for air and a threshold maximum value which is greater than the voxel values at the skin surface, and then click apply. That should basically create a segment which contains all voxels which are non-air. If you click “Show 3D” on the segmentation and then hide the volume rendering, you should be able to see your segment in the 3D view (where your volume rendering is now). That segmentation can be exported to OBJ files as described above and then imported into Unity.

If you actually only need the skin surface, the segment you generate this way may contain a lot of unnecessary internal structure (which you can get rid of by using other segmentation tools), but this is the quickest way I can think of to get an approximation of what you are asking for.

1 Like