3DSlicer VR and Unity

Hey there,
I’m pretty new in VR and development, maybe you can help me.
Well, i want to create a crossSection plane, which i can move with my controllers, so i can see the inside of an object, like in this video

my problem is, i really don’t know how to do this, i tried for many hours but it didn’t work.
I work with unity, windows mixed reality and looking for a solution. Is there a possibility to connect the 3D Slicer with unity and VR?

Would be so glad to hear from you, thanks a lot!

Slicer can display anything that you see in the 3D viewer in a virtual reality headset directly, using SteamVR, without Unity. Unity is really limited when it comes to medical image visualization: you have to work hard to get such basic features as reslicing volumes, volume rendering, and visualization of meshes that consists of more than a few hundred thousand points.

We only use Unity in connection with Slicer when we run standalone applications on Microsoft HoloLens - in that case Slicer sends transforms, images, and surface meshes through OpenIGTLink. However, probably this is not what you are looking for.

Currently, we find that moving your head inside organs works very well as a “dynamic clipping plane”, but of course having an additional independent cutting can be very useful.

Slicer’s virtual reality extension has already a built-in cutting plane feature. You can change controller mode by clicking one of the buttons on the controller then tilting the controller to choose mode. However, this is still experimental (you can only clip meshes, they are only clipped in the virtual reality view, etc.) but it’s there, so you can try it and maybe improve it.

Thanks a lot for your quick reply.

That sounds great, exactly the thing i was looking for.

Is it possible to see the example scene „slicer 4minute“ in VR?

And do you know if i can work with VTK datasets as well?

You can work with any data sets, and not just surface rendering but also direct volume rendering, in 2D, 3D, or 4D. Anything that you see in the 3D view you can show in the headset by a single click.

can I connect the hololens2 in slicer?

There have been a number of projects using HoloLens as a renderer for models created in Slicer, sending transforms and images interactively from Slicer to HoloLens, but I’m not aware of any mature extension and HoloLens client that would be ready to use.

Most groups that I know of scaled back on HoloLens projects after their initial enthusiasm, because while the device is awesome, it is still just not good enough for many clinical use cases. Main problems are convergence/accommodation mismatch at arm-length distance, tracking instability, limited field of view, difficult access to data streams, and limited on-board computational resources. While I heard that HoloLens2 has greatly improved ergonomics, most of the issues still remain the same.

1 Like

Thank you for your advice