I am trying to get the VR Module to run since a couple of hours now.
I tried with Quest 3 + Quest Controller.
I tried with Varjo VR3 and Vive Controllers.
I run into serious interaction problems.
Especially if I open the menu and try to select the cutting/probe tool. Using Quest 3 (in both OpenVR/OpenXR) it is not possible to select anything even though I manage to at least toggle thorugh the options.
With the Varjo XR I can select the tools with quite some hustle, but the way the interaction works (cutting plane) it is almost useless as I am forced to cut in a way that is perpendicular to my perspective.
Slicer 3D often shuts down without any error code and it is not clear why. (I reinstalled etc.)
I really like the looks of the module and to have VR integrated in Slicer 3D and being able to use all the modules and render the results in VR is really nice.
Did someone here manage to use it effectively? I have other software to render medical images and it works way better. I very much like the idea of open source however…
The problem is that SlicerVR is undergoing a complete backend reimplementation, and I think 5.6.2 contains a VTK that does not fully support the new one (I haven’t tested VR for a while as I am waiting for the announcement that the OpenXR backend fully works, and I intend to try it then - my interest is development and not usage so I can afford such wait).
I suggest the following two options:
5.4.0: It was a month after the last summer project week, when I remember that everything seemed to work again after a long period when they didn’t. Use OpenVR (install Steam and SteamVR first)
The latest 5.7.0: Yesterday at the Slicer Project Week meeting the Kitware developers said that other than the complex gestures (e.g. two-handed world manipulation - a feature that I consider essential but others may not) and certain minor things the OpenXR backend works well.
Sometimes it worked a little bit - but nowhere where it needs to be to be anything more than a “party trick”. So moving myself/my camera consistently works. Choosing tools from the menu was -at best- iffy. Using the tools on the rendered medical image was close to impossible. In the current state the plug in is more frustrating than anything else. For sure you will not want to use it. Especially if you have alternatives at hand.
I tried different versions of slicer (5.4.0, 5.6.2 (?!), 5.7.0).
I think the UI is not thought out and is not implemented correctly. I tried for hours to use it…
This is sad as Slicer offers a lot of nice plug ins and would be awesome as a medical image VR viewer.
Another thing I noticed is that if you combine for example a CT with a segmented organ/pathology and volumetrically render it in Slicer it will show on the desktop but in VR will only show the CT.
Looking at the commit history of the SlicerVR plugin, I can see you have most of the commits.
Your input on the future work of this plugin and device support (Meta Quest 3 or others) will be useful.
As I said this extension is not yet finished. The tools that you talk about are probably from the original VTK tech demo. We’ll need to add a configurable in-VR widget. I did some experiments a while ago
But since the extension has been undergoing profound changes, such DIY solutions are not worth maintaining, as everything they are based on is constantly changing. Unfortunately these efforts are quite slow (as I understand the OpenXR rework started several years ago and has not been finalized still).
Well this is why I think that this extension is what we need to improve, because the “alternatives” are Unity or Unreal Engine based solutions, which are fine for static scenes (great for education, you can see many examples published), but not really for patient-specific use cases where you need to work on various new patients every day.
I think this is inaccurate. Maybe you configure the volume rendering before activating VR, and the new VR view is not added to the volume rendering display node? Try setting up rendering first and then show it in VR.
I understand your frustration, but please keep in mind that you are talking about an open-source, entirely free piece of software, that can be maintained and developed either by funded projects (which are almost never about infrastructure like this extension but about driving use cases, and most of the time larger infrastructural work cannot be included), or in the free time of developers. If you can provide either of the two, the community will welcome your contribution.
I have been using MedicalHolodeck, which allows for pretty seamless display of volume rendering and segmentation data. Also able to view 4D/motion CT and MRI. The manual segmentation tools need work (Elucis is good for this), but AI segmentation with Totalsegmentator is integrated. This is a paid application though, and an open-source alternative would be great. I’ve tried SlicerVR on Quest 2, 3, and Index but it’s really difficult to navigate and manipulate images. Haven’t had issues showing volume renderings but the lighting can be strange, even with various combinations of dual/back lighting turned on and off.
Thanks for the feedback. We have an amazingly powerful and versatile foundation in Slicer for augmented and virtual reality that can easily overtake all existing commercial solutions.
However, right now there are no end-user workflows built on this foundation. Maybe because AI took almost all attention and funding away from AR/VR? Even Apple’s headset failed to generate any lasting excitement in the field. Or maybe the main challenge is to find a widely usable impactful clinical application of AR/VR.
Anyway, AR/VR applications in Slicer are evolving at a slow pace. If you can contribute with developer time or funding then a lot of progress could be achieved quite quickly.
I really appreciate all the work you and the other contributors have put into 3D Slicer and all of the extensions. I would love to contribute, but unfortunately I don’t know all that much about programming. I’m learning as I go, so hopefully I can help at some point.
@cpinter I am aware that slicer is an opensource project. When I describe the state of the vr plugin / addon I am not bashing it. It is just barely useable at the moment. I will think about contributing to the project as I see a lot of potential.
If the VR view is included, then I guess I or some other developer will need to try it and see why it does not show up. The best probably would be adding a new issue in the SlicerVR GitHub repository.
