Trying to view a volume rendering in VR using SlicerVirtualReality plugin. Using an occulus quest 2 using occulus link cable to a laptop. I load the DICOM data set, and set up the volume rendering so I see it on the laptop.
If I hit the VR icon in slicer 3D then put on my headset while it’s in SteamVR, I just get a blank empty space
If I hit the VR icon from inside the steam Vr space (by viewing desktop) I get this error:
“the action manifest for slicerapp-real was not found”
I don’t think any of us has tried the Oculus Quest 2 with SlicerVR yet, as it is not supported by OpenVR, and SlicerVR is not yet updated to OpenXR.
Please copy the json files from here to the folder from where you start Slicer.
Like this? This didn’t have any effect.
Also, can’t you run OpenVR applications on the quest 2 as long as you execute them in SteamVR and use the occuluslink?
I don’t know if occuluslink works (let us know if you get it working). But cloudxr works with Slicer 4.11 and SlicerVR, either from a local machine or a cloud hosted machine over wifi.
Are you positive you mean 4.11 and not 4.13? Just curious to know what did you use successfully CloudXR with (and 4.11 is pre-2021/02).
I used it with build Slicer not installed, but yes as far as I know this should work for the manifest errors.
About oculus link I also cannot give you information. I personally haven’t used wiress headsets with SlicerVR yet.
Yes, definitely. CloudXR works with OpenVR (not yet OpenXR). The stock binary Slicer-4.11.20210226 with the corresponding SlicerVR extension just works (if everything else is set up right, like the network ports, steam account, occulus client, etc).
I also tested 4.13 with CloudXR and I got an error in the client about. controller manifest that maybe could be fixed but I didn’t look more closely since 4.11 suits my purposes for now.
Got the Oculus Quest 2 at least halfway working in Slicer 4.11, impressive …
Prerequisites:
Session:
The segmentation appears floating in empty space, can be moved forwards and backwards (only) with the right controller’s thumb joystick back and forth, and can be rotated with gestures as soon as both trigger buttons are pressed.
Is this what to be expected or is there more in other headsets?
Thanks for the report @rbumm. It sounds like Airlink behaves basically the same as CloudXR. I’m not sure if CloudXR offers better performance but Airlink sounds a bit easier. Regarding other headsets, I think they offer similar features.
There should be better features in the Slicer 5 version using OpenXR, but that seems to be taking a while to sort out so it’s not available yet. Maybe someone can comment on any progress recently.
Thanks for the update @rbumm , quite useful!
We are currently in an “almost there” state with SlicerVR on many fronts.
I hope this answers your question!
Can you please describe your use case in VR? Also what is it that you would ideally do in VR. Thanks!
Thank you @cpinter.
I’m thrilled to see 3D Slicer supporting VR at all, in fact, it would be great to have that volume rendering available as well as a “laser”, which I do not yet find in the Oculus Quest 2.
My personal use cases would be to plan thoracoscopic access with a shrunk lung/tumor segmentation, provide anatomical teaching to students or use it for patient education. Possibilities seem to be endless as soon as one could overlay 3D Slicer segmentations and human bodies in mixed reality.
Saw a similar project at Brainlab’s headquarters using → Magic Leap One.
Driving and manipulating simulated “fibreoptics” and “instruments” into a (maybe bloated) body segmentation, being able to manipulate them in VR, and seeing the result on a virtual screen in VR space is something that immediately came into my mind when using Slicer VR the first time.
Please let me know if I can support this project or if you need further tests.
That’s an awesome topic.
@rbumm I think you can move completely however you need to grab the model and sometimes it’s difficult.
Very nice @Eserval!
Could you share briefly how you segmented the vessels and how you recorded the VR screen?
For vessels segmentation I use a workflow that creates masks and new masks inside then:
1- Create a lung mask including the mediastinal part of vessels (I use the Lung segment module)
2- Create a new mask using the threshold tool for the vessels
3- I perform some fine work on 3D to exclude some “trash” points
4- Use grow from seeds to separate the artery and vein.
To record the VR screen I use the OBS Studio. I just enable the screen playback in Steam VR options and recording with OBS.
I’m a thoracic surgeon and we are using routinely the 3D slicer in our lung resection cases and research:
Here at our University (University of Sao Paulo - Brazil), we are developing many projects involving thoracic surgery and 3D reconstruction (including VR, preoperative planning, and robotic surgery…). We are completely open to the exchange of experiences and collaborations.
I do not want to push that topic up too often, but are you referring to the Lung CT Segmenter as part of the Lung CT Analyzer extension
of which I am one of the principal authors? If yes this could be the start of direct cooperation because your goal is exactly my goal and I am a thoracic surgeon with a bit of programming knowledge. We could probably take the vessel segmentation workflow ideas to another topic then …
BTW impressive segmentation
I found the scissor effect in the free SteamVR application → “The Lab” → “Human body scan” very impressive and helpful to analyze an obvious volume rendering.
Hi Rudolf.
I think the functionality you show on the video is already on Slicer by PRISM Volume Rendering extension…
Please look at this:
https://githubcomets-vis-interactiveslicerprismrendering.readthedocs.io/en/latest/tutorials.html#plane-intersecting
You would just need to code a good VR handles interface to control the intersection planes I think
Mauro
Thats right Rudolf! By the way thanks for your excelent job.
I think we can work together!
@rbumm Thanks for pointing it out! We have ROI cropping in Slicer that is more or less the same except that instead of a plane you need to define a box, but this plane cutting seems quite easy to use for sure.
