Are you positive you mean 4.11 and not 4.13? Just curious to know what did you use successfully CloudXR with (and 4.11 is pre-2021/02).
I used it with build Slicer not installed, but yes as far as I know this should work for the manifest errors.
About oculus link I also cannot give you information. I personally haven’t used wiress headsets with SlicerVR yet.
Yes, definitely. CloudXR works with OpenVR (not yet OpenXR). The stock binary Slicer-4.11.20210226 with the corresponding SlicerVR extension just works (if everything else is set up right, like the network ports, steam account, occulus client, etc).
I also tested 4.13 with CloudXR and I got an error in the client about. controller manifest that maybe could be fixed but I didn’t look more closely since 4.11 suits my purposes for now.
1 Like
Got the Oculus Quest 2 at least halfway working in Slicer 4.11, impressive …
Prerequisites:
- Windows 10
- install Steam
- install SteamVR
- install Oculus PC app
- install Slicer 4.11
- install Slicer VR extension
Session:
- Put on the Quest 2 headset
- Adjust Guardian to local Guardian (circle around you)
- In Quest Home → touch settings
- In Settings, touch Airlink and connect to your graphics workstation
- (For airlink use a 5Ghz WLAN access, connect the PC to router via LAN cable)
- First run: In Airlink mode → touch the desktop symbol and start Steam VR
- Subsequent runs: → Touch SteamVR
- In SteamVR, you will see an empty giant space with ground and stars
- Press left controller menu button → menu appears → touch desktop symbol low left in the menu → normal desktop screen opens
- Start and adjust Slicer 4.11, load a volume and segmentation
- Swich to Slicer VR extension
- In Slicer top menu, press the headset symbol
The segmentation appears floating in empty space, can be moved forwards and backwards (only) with the right controller’s thumb joystick back and forth, and can be rotated with gestures as soon as both trigger buttons are pressed.
Is this what to be expected or is there more in other headsets?
2 Likes
Thanks for the report @rbumm. It sounds like Airlink behaves basically the same as CloudXR. I’m not sure if CloudXR offers better performance but Airlink sounds a bit easier. Regarding other headsets, I think they offer similar features.
There should be better features in the Slicer 5 version using OpenXR, but that seems to be taking a while to sort out so it’s not available yet. Maybe someone can comment on any progress recently.
1 Like
Thanks for the update @rbumm , quite useful!
We are currently in an “almost there” state with SlicerVR on many fronts.
- OpenXR support can be expected to be finalized soon in terms of the new Kitware-Robarts project (at least as far as I know using OpenXR is in the plans for this project - they want HoloLens2 support)
- In-VR widget exposing many Slicer features is on the way. The widget works, the laser pointer basically works. We currently struggle with that the texture update that shows in the normal 3D view doesn’t show up in VR. Anyway, it’s very close now.
- A VR “training course” as far as I know is also on the way. There will be some kind of OR with objects and little goals that walk you through the basics of how to use the controller, how to navigate in the scene, etc
- I’m personally planning many small additional features, but these will come slowly after we successfully add the in-VR widget with the laser
- Unfortunately there are several small but important issues with the latest SlicerVR after the recent VTK update, and maybe due to other updates. For example, volume rendering does not show up in VR at all. Or there are errors in the log that even slow down rendering. Some button events are not caught. Maybe these are the most important ones.
I hope this answers your question!
Can you please describe your use case in VR? Also what is it that you would ideally do in VR. Thanks!
Thank you @cpinter.
I’m thrilled to see 3D Slicer supporting VR at all, in fact, it would be great to have that volume rendering available as well as a “laser”, which I do not yet find in the Oculus Quest 2.
My personal use cases would be to plan thoracoscopic access with a shrunk lung/tumor segmentation, provide anatomical teaching to students or use it for patient education. Possibilities seem to be endless as soon as one could overlay 3D Slicer segmentations and human bodies in mixed reality.
Saw a similar project at Brainlab’s headquarters using → Magic Leap One.
Driving and manipulating simulated “fibreoptics” and “instruments” into a (maybe bloated) body segmentation, being able to manipulate them in VR, and seeing the result on a virtual screen in VR space is something that immediately came into my mind when using Slicer VR the first time.
Please let me know if I can support this project or if you need further tests.
1 Like
That’s an awesome topic.
@rbumm I think you can move completely however you need to grab the model and sometimes it’s difficult.
1 Like
Very nice @Eserval!
Could you share briefly how you segmented the vessels and how you recorded the VR screen?
For vessels segmentation I use a workflow that creates masks and new masks inside then:
1- Create a lung mask including the mediastinal part of vessels (I use the Lung segment module)
2- Create a new mask using the threshold tool for the vessels
3- I perform some fine work on 3D to exclude some “trash” points
4- Use grow from seeds to separate the artery and vein.
To record the VR screen I use the OBS Studio. I just enable the screen playback in Steam VR options and recording with OBS.
I’m a thoracic surgeon and we are using routinely the 3D slicer in our lung resection cases and research:
- Our dream is to develop a more autonomous process for lung vessel segmentation. I think that it’s not a very hard job for a developer, however for us, surgeons, It’s quite impossible.
Here at our University (University of Sao Paulo - Brazil), we are developing many projects involving thoracic surgery and 3D reconstruction (including VR, preoperative planning, and robotic surgery…). We are completely open to the exchange of experiences and collaborations.
1 Like
I do not want to push that topic up too often, but are you referring to the Lung CT Segmenter as part of the Lung CT Analyzer extension
of which I am one of the principal authors? If yes this could be the start of direct cooperation because your goal is exactly my goal and I am a thoracic surgeon with a bit of programming knowledge. We could probably take the vessel segmentation workflow ideas to another topic then …
BTW impressive segmentation
I found the scissor effect in the free SteamVR application → “The Lab” → “Human body scan” very impressive and helpful to analyze an obvious volume rendering.
Hi Rudolf.
I think the functionality you show on the video is already on Slicer by PRISM Volume Rendering extension…
Please look at this:
https://githubcomets-vis-interactiveslicerprismrendering.readthedocs.io/en/latest/tutorials.html#plane-intersecting
You would just need to code a good VR handles interface to control the intersection planes I think
Mauro
1 Like
Thats right Rudolf! By the way thanks for your excelent job.
I think we can work together!
1 Like
@rbumm Thanks for pointing it out! We have ROI cropping in Slicer that is more or less the same except that instead of a plane you need to define a box, but this plane cutting seems quite easy to use for sure.
Yes, and these “good VR handles” are the critical part 
Because we have everything we need in Slicer, but exposing it in a UI-less environment is the challenge. This is why we have eben trying to make time (and find funding) for such developments in the last three years. Hopefully soon at least the basics will be available!
FYI you don’t need Prism to do box cropping. What you need Prism for are the custom GPU shaders that for example allow sphere cropping, and a lot more things too.
I’m doing the same thing, except using occulus link (via cable) instead of airlink. I assume that has identical functionality other than how the link is physically created.
When I get to your last step, I am floating in the empty space, except my segmentation is nowhere to be found. I’ve pressed the joysticks back and forth but I never see anything except empty black space.
Ran into a similar problem when I tried to open a volume rendering - they seem not to be functional in VR at the moment. After doing a thresholded bone segmentation you need to activate it in 3D view (segment editor), then switch to Slicer VR and press the VR headset symbol - this should work in Slicer 4.11 as of May 10th 22.
Volume rendering not working: Please see my comment above for the current state of SlicerVR.
If you use a Slicer Preview version older than mid-February then volume rendering will work. It was broken with the update of VTK under Slicer on February 24 (if I remember the day correctly).
1 Like
I just tried the 2-2-22 version of 4.13.0 and I still can’t visualize a volume render in slicerVR.