SlicerVirtualReality issues

Trying to view a volume rendering in VR using SlicerVirtualReality plugin. Using an occulus quest 2 using occulus link cable to a laptop. I load the DICOM data set, and set up the volume rendering so I see it on the laptop.

  1. If I hit the VR icon in slicer 3D then put on my headset while it’s in SteamVR, I just get a blank empty space

  2. If I hit the VR icon from inside the steam Vr space (by viewing desktop) I get this error:
    “the action manifest for slicerapp-real was not found”

I don’t think any of us has tried the Oculus Quest 2 with SlicerVR yet, as it is not supported by OpenVR, and SlicerVR is not yet updated to OpenXR.

Please copy the json files from here to the folder from where you start Slicer.

Like this? This didn’t have any effect.

image

Also, can’t you run OpenVR applications on the quest 2 as long as you execute them in SteamVR and use the occuluslink?

I don’t know if occuluslink works (let us know if you get it working). But cloudxr works with Slicer 4.11 and SlicerVR, either from a local machine or a cloud hosted machine over wifi.

Are you positive you mean 4.11 and not 4.13? Just curious to know what did you use successfully CloudXR with (and 4.11 is pre-2021/02).

I used it with build Slicer not installed, but yes as far as I know this should work for the manifest errors.

About oculus link I also cannot give you information. I personally haven’t used wiress headsets with SlicerVR yet.

Yes, definitely. CloudXR works with OpenVR (not yet OpenXR). The stock binary Slicer-4.11.20210226 with the corresponding SlicerVR extension just works (if everything else is set up right, like the network ports, steam account, occulus client, etc).

I also tested 4.13 with CloudXR and I got an error in the client about. controller manifest that maybe could be fixed but I didn’t look more closely since 4.11 suits my purposes for now.

1 Like

Got the Oculus Quest 2 at least halfway working in Slicer 4.11, impressive …
Prerequisites:

  • Windows 10
  • install Steam
  • install SteamVR
  • install Oculus PC app
  • install Slicer 4.11
  • install Slicer VR extension

Session:

  • Put on the Quest 2 headset
  • Adjust Guardian to local Guardian (circle around you)
  • In Quest Home → touch settings
  • In Settings, touch Airlink and connect to your graphics workstation
  • (For airlink use a 5Ghz WLAN access, connect the PC to router via LAN cable)
  • First run: In Airlink mode → touch the desktop symbol and start Steam VR
  • Subsequent runs: → Touch SteamVR
  • In SteamVR, you will see an empty giant space with ground and stars
  • Press left controller menu button → menu appears → touch desktop symbol low left in the menu → normal desktop screen opens
  • Start and adjust Slicer 4.11, load a volume and segmentation
  • Swich to Slicer VR extension
  • In Slicer top menu, press the headset symbol

The segmentation appears floating in empty space, can be moved forwards and backwards (only) with the right controller’s thumb joystick back and forth, and can be rotated with gestures as soon as both trigger buttons are pressed.

Is this what to be expected or is there more in other headsets?

1 Like

Thanks for the report @rbumm. It sounds like Airlink behaves basically the same as CloudXR. I’m not sure if CloudXR offers better performance but Airlink sounds a bit easier. Regarding other headsets, I think they offer similar features.

There should be better features in the Slicer 5 version using OpenXR, but that seems to be taking a while to sort out so it’s not available yet. Maybe someone can comment on any progress recently.

1 Like

Thanks for the update @rbumm , quite useful!

We are currently in an “almost there” state with SlicerVR on many fronts.

  • OpenXR support can be expected to be finalized soon in terms of the new Kitware-Robarts project (at least as far as I know using OpenXR is in the plans for this project - they want HoloLens2 support)
  • In-VR widget exposing many Slicer features is on the way. The widget works, the laser pointer basically works. We currently struggle with that the texture update that shows in the normal 3D view doesn’t show up in VR. Anyway, it’s very close now.
  • A VR “training course” as far as I know is also on the way. There will be some kind of OR with objects and little goals that walk you through the basics of how to use the controller, how to navigate in the scene, etc
  • I’m personally planning many small additional features, but these will come slowly after we successfully add the in-VR widget with the laser
  • Unfortunately there are several small but important issues with the latest SlicerVR after the recent VTK update, and maybe due to other updates. For example, volume rendering does not show up in VR at all. Or there are errors in the log that even slow down rendering. Some button events are not caught. Maybe these are the most important ones.

I hope this answers your question!

Can you please describe your use case in VR? Also what is it that you would ideally do in VR. Thanks!

Thank you @cpinter.

I’m thrilled to see 3D Slicer supporting VR at all, in fact, it would be great to have that volume rendering available as well as a “laser”, which I do not yet find in the Oculus Quest 2.

My personal use cases would be to plan thoracoscopic access with a shrunk lung/tumor segmentation, provide anatomical teaching to students or use it for patient education. Possibilities seem to be endless as soon as one could overlay 3D Slicer segmentations and human bodies in mixed reality.
Saw a similar project at Brainlab’s headquarters using → Magic Leap One.

Driving and manipulating simulated “fibreoptics” and “instruments” into a (maybe bloated) body segmentation, being able to manipulate them in VR, and seeing the result on a virtual screen in VR space is something that immediately came into my mind when using Slicer VR the first time.

Please let me know if I can support this project or if you need further tests.

1 Like

That’s an awesome topic.
@rbumm I think you can move completely however you need to grab the model and sometimes it’s difficult.

1 Like

Very nice @Eserval!

Could you share briefly how you segmented the vessels and how you recorded the VR screen?

For vessels segmentation I use a workflow that creates masks and new masks inside then:

1- Create a lung mask including the mediastinal part of vessels (I use the Lung segment module)

2- Create a new mask using the threshold tool for the vessels

3- I perform some fine work on 3D to exclude some “trash” points

4- Use grow from seeds to separate the artery and vein.

To record the VR screen I use the OBS Studio. I just enable the screen playback in Steam VR options and recording with OBS.

I’m a thoracic surgeon and we are using routinely the 3D slicer in our lung resection cases and research:

  • Our dream is to develop a more autonomous process for lung vessel segmentation. I think that it’s not a very hard job for a developer, however for us, surgeons, It’s quite impossible.

Here at our University (University of Sao Paulo - Brazil), we are developing many projects involving thoracic surgery and 3D reconstruction (including VR, preoperative planning, and robotic surgery…). We are completely open to the exchange of experiences and collaborations.

1 Like

I do not want to push that topic up too often, but are you referring to the Lung CT Segmenter as part of the Lung CT Analyzer extension

image

of which I am one of the principal authors? If yes this could be the start of direct cooperation because your goal is exactly my goal and I am a thoracic surgeon with a bit of programming knowledge. We could probably take the vessel segmentation workflow ideas to another topic then …

BTW impressive segmentation

@cpinter

I found the scissor effect in the free SteamVR application → “The Lab” → “Human body scan” very impressive and helpful to analyze an obvious volume rendering.

Hi Rudolf.

I think the functionality you show on the video is already on Slicer by PRISM Volume Rendering extension…

Please look at this:
https://githubcomets-vis-interactiveslicerprismrendering.readthedocs.io/en/latest/tutorials.html#plane-intersecting

You would just need to code a good VR handles interface to control the intersection planes I think

Mauro

1 Like

Thats right Rudolf! By the way thanks for your excelent job.
I think we can work together!

1 Like

@rbumm Thanks for pointing it out! We have ROI cropping in Slicer that is more or less the same except that instead of a plane you need to define a box, but this plane cutting seems quite easy to use for sure.