3D Slicer Virtual Reality Experience with Meta Quest 3

Support
1

I’m writing to share my initial experience using the 3D Slicer VR extension with a Meta Quest 3 on Windows 11 and to ask for some guidance.

My Setup:

  • Headset: Meta Quest 3

  • PC OS: Windows 11

  • Connection Methods Tested: Meta Link (Air Link) and SteamVR

Observations:

  1. Meta Link (Wireless): The connection was laggy and unstable. I attribute this to my 2.4 GHz Wi-Fi network and the lack of a direct cable connection.

  2. SteamVR (Wireless): This provided a much smoother and more responsive experience, even on the same network configuration. The rendering and head-tracking were excellent.

  3. 3D Slicer VR Extension (via SteamVR): While the scene rendered perfectly, I encountered issues with controller bindings. Most controls were non-functional, except for basic vertical movement (up and down). My attempts to reconfigure bindings did not resolve the issue.

  4. Standalone SlicerVR Application: In contrast, launching the SlicerVR application directly was far more successful. All controller functions worked as expected, allowing for full interaction with the models.

Based on this, I have a few questions for the community:

  • Is the controller binding issue with the VR extension a known problem for the Meta Quest 3?

  • Are there recommended SteamVR binding profiles or a specific setup guide for the Quest 3 that I might have missed?

  • What is the recommended workflow for the best VR experience? Is using the standalone SlicerVR app the preferred method over launching it from within the main 3D Slicer application?

Thank you for any insights or advice you can offer.

Thank you

3D Slicer Virtual Reality Extension

Direct App

2

Thanks for sharing your experience!

Is the controller binding issue with the VR extension a known problem for the Meta Quest 3?

Looking at the DeveloperGuide, it looks like controller bindings are directly fetch from VTK which does not provide default bindings for the Meta Quest 3. I have a draft PR on VTK that should provide support for basic interactions with this device when using OpenXR as a backend.

You can try to locate the json files provided by the 3D Slicer VR Extension on your machine and apply the modifications from my PR and it should work.

3

@manjula Just letting you know that the SlicerVR extension is up again since quite recently, so I suggest using the latest Slicer / SlicerVR.

4

Hi @manjula, @cpinter and @LucasGandel,

@lassoan and @Sunderlandkyl at Queens have done some great progress in the past couple of months regarding support for SlicerVR on the Quest. The default VR interaction is working again on the Quest 3 using the OpenXR backend. The event handling system has been cleaned and refactored and it is now possible to define your own interaction in Python. Additionally, Kyle is working on supporting the passthrough mode of the Quest in SlicerVR. Some of these changes are already available in the nightly build of Slicer.

We are interested in knowing more about who’s using SlicerVR at the moment to see if we can join forces. There is also a possibility to have VR related projects at the next NAMIC Project Week if some are interested.

Please let us know where you are at with SlicerVR.

5

Hi Simon,

Where should I try?

I started integrating my old VR widget work into the latest SlicerVR, so I’ll be using it for this integration. Also we have some custom apps where I plan to offer SlicerVR for “advanced visualization”, but basically as an eye candy perk.

Just yesterday I installed SlicerVR on 5.10, and I wasn’t able to do any interaction. I didn’t see the controllers (not even as those green polygon things - using MetaQuest 3), wasn’t able to fly, nothing. The set camera from reference view function also worked strange (it acted as if the viewing direction was exactly towards my right side instead of forward). So it was not really usable.

What should I try to see what is the latest state? Where is the repo you have all the new work? Thanks!

@lassoan @Sunderlandkyl

6

It is recommended to use a recent Slicer Preview Release, OpenXR interface, and Meta Quest 3 headset. Slicer-5.10 may work with some controllers and backends, but we only test with the Slicer Preview Release and Meta Quest 3.

I’ve updated the description of interactions that are currently supported: GitHub - KitwareMedical/SlicerVirtualReality: A Slicer extension that enables user to interact with a Slicer scene using virtual reality. · GitHub

Kyle’s changes require VTK patches, and OpenXR SDK update. Integration of these is in progress (we expect to get them integrated within 1-2 weeks, in Slicer-5.12).

7

Hi Andras,

Could you share the location of the changes that have already been pushed to the public repository. I think it would help to understand how things have changed in terms of interaction management and where we can look if a specific interaction is not working as expected.