SlicerVirtualReality extension for recent versions

Billuc · December 20, 2023, 8:36pm

Hi !
I am new to Slicer, but I am interested in the SlicerVirtualReality extension. I wanted to try it with the latest versions of Slicer (5.6 or 5.7), but I couldn’t find it in the extension manager, while I could find it with Slicer 5.2.2.
Has this extension not been compiled for the most recent versions ? Do I have to compile it myself if I want to try it out ?

Thanks in advance.
Luc

muratmaga · December 21, 2023, 5:00am

I believe it is broken for current stable and preview. Use it with 5.4.

jcfr · December 21, 2023, 7:58am

The extension is actively being worked on and you should expect a working OpenVR, OpenXR as well as experimental OpenXRRemoting integration within 1 or 2 weeks. Thanks for your patience

Billuc · January 10, 2024, 4:00pm

@jcfr I tried to use the extension today, as I saw you made updates and the extension was available for Slicer5.6.1. I could install it, but I got some problems…
I have a Meta Quest 2, so that is maybe part of the reason why I got these problems, but I thought that since I could use SteamVR, it should work.
The thing is that whatever I do, whatever option I change, whatever data I try to open, I always get the same result : a blank scene, with the controllers at the top of my FOV (moving alongside my head, but also moving when I move the controller) and no model appearing (or not in my FOV).

Are you aware of this ? Or maybe the new version isn’t stable yet ?
Thanks in advance.
Luc

pieper · January 10, 2024, 7:19pm

I don’t know if it’s still an issue, but at one point I know that you needed to have some data loaded in the Slicer scene for the VR to initialize correctly.

Billuc · January 11, 2024, 8:11am

Unfortunately, I did have some data loaded. I haven’t tried initializing the VR before loading data, might try later…

petr.raunigr · December 11, 2024, 2:40pm

Sorry for re-opening a long dead post… did you (or anyone else) ever figured it out? I’m dealing with the exact same issue… blank space and controllers hanging at top of my FOV (rotating correctly).

Thanks.

cpinter · December 11, 2024, 3:23pm

Are you using the latest Slicer? Which backend do you use OpenXR or OpenVR (via SteamVR)?

petr.raunigr · December 12, 2024, 8:55am

Hello, so…

3D Slicer: 5.7.0-2024-11-26 r33130 / c5e8d83
SlicerVirtualReality: cloned 3. 12. 2024 (can’t find version info)
SteamVR: 2.8.8
Varjo Base: 4.3.0.14 (available update to 4.5.1.18)

My setup consists of HTC Vive controllers (1st edition) and Varjo Aero headset + SteamVR.

I tried to use SteamVR OpenXR, I also tried to set Varjo (in their software Varjo Base) as an OpenXR provider + OpenVR provider.

In SlicerVirtualReality settings I tried to set OpenXR and also OpenVR but I have the same problem for both of them but there are few differencies.

(Examples below were captured with no data loaded. It doesn’t matter because even if I load any kind of data the result would be the same)

Set OpenXR

Console output

Switch to module: “Welcome”
Switch to module: “VirtualReality”
Initializing “OpenXR” XR backend (1/1)
void __cdecl qMRMLVirtualRealityView::updateViewFromReferenceViewCamera(void) failed: no reference view node is set
Optional extensions HandTracking is supported
Runtime Name: SteamVR/OpenXRRuntime Version: 2.8.8
Debug: In vtkOpenXRManager.cxx, line 879
(nullptr): Successfully got XrSystem with id 1153015959527621024 for HMD form factor.

System Properties for system id:1153015959527621024, with name “SteamVR/OpenXR : lighthouse”, vendorID=10462
Max Layers : 16
Max Swapchain Height: 2692
Max Swapchain Width : 3140
Orientation Tracking: True
Position Tracking : True
Hand Tracking : 1
Runtime supports 1 view configurations
Type XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO: FOV mutable: True
Runtime supports 3 reference spaces:
XR_REFERENCE_SPACE_TYPE_VIEW
XR_REFERENCE_SPACE_TYPE_LOCAL
XR_REFERENCE_SPACE_TYPE_STAGE
Debug: In vtkOpenXRManager.cxx, line 1030
(nullptr): Runtime supports 8 swapchain formats

View Configuration View 0
Resolution : Recommended: 3140x2692, Max: 8192x8192
Swapchain Samples: Recommended: 1, Max: 1
View Configuration View 1
Resolution : Recommended: 3140x2692, Max: 8192x8192
Swapchain Samples: Recommended: 1, Max: 1
XR backend " OpenXR " initialized
ActionManifestPath: “D:/Slicer_plugins/SlicerVirtualReality/build_debug/vtkRenderingOpenXR-build/vtkRenderingOpenXR/”
Number of registered displayable manager: 8
Registered displayable managers:
vtkMRMLModelDisplayableManager
vtkMRMLThreeDReformatDisplayableManager
vtkMRMLCrosshairDisplayableManager3D
vtkMRMLMarkupsDisplayableManager
vtkMRMLSegmentationsDisplayableManager3D
vtkMRMLTransformsDisplayableManager3D
vtkMRMLLinearTransformsDisplayableManager
vtkMRMLVolumeRenderingDisplayableManager
Debug: In vtkOpenXRManager.cxx, line 264
(nullptr): Session started.

Debug: In vtkOpenXRManager.cxx, line 1197
(nullptr): Create action set vtk-actions: VTK actions

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/khr/simple_controller

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/htc/vive_controller

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/oculus/touch_controller

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/hp/mixed_reality_controller

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/valve/index_controller

Debug: In vtkOpenXRManager.cxx, line 1349
(nullptr): SuggestActions for profile : /interaction_profiles/microsoft/hand_interaction

Debug: In vtkOpenXRManager.cxx, line 552
(nullptr): Could not suggest actions [XR_ERROR_PATH_UNSUPPORTED].

failed at glDeleteFramebuffers 16 OpenGL errors detected
0 : (1282) Invalid operation
1 : (1282) Invalid operation
2 : (1282) Invalid operation
3 : (1282) Invalid operation
4 : (1282) Invalid operation
5 : (1282) Invalid operation
6 : (1282) Invalid operation
7 : (1282) Invalid operation
8 : (1282) Invalid operation
9 : (1282) Invalid operation
10 : (1282) Invalid operation
11 : (1282) Invalid operation
12 : (1282) Invalid operation
13 : (1282) Invalid operation
14 : (1282) Invalid operation
15 : (1282) Invalid operation

Instead of a green headset icon + two green controllers icons I get this orange warning (sometimes there are all 3 of them).
Also I don’t see controller models in VR but only green triangles.

Set OpenVR

Console output

Switch to module: “Welcome”
Switch to module: “VirtualReality”
Initializing “OpenVR” XR backend (1/1)
void __cdecl qMRMLVirtualRealityView::updateViewFromReferenceViewCamera(void) failed: no reference view node is set
XR backend " OpenVR " initialized
ActionManifestPath: “D:/Slicer_plugins/SlicerVirtualReality/build_debug/vtkRenderingOpenVR-build/vtkRenderingOpenVR/”
Number of registered displayable manager: 8
Registered displayable managers:
vtkMRMLModelDisplayableManager
vtkMRMLThreeDReformatDisplayableManager
vtkMRMLCrosshairDisplayableManager3D
vtkMRMLMarkupsDisplayableManager
vtkMRMLSegmentationsDisplayableManager3D
vtkMRMLTransformsDisplayableManager3D
vtkMRMLLinearTransformsDisplayableManager
vtkMRMLVolumeRenderingDisplayableManager

This time there is a correct icon for a headset and sometimes also for controllers (don’t know what causes controller icons to show up though).

This time I can see controller models in VR but they have the same problem… sitting at top of my FOV. Moreover, I can see a strange cube/square:

A possible clue
I see controllers and the cube (in case of OpenVR) twice… like if it’s rendered for both eyes separately or no depth perception is happening.

Summary:
I tried to use OpenVR, OpenXR, switch between OpenXR/VR provider from SteamVR to Varjo Base and back. I tried loading MHD data and tried VR without loading any data. The only difference I can see are different icons representing VR headset and controllers in 3D Slicer and different controller models in VR (green triangles for OpenXR, correct models for OpenVR) + in case of OpenVR I also see a cube/square. I also can see controller models/triangles + cube for each eye separately.

EDIT:
If I select OpenVR and in SlicerVirtualReality settings check the “Controller Transform” option this errors spam my console:

Console output

void __cdecl qMRMLVirtualRealityViewPrivate::updateTransformNodeAttributesFromDevice(class vtkMRMLTransformNode *,enum vtkEventDataDevice,unsigned int) : Unable to retrieve LeftController pose
void __cdecl qMRMLVirtualRealityViewPrivate::updateTransformNodeAttributesFromDevice(class vtkMRMLTransformNode *,enum vtkEventDataDevice,unsigned int) : Unable to retrieve RightController pose

petr.raunigr · December 18, 2024, 12:25pm

Ok, so the problem was more about me not knowing how to work with 3D Slicer rather than a bug in the extension.

As I mentioned, I loaded data, drag and dropped a volume model into the 3D scene and I thought that’s it.

What I needed to do was to check VirtualRealityVIew in the Volume Rendering module (see the screenshot below). Other thing that properly initialized a VR scene was to add one of the 3 CT slices into the 3D scene.

UX proposition
I find it still a bit weird that not doing that I don’t have a properly running VR scene (problems described in the previous post). I would expect that if I don’t properly prepare a model/data that I would at least end up with an empty VR space with controllers properly synced etc.

cpinter · December 18, 2024, 3:26pm

The VR view is not added to the views where the volume rendering appears, because by the time you show it, there is no VR view yet. So it needs to be manually added if you do things in this order. Great that you figured this out yourself! (I think that there have been topics here in the forum where we describe this though)

I agree. I’m not sure why not even the controllers appear. Maybe it’s the result of the undefined view parameters.

Topic		Replies	Views
Does SlicerVirtualReality work with 5.6? Support virtual-reality	2	165	December 11, 2023
Virtual reality Support virtual-reality	7	621	May 2, 2021
Connecting HTC vive with 3D slicer Support	38	4662	December 16, 2018
SlicerVR using Valve Index Support loadable-module	8	845	September 13, 2021
Black screen on VR Support virtual-reality	11	614	July 27, 2022

SlicerVirtualReality extension for recent versions

Related topics