How to configure VR controls interactions in Python?

Development
, ,
1

Hi,

I’m developing a 3DSlicer module using the virtual reality module and a Valve Index headset.
I’m trying to configure my own controller interactions in order to:

  1. Deactivate all default VR controls
  2. Add new customized VR controls (link the joystick movements to a transform so that a model moves accordingly // use the L/R triggers as “Previous” and “Next” buttons)

The closest I’ve been is by getting the vtkRenderWindowInteractor3D using slicer.modules.virtualreality.viewWidget().interactor(). The good point is I’m able to remove event observers from this interactor so that the default VR controls don’t work anymore.
But, out of 51 default event observers that I found in the vtkRenderWindowInteractor3D, it is enough to deactivate only 3 of them so that no command works anymore: PinchEvent, Move3DEvent and Button3DEvent (the 48 others observers can be removed and it doesn’t seem to change anything).
This is an issue for my second goal, because as I want to allocate specific VR controls to individual events, it seems that all controls are accessible through 3 observers. Furthermore, I’d like to get data that come from events (particularly 2D vector for joysticks).

Then I found this code from vtkVirtualRealityViewInteractorStyle.cxx

//----------------------------------------------------------------------------
// Copied from vtkOpenVRInteractorStyle::SetupActions
//------------------------------------------------------------------------------
void vtkVirtualRealityViewInteractorStyle::SetupActions(vtkRenderWindowInteractor* iren)
{
  vtkOpenVRRenderWindowInteractor* oiren = vtkOpenVRRenderWindowInteractor::SafeDownCast(iren);

  if (oiren)
  {
    oiren->AddAction("/actions/vtk/in/Elevation", vtkCommand::Elevation3DEvent, true);
    oiren->AddAction("/actions/vtk/in/Movement", vtkCommand::ViewerMovement3DEvent, true);
    oiren->AddAction("/actions/vtk/in/NextCameraPose", vtkCommand::NextPose3DEvent, false);
    oiren->AddAction("/actions/vtk/in/PositionProp", vtkCommand::PositionProp3DEvent, false);
    oiren->AddAction("/actions/vtk/in/ShowMenu", vtkCommand::Menu3DEvent, false);
    oiren->AddAction("/actions/vtk/in/StartElevation", vtkCommand::Elevation3DEvent, false);
    oiren->AddAction("/actions/vtk/in/StartMovement", vtkCommand::ViewerMovement3DEvent, false);
    oiren->AddAction("/actions/vtk/in/TriggerAction", vtkCommand::Select3DEvent, false);
  }
}

But the interactor I use is from the vtkRenderWindowInteractor3D class, which doesn’t have these lines. So I got access to the interactorStyle vtkVirtualRealityViewInteractorStyle using interactor.GetInteractorStyle() but this “object has no attribute ‘SetupActions’”.
Plus, the above vtkCommand are not understood neither by the interactor nor by the interactorStyle.

How should I do? If I’m not on the right track at all, what is the right way to change controls? Can I do this in Python? If not, what are the alternatives?

Sorry if it’s not clear but it’s also not in my mind, I hope I provided enough informations.
Thanks you very much,

[Slicer 4.11.20210226 - Python 3.7]
PS: I’m using an old 3DSlicer version because VR controllers didn’t work in the latest version last time I checked (2023-03-03).

2

The 3D Slicer Virtual Reality extension is still not working in Slicer 5.3.0-2023-04-27.

Symptoms on Windows 11, Oculus Quest2 headset, JSON files copied:

Connecting the headset via SteamVR works, and rendering starts.
You can fly the object around by pressing the right joystick and moving the right controller to a different axis.
Trying to “grab” the object with the grab keys of both controllers - for rotating it etc. - immediately erases the object from the screen
:frowning:

3

Grabbing objects using the two controllers was never a feature of SlicerVR. The two-handed gesture controlled the camera. This is the last part that still does not work. Grabbing and moving objects as before (with one controller) works for me.

4

Thanks for your answer!
I finally stayed on Slicer 4.11.20210226 but used OpenVR2Key to map VR controllers inputs to keyboard inputs. Then, I used QShortcut objects to listen to keyboard events and to callback my own functions (moving a needle according to one joystick and spinning a head model according to the other one).
If it works in Slicer 5.3.0-2023-04-27 (I don’t see why it wouldn’t), it should be possible to recreate all default SlicerVR controls this way (a bit of work :grimacing:)

1 Like
5

Thanks for the info!
I am also hoping to be able to create my own mapping of controller buttons to actions.
Is the workaround described above best way to acheive this?
Any code snippet you could share with me would be greatly appriciated!

6

The interaction in virtual reality in Slicer is being reworked right now to use as much as possible from VTK, yet keep the flexibility and access to many more features in Slicer. Maybe @jcfr can offer more information on timelines and what exactly will be available in the near future.

2 Likes