Stereo Off-Axis Projection and Mouse Interactions

Support
, ,
1

:robot: This post was written with AI assistance. But I’m still human

Hi,

I’m new to 3D Slicer and VTK. I am trying to build a “virtual window” effect using a custom stereo display that gives me real-time viewDistance and eyeSeparation.

My goal is to let the user interact normally with the 3D volume using standard mouse controls (rotate, pan, zoom). From my understanding, to achieve this “virtual window” effect while preserving standard mouse interactions, the physical screen coordinates must dynamically track the camera: the screen center should sit exactly at the camera position, and the EyePosition should be placed behind it (camera.GetPosition() - viewDist * dop).

However, as soon as camera.SetUseOffAxisProjection(True) is invoked, it feels like ViewUp is locked, and the user can no longer rotate the camera, only translate it. This leads to broken math on my side during mouse movements.

Here is how I’m updating the frustrum :

  layoutManager = slicer.app.layoutManager()
  threeDWidget = layoutManager.threeDWidget(0)
  renderWindow = threeDWidget.threeDView().renderWindow()
  renderer = renderWindow.GetRenderers().GetFirstRenderer()
  camera = renderer.GetActiveCamera()
  p = self.stereoParams
  pixelPitch = p['pixelPitch'] # unit : meter
  viewDist = p['viewDistance'] # Distance User to screen (m)
  eyeSeparation = p['eyeSeparation'] # IPD (m)

  viewportSizePx = renderWindow.GetSize()
  if viewportSizePx[0] <= 0 or viewportSizePx[1] <= 0:
      return # Window minimized or invalid
  
  viewportWidth = viewportSizePx[0] * pixelPitch # viewport width (m)
  viewportHeight = viewportSizePx[1] * pixelPitch # viewport height (m)
  halfW = viewportWidth / 2.0
  halfH = viewportHeight / 2.0

  up = np.array(camera.GetViewUp())
  dop = np.array(camera.GetDirectionOfProjection())
  cam = np.array(camera.GetPosition())

  right = np.cross(dop, up)
  rMag = np.linalg.norm(right)
  right /= rMag

  bl = cam - halfW * right - halfH * up
  br = cam + halfW * right - halfH * up
  tr = cam + halfW * right + halfH * up

  camera.SetUseOffAxisProjection(True)
  camera.SetScreenBottomLeft(bl)
  camera.SetScreenBottomRight(br)
  camera.SetScreenTopRight(tr)
  camera.SetEyePosition(camera.GetPosition()- viewDist * dop)
  camera.SetEyeSeparation(eyeSeparation)

It seems that I’m not doing things right.
Is there any documentation or existing examples on implementing off-axis projection with dynamic screen coordinates within Slicer?

Thanks for any help!

2

From a quick read to the code I think you forgot to use the set methods for the camera (e.g. SetViewUp, SetDirectionOfProjection, SetPosition)

3

Hi Mauro,

Thanks for the reply!

From my undestanding Slicer’s mouse interactor is dynamically updating the camera’s Position, ViewUp, and DirectionOfProjection as the user interacts, and my code reads those updates to dynamically recalculate the screen coordinates.

The core issue is that as soon as SetUseOffAxisProjection(True) is enabled, the mouse interactor loses the ability to rotate the camera and the ViewUp vector becomes completely locked. Only translation continues to work.

It feels like VTK’s internal off-axis calculations are fighting or overriding the interactor’s ability to modify the camera orientation. Is there a specific way to configure the interactor style or the camera so that rotation updates are still processed properly in off-axis mode?

4

If you just want to do since quick testing in python then you can add an observer with high priority that captures and consumes mouse move and button events. If you want to have a proper solution then you probably need to investigate what happens exactly in the mouse event handler in C++.

5

Hello Mr.Andras Lasso!

My name is Tomás. I´m a Medical Student from Portugal and I´m currently exploring the 3D Slicer app and it´s extensions for CT scan images of the spine. I am new on this 3D Slicer forum, so I wasn´t able to send you a message. Is it possible to ask you some questions via e-mail or even in this forum through direct messaging if feasible? Thank you for your time in advance!