Stereo Off-Axis Projection and Mouse Interactions

Support
, ,
1

:robot: This post was written with AI assistance. But I’m still human

Hi,

I’m new to 3D Slicer and VTK. I am trying to build a “virtual window” effect using a custom stereo display that gives me real-time viewDistance and eyeSeparation.

My goal is to let the user interact normally with the 3D volume using standard mouse controls (rotate, pan, zoom). From my understanding, to achieve this “virtual window” effect while preserving standard mouse interactions, the physical screen coordinates must dynamically track the camera: the screen center should sit exactly at the camera position, and the EyePosition should be placed behind it (camera.GetPosition() - viewDist * dop).

However, as soon as camera.SetUseOffAxisProjection(True) is invoked, it feels like ViewUp is locked, and the user can no longer rotate the camera, only translate it. This leads to broken math on my side during mouse movements.

Here is how I’m updating the frustrum :

  layoutManager = slicer.app.layoutManager()
  threeDWidget = layoutManager.threeDWidget(0)
  renderWindow = threeDWidget.threeDView().renderWindow()
  renderer = renderWindow.GetRenderers().GetFirstRenderer()
  camera = renderer.GetActiveCamera()
  p = self.stereoParams
  pixelPitch = p['pixelPitch'] # unit : meter
  viewDist = p['viewDistance'] # Distance User to screen (m)
  eyeSeparation = p['eyeSeparation'] # IPD (m)

  viewportSizePx = renderWindow.GetSize()
  if viewportSizePx[0] <= 0 or viewportSizePx[1] <= 0:
      return # Window minimized or invalid
  
  viewportWidth = viewportSizePx[0] * pixelPitch # viewport width (m)
  viewportHeight = viewportSizePx[1] * pixelPitch # viewport height (m)
  halfW = viewportWidth / 2.0
  halfH = viewportHeight / 2.0

  up = np.array(camera.GetViewUp())
  dop = np.array(camera.GetDirectionOfProjection())
  cam = np.array(camera.GetPosition())

  right = np.cross(dop, up)
  rMag = np.linalg.norm(right)
  right /= rMag

  bl = cam - halfW * right - halfH * up
  br = cam + halfW * right - halfH * up
  tr = cam + halfW * right + halfH * up

  camera.SetUseOffAxisProjection(True)
  camera.SetScreenBottomLeft(bl)
  camera.SetScreenBottomRight(br)
  camera.SetScreenTopRight(tr)
  camera.SetEyePosition(camera.GetPosition()- viewDist * dop)
  camera.SetEyeSeparation(eyeSeparation)

It seems that I’m not doing things right.
Is there any documentation or existing examples on implementing off-axis projection with dynamic screen coordinates within Slicer?

Thanks for any help!

2

From a quick read to the code I think you forgot to use the set methods for the camera (e.g. SetViewUp, SetDirectionOfProjection, SetPosition)

3

Hi Mauro,

Thanks for the reply!

From my undestanding Slicer’s mouse interactor is dynamically updating the camera’s Position, ViewUp, and DirectionOfProjection as the user interacts, and my code reads those updates to dynamically recalculate the screen coordinates.

The core issue is that as soon as SetUseOffAxisProjection(True) is enabled, the mouse interactor loses the ability to rotate the camera and the ViewUp vector becomes completely locked. Only translation continues to work.

It feels like VTK’s internal off-axis calculations are fighting or overriding the interactor’s ability to modify the camera orientation. Is there a specific way to configure the interactor style or the camera so that rotation updates are still processed properly in off-axis mode?