# Quest VR controllers as live trackers

**URL:** <https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901>\
**Category:** Support\
**Tags:** surgical-navigation\
**Created:** [December 30, 2024, 8:19am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901 "2024-12-30T08:19:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [December 30, 2024, 8:19am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/1 "2024-12-30T08:19:47Z")

</div>

Hi,  
I recently came across 3d slicer and I was wondering if this software is capable of assisting me with a research project. I would like to do real time 2d ultrasound scanning and 3d volumetric reconstruction in virtual reality using a meta quest 3 vr controller as the spatial tracker for the ultrasound probe. Is this currently possible using 3d slicer and its extensions??

Thank you!

---

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [December 30, 2024, 2:20pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/2 "2024-12-30T14:20:28Z")

</div>

I should clarify that I am interested in doing this using quest 3’s augmented reality mode and 3d printing a mount for the quest controller that will attach to the ultrasound probe.

---

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [January 1, 2025, 8:23pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/3 "2025-01-01T20:23:48Z")

</div>

Is anyone able to tell me if this is currently possible? Thank you.

---

<div class="post-metadata">

**Author:** ![cpinter](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/cpinter/32/7995_2.png) [@cpinter](https://discourse.slicer.org/u/cpinter)\
**Post date:** [January 7, 2025, 10:55am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/4 "2025-01-07T10:55:08Z")

</div>

Do you mean this?

1. Load a 3D volume that you want to virtually scan with ultrasound
2. Show the volume in AR using Quest 3
3. Simulate ultrasound images at the probe position determined by the controller

1 and 3 are possible, about 2 I’m not sure. Or if you thought about a different workflow please describe yours in detail.

---

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [January 7, 2025, 1:24pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/5 "2025-01-07T13:24:38Z")

</div>

Thanks for your reply!

Workflow would be:

1. attach a quest 3 controller to an ultrasound probe using a 3d printed mount (in order to use the controller as a tracker in 3d slicer for the ultrasound probe)
2. Put on the quest 3 in passthrough mode (AR)
3. Slowly live scan a model with the ultrasound probe, displaying each slice superimposed over the model in AR directly below the ultrasound probe
4. Once a sufficient 2d scan is collected, have 3d Slicer then do a 3d volumetric reconstruction of the images and display the 3d reconstruction in AR superimposed over the model (below the ultrasound probe) instead of just displaying the 2d ultrasound images.

Hope that makes sense. Thanks again for your help.

---

<div class="post-metadata">

**Author:** ![cpinter](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/cpinter/32/7995_2.png) [@cpinter](https://discourse.slicer.org/u/cpinter)\
**Post date:** [January 7, 2025, 1:38pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/6 "2025-01-07T13:38:55Z")

</div>

Yes, thank you for the description!

1. I’m not sure how practical it is to attach a controller, but it is possible, if you want to avoid the need of an external tracker. Maybe Meta manufactures smaller hardware that supports this better. In any case 1) there needs to exist a reference coordinate system fixed to the “patient”, and 2) the US probe needs to be calibrated in reference to the tracked controller to obtain the ControllerToImage transform
2. The Kitware team must know where the OpenXR support stands currently in Slicer. I haven’t tried this feature and not following the efforts on the commit level so cannot give you an answer whether this is already possible or not
3. This should be OK if the above 2 are solved
4. Also possible, using SlicerIGT features

---

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [January 7, 2025, 1:44pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/7 "2025-01-07T13:44:35Z")

</div>

Ok so it sounds like I need to check with the Kitware team. Do you know who to contact there?

---

<div class="post-metadata">

**Author:** ![cpinter](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/cpinter/32/7995_2.png) [@cpinter](https://discourse.slicer.org/u/cpinter)\
**Post date:** [January 7, 2025, 1:46pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/8 "2025-01-07T13:46:01Z")

</div>

@jcfr or @LucasGandel might be able to tell you if there is AR support via OpenXR in the current version.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [January 7, 2025, 3:11pm UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/9 "2025-01-07T15:11:05Z")

</div>

For tracking and volume reconstruction using the VR controller as tracker you don’t need any developments, it all works, just enable the controller transforms in the Virtual Reality module in Slicer. For volume reconstruction (and for meaningful display of the ultrasound image) you need to figure out the transform between the controller’s coordinate system and the image. You can do this visually or by using any ultrasound probe calibration tools (see [SlicerIGT tutorials](https://www.slicerigt.org/wp/user-tutorials/)).

That said, the VR controllers are very bulky and inaccurate. The good visual alignment of the controller and its AR rendering is misleading: the actual absolute tracking accuracy is quite bad compared to even an inexpensive tracker like the [OptiTrack Duo](https://www.optitrack.com/cameras/v120-duo/).

For augmented reality display using Meta Quest 3, you need to improve the Virtual Reality module in Slicer: the module needs to request the camera stream from the headset and then display those images in the view’s background. Since Microsoft discontinued the HoloLens2, it is quite likely that new projects will use Meta Quest 3 for augmented reality (due to quality, availability, and price) and eventually somebody will implement this, but I don’t know about anyone working on this right now.

---

<div class="post-metadata">

**Author:** ![sberns](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@sberns](https://discourse.slicer.org/u/sberns)\
**Post date:** [January 8, 2025, 3:41am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/10 "2025-01-08T03:41:30Z")

</div>

Thank you very much for your input. It was very informative!

---

<div class="post-metadata">

**Author:** ![LucasGandel](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lucasgandel/32/18202_2.png) [@LucasGandel](https://discourse.slicer.org/u/LucasGandel)\
**Post date:** [January 10, 2025, 10:35am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/11 "2025-01-10T10:35:41Z")

</div>

We don’t have anyone working on VTK rendering in passthrough mode at Kitware Europe currently, but we have submitted a few open calls for projects to do so recently (including integration in SlicerVR). I’ll make sure to let you know if things move forward.  
When using Quest 3, I think you must use Air Link as opposed to connecting it using the Quest Link cable, otherwise it will request OpenGLES as the rendering backend which is not supported by SlicerVR as far as I know.

---

<div class="post-metadata">

**Author:** ![LucasGandel](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lucasgandel/32/18202_2.png) [@LucasGandel](https://discourse.slicer.org/u/LucasGandel)\
**Post date:** [March 11, 2025, 9:15am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/12 "2025-03-11T09:15:45Z")

</div>

We investigated the possibility to enable passthrough in the VTK-OpenXR code and while using the XR\_FB\_passthrough extension works as expected, the very low framerate we get with the Meta Quest 2 connected with AirLink makes the approach unusable. We will do further testing with higher bandwidth or using a Link cable, stay tuned.

---

<div class="post-metadata">

**Author:** ![LucasGandel](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lucasgandel/32/18202_2.png) [@LucasGandel](https://discourse.slicer.org/u/LucasGandel)\
**Post date:** [March 14, 2025, 7:28am UTC](https://discourse.slicer.org/t/quest-vr-controllers-as-live-trackers/40901/13 "2025-03-14T07:28:23Z")

</div>

Connecting the MetaQuest 2 to the workstation with Link (i.e. using a cable) gives real-time performances with passthrough enabled.