# Slicer/Unity bridge, interest?

**URL:** <https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795>\
**Category:** Development\
**Created:** [October 1, 2020, 2:05pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795 "2020-10-01T14:05:57Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![adamrankin](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/adamrankin/32/155_2.png) [@adamrankin](https://discourse.slicer.org/u/adamrankin)\
**Post date:** [October 1, 2020, 2:05pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/1 "2020-10-01T14:05:58Z")

</div>

Hello all,

Our research group has some funds available for VR infrastructure development, and we are exploring the value of building a Slicer/Unity bridge.

Our primary goal is to enable Slicer on the HoloLens (and HoloLens 2) by streaming position data to Slicer, and streaming stereo rendered views back to HoloLens.

My first question is, has anyone started exploring this approach?

My second question is, is there interest from the community that investing in this area is worthwhile?

Thank you all for any feedback you have. Additionally, if you see this post and know of a research group who might be interested in this discussion, could you send them a link to this post?

Kind regards,  
Adam

@pieper @lassoan @cpinter @jcfr @franklinwk @Sunderlandkyl

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [October 1, 2020, 2:43pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/2 "2020-10-01T14:43:28Z")

</div>

That could be very cool, yes, I’d be interested in looking into this. We did something [somewhat similar that a while back](https://www.na-mic.org/wiki/VR_Radiology), where we served rendered images to use as texture maps in VR, but not the whole rendering.

The biggest question to me in this approach is minimizing latency to avoid simulator sickness issues. @adamrankin have you investigated what the best case framerate would be just for the data transfer (i.e. head motion -\> tracking data from hololens -\> slicer -\> render -\> transfer image -\> display image). If we know that’s possible then we can look into any optimizations required.

I’m interested in this not just for VR, but for the general topic of real-time render servers (e.g. for cloud).

---

<div class="post-metadata">

**Author:** ![adamrankin](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/adamrankin/32/155_2.png) [@adamrankin](https://discourse.slicer.org/u/adamrankin)\
**Post date:** [October 1, 2020, 4:09pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/3 "2020-10-01T16:09:32Z")

</div>

Another alternative might be an OpenXR backend for VTK. Although this doesn’t bypass all the UWP requirements for running on HoloLens 1/2.

---

<div class="post-metadata">

**Author:** ![adamrankin](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/adamrankin/32/155_2.png) [@adamrankin](https://discourse.slicer.org/u/adamrankin)\
**Post date:** [October 1, 2020, 7:21pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/4 "2020-10-01T19:21:29Z")

</div>

Or perhaps holographic remoting

> **[Add holographic remoting - Mixed Reality](https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/add-holographic-remoting)**
>
> Explains how to use Holographic Remoting to render holograms to a HoloLens over the network.

---

<div class="post-metadata">

**Author:** ![dzenanz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/dzenanz/32/1992_2.png) [@dzenanz](https://discourse.slicer.org/u/dzenanz)\
**Post date:** [October 1, 2020, 9:46pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/5 "2020-10-01T21:46:27Z")

</div>

@agirault Is there already some connection between VTK and Unity? Wasn’t there some work based on [vtktounity](https://gitlab.com/3dheart_public/vtktounity)?

---

<div class="post-metadata">

**Author:** ![agirault](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/agirault/32/74_2.png) [@agirault](https://discourse.slicer.org/u/agirault)\
**Post date:** [October 2, 2020, 1:22pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/6 "2020-10-02T13:22:30Z")

</div>

@dzenanz, yes, the link you shared is a nice open-source effort, though the last time I looked, it required a lot of hacks to make it work with VTK.

The latest work I’m aware of would be using [ActiviZ 9.0](https://www.kitware.eu/activiz/) (C# wrapping of VTK) which the Kitware Europe team have integrated into Unity as a [paid Unity module](https://assetstore.unity.com/packages/essentials/tutorial-projects/vtkunity-activiz-163686). There is a [free plugin](https://assetstore.unity.com/packages/essentials/tutorial-projects/vtkunity-medicalviewer-162395) to demonstrate medical imaging capabilities. None of this is open-source though.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [October 2, 2020, 1:33pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/7 "2020-10-02T13:33:10Z")

</div>

From a quick look at the vtk unity connections it appears they work by sharing an opengl context, but as I understand it this would not work for hololens.

---

<div class="post-metadata">

**Author:** ![adamrankin](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/adamrankin/32/155_2.png) [@adamrankin](https://discourse.slicer.org/u/adamrankin)\
**Post date:** [October 2, 2020, 1:34pm UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/8 "2020-10-02T13:34:48Z")

</div>

More importantly, it’s not just VTK that has to work. HoloLens requires UWP apps, which would require modifications to **all** underlying libraries.

---

<div class="post-metadata">

**Author:** ![abhiazadz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/abhiazadz/32/14358_2.png) [@abhiazadz](https://discourse.slicer.org/u/abhiazadz)\
**Post date:** [February 19, 2022, 11:38am UTC](https://discourse.slicer.org/t/slicer-unity-bridge-interest/13795/9 "2022-02-19T11:38:51Z")

</div>

Hey, I am trying to find a way to send Fiducials from Unity to Slicer through OpenIGTLink, can you please tell me how I can do it? I have never done cross-platform connections before.