Our research group has some funds available for VR infrastructure development, and we are exploring the value of building a Slicer/Unity bridge.
Our primary goal is to enable Slicer on the HoloLens (and HoloLens 2) by streaming position data to Slicer, and streaming stereo rendered views back to HoloLens.
My first question is, has anyone started exploring this approach?
My second question is, is there interest from the community that investing in this area is worthwhile?
Thank you all for any feedback you have. Additionally, if you see this post and know of a research group who might be interested in this discussion, could you send them a link to this post?
That could be very cool, yes, I’d be interested in looking into this. We did something somewhat similar that a while back, where we served rendered images to use as texture maps in VR, but not the whole rendering.
The biggest question to me in this approach is minimizing latency to avoid simulator sickness issues. @adamrankin have you investigated what the best case framerate would be just for the data transfer (i.e. head motion -> tracking data from hololens -> slicer -> render -> transfer image -> display image). If we know that’s possible then we can look into any optimizations required.
I’m interested in this not just for VR, but for the general topic of real-time render servers (e.g. for cloud).
@dzenanz, yes, the link you shared is a nice open-source effort, though the last time I looked, it required a lot of hacks to make it work with VTK.
The latest work I’m aware of would be using ActiviZ 9.0 (C# wrapping of VTK) which the Kitware Europe team have integrated into Unity as a paid Unity module. There is a free plugin to demonstrate medical imaging capabilities. None of this is open-source though.
From a quick look at the vtk unity connections it appears they work by sharing an opengl context, but as I understand it this would not work for hololens.
Hey, I am trying to find a way to send Fiducials from Unity to Slicer through OpenIGTLink, can you please tell me how I can do it? I have never done cross-platform connections before.