I am currently trying to find out how I could stream one of Slicers slices (e.g. the red slice) to another device. There seem to be lots of options to receive image streams in Slicer (e.g. PlusToolkit, OpenIGTLink) though I could not find a tutorial how I could send the image data to another client.
First I tried to use the OpenIGTLink interface with python. So far I could not find the suitable vtkMRML…Node for sending an ImageMessage. If I understood correctly, I can use RegisterOutgoingMRMLNode on an vtkMRMLIGTLConnectorNode to send data to a client. Though it is not so clear for me, what kind of node to attach for sending the image of a slice.
My current approach now makes use of the pyIGTLink module which supports ImageMessages. Would this be the correct way to go?
Furthermore I have some concerns about the streaming performance. The receiver is a mobile device, so we use WLAN. Our goal is to implement a live streaming of the slice. Not sure whether this is possible by sending raw image data. The C++ build of OpenIGTLink has examples for video streaming. For example they are using the VP9 codec for compression. I assume this feature wont be available in any python implementation of OpenIGTLink.
Could I implement video streaming in Slicer if I would build Slicer in C++ and extending it? So far I have only used the Python programming features of Slicer. If doable, I would switch to C++ to implement video streaming.
Probably the C++ implementation is faster and it interferes less with the main application GUI.
The bottleneck can be wifi bandwidth, which you may address by using a dedicated wifi access point with high-speed protocols.
You may drastically reduce the required bandwidth (by 90-99%) by using compression, but then you may run into performance issues due to the overhead of compression. h264 compression/decompression is hardware-accelerated on most platforms (even mobile), so that would be likely to work. However, there are inconveniences with licensing (you cannot ship h264 libraries to users without paying license fee or at least keeping track of the number of instances distributed to users). VP9 has no licensing issues but VP9 compression is very slow, and VP9 decompression is not very fast either, so there is a higher chance that it’ll limit your performance.
If you only need to stream video (no transforms, no metadata, etc.) then you might also look into consumer video streaming protocols.
We have been also successfully using VNC to allow clinicians to use Slicer in the operating room on a tablet. We even show live ultrasound video, but the main interaction happens on static images:
Thanks for the many hints. I will try to use the vtkMRMLScalarVolumeNode first and test how it performs.
Probably the C++ implementation is faster and it interferes less with the main application GUI.
By this you mean that building Slicer on my own and integrating the C++ OpenIGTLink will be faster, right? I assume OpenIGTLink is already somewhere in the build environment as Slicer already supports IGTL. Ideally I would extend the Slicer IGTL module to stream slices via video streaming.
We have been also successfully using VNC to allow clinicians to use Slicer in the operating room on a tablet.
The video looks really nice! Though I guess VNC won’t be an option for us as the mobile client will receive the stream in an application which is being developed.
Slicer already has compressed streaming video support, so you don’t need to build Slicer. You can implement a scripted module that captures a view and saves it into a vtkMRMLScalarVolumeNode and OpenIGTLinkIF module takes care of the rest.
You can use VNC protocol to show (and interact with) Slicer in a webpage using noVNC. For example, this is how Slicer docker containers work: Slicer runs on a cloud server and you interact with it in your web browser.
I did some testing with the vtkMRMLScalarVolumeNode and could finally send out a slice to my client. Unfortunately I found out that this feature works with Slicer 4.8 but is not yet working with the Slicer 4.11 version (did not test 4.10 yet).
I am not so sure though whether this is a known issue or something which should be reported. Also not sure if this something for the OpenIGTLink Extension or the Slicer main app.
Would it help to report the issue? If yes where? Or since this is about a nightly build it not of interest?
There was another bug in my code so the sending via OpenIGTLink now works correctly and I can receive the image message now.
Unfortunately though, Slicer silently crashes if the image resolution increases. I assume that the copy process is taking too much time, such that the main thread somehow crashes.
In cases where sending the red slice works, I also found that scrolling the MPRs in the red slice is much slower than usual. Would it be possible to shift the sending of the IGTL message to another thread such that scrolling works smoother again? This could also prevent the crash in case of a bigger image.
Here some code what I did to send the red slice. I am creating a vtkWindowToImageFilter and set the render-window ot the red slice as input. Then I create a vtkMRMLScalarVolumeNode and attach the output of the filter as image data. When the red slice was modified I have to manually call Update() and Modified() on the image filter, otherwise the data is not updated and sent. Maybe there is also some bug?
We could implement background sending in OpenIGTLink module (make a copy of the data and send it on a separate thread), but implementing real-time broadcasting from Slicer is not a priority right now.
You have many options to choose to go forward:
implement background sending of images in OpenIGTLink module in Slicer (we’ll be able to help you getting started)
implement your own sending mechanism as a C++ loadable module in Slicer
use an existing screen sharing implementation (VNC even has C++, C#, javascript implementations, so it should not be hard to set it up even inside a mobile application)
What is your end goal and the workflow that you are trying to implement now?
If the red slice has lower resolution, scrolling the MPR in the red slice hangs/slows down. If for example I choose red slice only layout, Slicer actually crashes.
Thanks for the many help and suggestions that you proposed. Aim of my project is to create a small prototype for navigation. We plan 3D positions in the MPRs and next want to navigate the surgeon to the planned positions via a tracked tool. Therefore we want to stream the MPRs to a mobile device which is attached to the tracked device.
Concerning my options:
Slicer Web Service: Looks interesting to me. This would mean that the client regulary requests an image of a slice. Would this also run in the main thread leading to slow downs? Goal is to have some kind of a stream, so I would repeatedly request an updated image.
VNC: Still not so sure if this would make sense. We do not want to see the full slicer UI. For now we also dont want to use the UI. Would VNC allow to grab red, green or yellow slice only?
The other two options might also be worth looking into it.
I also started looking into video streaming. Could you give me a hint how I can switch from sending IMAGE data to VIDEO data? In OpenIGTLinkIF there is a checkbox “UseStreamingVolume”, but this does not seem to change the message type.
Yes, with the current setup the client (browser or other software) requests the data when it wants it, so it can control the timing to match display and networking speeds. The response does run in the main thread, but the network transmissions are event driven so generally it shouldn’t be too bad.
The other thing the offered by the SlicerWeb approach is structured access to loaded data (like the list of volumes) and the option to add hooks to control Slicer’s behavior from the remote device.
Do you mean Slicer not responding or the process exits (does not show up anymore in task manager)?
It seems that you may be prematurely optimizing things and maybe not the most important ones. Over the years, we have implemented many treatment planning and navigation systems and learnt that in prototyping phase the most important is to be flexible: be able to quickly update your system based on what you learn by using it in practice, add new features, quickly try alternative approaches, etc. Prototyping phase is quite long, it lasts until you have worked out a good workflow, solid software and hardware, and validated their safety and efficacy on about 100 patients at a few sites.
During this phase, there is always an engineer in the room, who needs to be aware what’s happening, what exactly the system is doing, what is shown on the system’s each display, and react promptly to the surgeon’s requests (answer questions, assist with doing something when the surgeon asks for it, adjust views, discover potential issues before they cause any problem, etc.). For this, screen sharing is a good solution - you can use multiple screens, mirror screens or selected windows to remote monitors (surgeon’s console, overhead monitor) and allow control from some of them (e.g., you can allow the surgeon to press buttons on his console). This also comes for free, as you can find remote desktop applications for all platforms. You may even use hardware solutions if you prefer, such as wired or wireless touchscreen displays.
@Sunderlandkyl can we configure an image connection to send out compressed live image stream?
Slicer process completely exits. The whole UI disappears and also in the task manager slicer.exe is removed. I tested once more by using the ImagerClient3.exe of the OpenIGTLink examples to make sure that the receiver side is as expected. Slicer still crashes if the ImagerClient connects and I change the layout to red slice only. The ImagerClient actually also shuts down with the error message “Message size information and actual data size dont’t match.”.
Maybe during the resize of the red slice widget some inconsistencies occur which lead to the crash? The image renderer maybe also needs to be recreated?
Another observation on this: Simply resizing the whole Slicer application also leads to a resize of the red slice. This works quite well until a certain size has been reached. At a certain size (which again means higher resolution of the red slice) Slicer crashes again.
Maybe that’s a good point. The mobile app already supports navigation features. Streaming the live MPR would be an additional feature which might be valuable for the surgeons. In fact the first step is to evaluate whether attaching a mobile device improves the usability of the navigation system.
This is useful information. Probably somewhere the error is not handled correctly and that causes the crash. It would be great if you could provide a step-by-step description of how to reproduce it and submit it as a bug report at Issues · openigtlink/SlicerOpenIGTLink · GitHub.
I would be interested to implement a loadable module in C++ with one of the approach which was discussed earlier. Can you give me pointers to get started? I would like to send all the 3 planes to the edge device.
