First of all, thank you for your effort in developing 3DSlicer as it is a great platform.
I am developing a 3DSlicer module for surgical tool navigation. I want to update the 3D visualization of a model every time I get a new transform matrix of the tool geometrical model. I am getting a new transform from an OPCUA server periodically.
Currently, I am able to update the visualization as desired only when I actively hover the mouse over the multiplanar reconstruction slices (RGY).
I suspect there is a connection with the DataProbe and the CrosshairNode events that are fired during this interaction with the slices, but I cannot identify what I potentially could use from DataProbe.py.
Find the GIF attached as a clarification of what I explained above. The movement is kind of fast because I get a new transform every 100ms.
Are there any events or event-related callbacks I can trigger for updating the 3D visualization as described? This is, every time I get a new transform. Which would be the “correct” way or the recommended practice for this? I would greatly appreciate your help
This displaying tracked objects is exactly what Slicer IGT is designed for. Likely you can just use Slicer IGT directly, but even if you want to create your own version, you may find inspiration in how Slicer IGT gets the view to update.
Thank you for your reply. We already explored the possibility of utilizing SlicerIGT and SlicerIGTLink, but unfortunately, we are constrained to using an OPCUA protocol to communicate between Slicer and the rest of the elements of our application. This prevents us from using the alternative you suggested. I will look for any useful bits of code in the SlicerIGT codebase, but I think this will be excruciatingly slow until I come up with something.
I am still in search of a way to update the visualization automatically with native 3DSlicer functions, without relying on external modules.
I forgot to add that I already tried with slicer.app.processEvents() but it does not do the trick for me.
"
In 3D Slicer, updating the visualization of a model usually involves two primary steps:
Update the transform matrix associated with the model.
Invoke the render function to refresh the visualization.
If your model only updates when you hover over the multiplanar reconstruction slices (RGB), it may indeed be related to events triggered by DataProbe.py or CrosshairNode interactions, which are likely causing a render update.
Here’s a hypothetical scenario: if your OPCUA server updates the transform every 100ms, but the render update happens less frequently (only when you interact with the slices), then the visualization will seem to only update during those interactions.
To solve this problem, you could consider the following approaches:
Manual Rendering: Whenever you receive a new transform matrix, not only should you update the matrix, but you should also explicitly call the render function to refresh the visualization. In VTK (the visualization library that 3D Slicer is built upon), this can typically be done using the Render() method.
Observer Pattern: If your problem is indeed related to certain events, you can leverage the observer pattern. Create an observer that listens for changes in the transform matrix and calls the rendering function whenever an update is received. In VTK and 3D Slicer, this could be accomplished using the AddObserver() method with an appropriate event like vtkCommand::ModifiedEvent or vtkMRMLTransformableNode::TransformModifiedEvent.
Check for DataProbe.py or CrosshairNode Events: If you suspect these components are causing the updates, review their event logic to see if there’s an event you can leverage to trigger the rendering of your model more frequently.
Please keep in mind that calling the render function too frequently can consume considerable computational resources, potentially slowing down your module or even the whole 3D Slicer application. You’ll need to balance the need for fast updates with the performance requirements of your module and 3D Slicer.
"
Unfortunately, it does not work, because my tool is represented by a Model and thus it is not a LabelMap node. Thanks for the suggestion, though.
Regarding the GPT-4 response you provided, I tried the 3 possible solutions, but none of them worked for me.
I think I may have identified the origin of the problem but I do not know how to solve it. I’ll try to provide additional context so the problem is described more in-depth:
I am receiving the new tool transform via an OPCUA connection, using the asyncua Python library. My bad I forgot to mention that I am developing my module in Python.
The transform is received in the 3DSlicer module by subscribing the client (3DSlicer) to a variable of the OPCUA Server, which holds the data I am interested in (the tool transform). Whenever this variable changes in the server, the client should enter a function called “datachange_notification” which is in charge of updating the model properties. I noticed that the execution does not enter “datachange_notification” until I manually interact with the mentioned slices by hovering the mouse over them.
I suspect that asyncua has a thread being blocked, which cannot advance until I trigger some kind of event/interaction with the GUI. I want to unblock this automatically, without interacting with the GUI, so the 3D visualization is updated in real-time, automatically.
Find below attached a sample Python Interactor message whenever 3DSlicer detects a change in the variable which is subscribed to:
This is returned when I open the connection, no GUI interaction needed:
An approach would be (my brain ) to use a QTimer in your main thread that periodically checks for updates in your tool transform and refreshes the GUI. See a QTimer example here.
Or (suggested by GPT-4) you could define a Qt signal in your main thread that’s connected to a slot function responsible for updating the model in the GUI. You emit this signal from your asyncua callback function, and thanks to Qt’s queued connections, the slot function will be executed in the main thread. Qt’s signal-slot system is thread-safe, and it ensures that the GUI updates are performed in the correct thread.
Probably not working, GPT ! … if the asyncua callback function does not work.
I agree with @rbumm that a QTimer based solution should work. Ideally though the application should be fully event driven so that operating system level activity wakes up the application when there’s work to be done. Timers are a kind of “busy loop” that can be inefficient.
Slicer uses Qt’s event loop, while python’s asyncio introduces a different event loop. Generally Qt’s is sufficient and you might be able to use something like QSocketNotifier if you’re getting your data via a network connection. If you really need to use both event loops you may explore something like qasync but I don’t know if it’s been used in Slicer.
Based on the gif you attached the parent transform of the model is changed as expected. So what I assume is that the needle model node has a parent transform, the matrix of which is updated from the data received from OPCUA. This transform modification should be enough to trigger rendering I think. As a test and possible workaround, can you call Modified() on the needle model’s display node?
I would recommend to write a small component that converts from OPCUA to OpenIGTLink. The simplest is probably to write it in Python - receive messages using python-opcua and send them using pyigtl. The entire implementation may be just 5-10 lines. Since this bridge runs in simple while loop in a separate process, you don’t need to deal with events, timers, multithreading, etc.
Thanks to everyone that participated in the topic trying to provide solutions to the issue. I tried many of your suggestions but I could not make any of them work. Sorry for the late reply, I have been on a long-awaited vacation.
Regarding QTimers, conceptually they do not meet the requirements of our use case, as they need to match the refresh rate of the server. We did not contemplate using them and another side project that used them brought problems to our application. In addition, we want to work fully event-driven, so as @pieper pointed out, this could be inefficient.
I tried it but it did not work, thank you for your suggestion though!
As for using QSocketNotifier, I could not achieve the object to “listen” to the server socket. The app does not show any render changes until I manually interact with the slice views. Perhaps I did not correctly configure the QSocketNotifier.
We found this as the best fit for our use case, but we did not fully comprehend your suggestion, @lassoan. Is the component running in our custom Slicer module? How are OPCUA messages translated to OpenIGTLink? We are constrained to using OPCUA as the main communication protocol between our system and the Slicer module. Could you extend on your answer please?
After some thought, we arrived at a possible solution. We could launch a separate process from our custom Slicer module that receives the OPCUA messages and translates them to OpenIGTLink, so they can be forwarded to the main process of the Slicer application. Using pyigtl, we could receive these messages. Is this close to what you were trying to convey? Do you find it feasible?
I created a small IGTL server-client pair based on the examples of pyigtl.
Server side code:
import pyigtl
from time import sleep
import numpy as np
server = pyigtl.OpenIGTLinkServer(port=18944, local_server=False)
print(f"Starting server at: {server.host}:{server.port}")
timestep = 0
while True:
if not server.is_connected():
# Wait for client to connect
sleep(0.1)
continue
print("Server connected")
# Generate transform
timestep += 1
# Send transform
transform = np.random.rand(4, 4) @ np.eye(4)
print(f"time: {timestep} transform: ({transform.ravel()})")
transform_message = pyigtl.TransformMessage(transform, device_name="MyTransform")
server.send_message(transform_message, wait=True)
sleep(1)
# Since we wait until the message is actually sent, the message queue will not be flooded
After starting the server and client pair, each in a different Python process, on the client side, the execution gets halted in the line that defines input_message. If I adjust the timeout to a value other than -1, the client receives the message once, and the execution finishes. In 3DSlicer, I want to update the transform every time I get the message, and I do not wish the module to hang in the line that defines input_message.
In addition, I do not want to use 3DSlicer’s IGTLinkF module, I would like to manage the connection entirely from the code of my module. Can IGTLinkF’s logic be replicated using pyigtl?
I don’t think it is feasible to match the performance of the OpenIGTLinkIF module in Slicer using Python scripting. OpenIGTLinkIF module is a result of over 15 years of development, usage, and optimization on many projects, on all supported operating systems. You can decide to implement a new Python scripted module for this, but then of course we won’t be able to help you with that.
On the other hand, is there any way to access the IGTLinkF logic from Python? This way I could use its functionality from my code. The ultimate objective would be to eliminate the user interaction with the IGTLinkF module, and only use our custom module.
Sorry that I formulated my question poorly in my previous post, this is what I meant.
We finally managed to get the communication up and running fluently, and it runs super smoothly! I would like to express my biggest gratitude to all the people that contributed to solving our problem
In case anyone faces the same problem (or a similar one) find below a summarized description of how we implemented the solution:
The architecture is described in a diagram in one of my previous replies:
Every time a new connection is requested to be opened from the custom 3DSlicer module, a new process that runs the OPCUAtoIGTL translator is launched from it, using slicer.util.launchConsoleProcess(commandOfInterest). commandOfInterest is built concatenating the PythonSlicer.exe path (in the case of Windows systems), the absolute path to the OPCAtoIGTL translator script, and the necessary CLI arguments that are fed to it.
The OPCUAtoIGTL translator does the following:
–It receives the OPCUA ServerIP and port, and the IGTL Server port as CLI arguments. Then it parses them.
– After parsing, an OPCUA client is set up (not connected yet) to connect to the OPCUA Server.
– The IGTLink Server is initiated in 127.0.0.1:<parsedPortNumber>
– Only, and only when the IGTLink Server is connected to the IGTLink client (which is set up in the Custom 3DSlicer Module), the OPCUA Client connects to the OPCUA Server.
– The OPCUA Client subscribes to a variable of interest, which the OPCUA Server is constantly updating. The variable value is processed in the OPCUAtoIGTL translator (so, actually, it is translated)
– To avoid flooding any buffers and 3DSlicer hanging without updating the transform, (this happened to me), the IGTL Server sends an update every 100ms, which is an acceptable frame rate for navigation purposes.
If the user wants to close the connection, the OPCUAtoIGTL process is killed, and the IGTL Client is deleted in the 3DSlicer scene. This way the process list and the scene node tree remain clean.
Feel free to message me in case you need more details. Special thanks again to @lassoan for the help
Thanks for the update, it is great to hear that everything worked out well!
In case someone else wants to connect Slicer to OPCUA (or similar protocol) - could you post here the link to the github repository of your OPCUAtoIGTL translator Python script?
