Hello, I am trying to have a 3D view camera in Slicer follow the position and orientation of a tracked tool using SlicerIGT. By following the tutorials I have successfully tracked a tool and gone through the pivot and spin calibrations and can see the tracked tool and registered object moving through 3D space in correct relation to each other in the 3D Slicer views. I now want to set up the second 3D view camera to be looking out from the tip of the tool (like a laryngoscope camera).
I think this is what the “Bullseye” mode of the Viewpoint module of SlicerIGT should do, but I think I am maybe not using it correctly. I created a linear transform node, placed it as the parent transform node of the camera, and chose it as the “Camera positioning transform”. I wasn’t sure whether to have the parent transform of that camera transform be the “ToolTipToTool” transform or the “ToolToReference” transform, so I have tried both. I chose 6DOF and did not change the zoom or translation settings and left “Parallel projection” unchecked. I clicked “Enable Bullseye View Mode”, and the camera view did successfully move around with the tool, but the location and orientation of the view was not what I wanted. Instead of looking outward towards in the direction the tool pointed, the camera looked backwards along the shaft of the tool. When I tried some simple modifications to the camera parent transform (like changing signs of the ones on the diagonal) I got changes in view orientation, but also undesired changes in camera position.
I’m sure all I need is a transform that points the camera in the direction I want in the location I want relative to the tool, and then the parent tracked transforms will move things around as I would like, but I’m having trouble figuring out how to get this set up properly. I’m also not sure whether I need to (in addition to having the proper parent transform on the camera) adjust the camera FocalPoint or ViewUp direction, or whether these will basically just be part of the parent transform. It seems like the Bullseye View Mode is probably designed to set the camera up more or less in the way I want, but I’m not sure what I am doing wrong in using it, and I can’t find any documentation which would help me sort through it. Specifically, I want to know:
what should the transform hierarchy look like?
which transform in that hierarchy should be chosen as the “camera positioning transform” in the Viewpoint module?
what should happen when I enable Bullseye View Mode?
what should happen when I disable it again? (in my experiments so far, disabling seems to have no effect, the camera continues to move with the tracked tool (because the tracked transform is still above the camera in the transform hierarchy))
do I need to do anything to change or reset the FocalPoint or ViewUp direction on the camera before or after using Bullseye Mode?
Also, if these questions would be better asked somewhere else, please let me know and I will ask there.
Thr Viewpoint module was implemented for exactly this purpose.
Do not place the camera under a transform. Viewpoint module takes care of setting up the camera and putting the camera under a transform will probably interfere with it.
what should the transform hierarchy look like?
The Viewpoint module does not require any additional transform. You just need the tooltip to world transform hierarchy.
which transform in that hierarchy should be chosen as the “camera positioning transform” in the Viewpoint module?
ToolTipToX (usually ToolTipToTool, which is under ToolToReference, which is under ReferenceToRas).
what should happen when I enable Bullseye View Mode?
The camera works as an endoscopic camera mounted on the tool.
what should happen when I disable it again?
The camera stops moving.
do I need to do anything to change or reset the FocalPoint or ViewUp direction on the camera before or after using Bullseye Mode
You can slightly adjust the camera, but usually not needed.
Thanks for the guidance. I have now tried using the Bullseye mode exactly as you suggest, and the results are as follows: The camera moves as if it were located at the tool tip, as desired, but it looks backwards up the shaft of the tool rather than forwards out from the tip of the tool. This is after performing the pivot and spin calibrations on the tool, doing fiducial point registration as outlined in tutorial U-12 and verifying that the spatial relationships in Slicer between the tracked tool (just the default ball and stick model) and the imported surface model with fiducials (representing the physical object) correspond to their relative locations and orientations in physical space (just saying that the calibration and registration seems to have worked well). The ball end of the stick represents the tool tip, correct? I would be happy to share an mrb file if that would be helpful, but I don’t know if it would be useful or not without the live tracking.
You can record live tracking data using Sequences module and then you have everything in one file that you can replay using Sequences module. See U-11 SlicerIGT Tutorial.
Use the needle model that is generated by Create Models module in SlicerIGT extension. The pointer model that OpenIGTLinkIF module creates may use a different direction for the third axis of the StylusTip coordinate system. You can easily swap the z axis direction in the Pivot calibration module.
@mikebind Were you ever able to get this to work? I am having the same problem and although I can switch the z axis using the Pivot Calibration module that only switches the body of the needle, never the way the camera is facing. If you were able to figure this out I’d be interested to learn how you did it. Thanks!
I will let you know if I get it working. I’m only at the location where this equipment is intermittently, and have not gotten back to this aspect of this project yet. Hopefully within the next week or so I’ll be able to give this a try.
What is the best way to receive the ToolTipToTool-transformation matrix from a Medtronic StealthStation (S7) (electromagnetical tracking) via PlusServer and openIGTlink?
The “XToTool”-transformation obviously already delivers the “XToToolTip” Transformation. Is it possible to to write the configfile in a way that it delivers the ToolTipToTool-transformation-matrix?
You can incorporate transformation matrices from calibrations in a <CoordinateDefinitions> section of your configuration file. Here is an example of what that section could look like, with two supplied matrices, one from a fiducial calibration connecting a fixed sensor to an image volume space, and one which is just used to swap coordinate axis directions to align the coordinate axes of a needle model with the sensor directions.
To get these transformations (or concatenated transformations which use these) sent to Slicer, you also need to include appropriately named <Transform> elements to the <PlusOpenIGTLinkServer> section. For example, this is what mine looks like:
I’m not sure if this addresses what you are asking about. I am working with an electromagnetic rather than optical tracker, but I assume the system is generally similar and hope that some of the above is helpful.
As a workaround one can define an additional matrix, which is basically a copy of the NeedleTipToStylusSensor, BUT the sign of the z-axis (+/-) is changed. Then take the minimum front/back-translation in the viewpoint-module (-0.01).