Always display annotation or model on top in 3D view

Prashant_Pandey · November 13, 2019, 4:04am

I’m using volume rendering to show a CT volume being cut with a tracked tool. The scene also contains a sugical target, which is represented by a cylinder model and an annotation ruler. Is it possible to show the model and ruler so that they are always visible, even when ‘occluded’ by the rendering?

I’ve attached two images to show what I mean. In the first one the model and annotation ruler are visible because the rendering is clipped ‘behind’ the target. In the second it’s not possible to see the model/ruler becayse the volume is clipped ‘in front’ of the target. I’ve played around with rendering opacity but this does not provide a good visual of the model unless the opacity is very low.

Prashant_Pandey · November 18, 2019, 8:45pm

@lassoan Any ideas on how to get started on this?

lassoan · November 18, 2019, 9:49pm

You can get access to the VTK actor and change its rendering layer. Of course, the application does not expect such low-level manipulation, so there might be problems with picking, placing points, etc.

In general, I would avoid such 2.5D view modes and use proper 3D and 2D views instead (you can tune cropping and transparency settings to make sure everything important is visible).

For aligning tool with a trajectory you can show simplified axial and side views of just the tool and the trajectory (bullseye and progress views).

Prashant_Pandey · November 19, 2019, 12:13am

Okay thanks. I know it’s not ideal to do this kind of visualization, but the purpose is actually to compare it to more ‘conventional’ views such as 2D bullseye and progress views which we have already implemented.

From my understanding, 3D views are not very useful for precise navigation but rather rough global navigation (unless you have suggestions for what works well in 3D), so I am trying to make these views more ‘useful’ by augmenting with 2D details such as bullseye or the trajectory projection mentioned in the top post.

Is there a brief description or explanation of how to get access to the vtk actor?

lassoan · November 19, 2019, 1:03am

You can get the model displayable manager as described in this example in the script repository, then get vtkActor corresponding to a model display node.

pieper · November 19, 2019, 2:08am

@Prashant_Pandey another approach might be to provide a custom shader that cuts away the part of the volume that is obscuring the annotation. Something like this example: https://www.youtube.com/watch?v=yiEI_yBMu8k (from these experiments)

Prashant_Pandey · December 4, 2019, 8:54pm

I got round to trying your suggestions. After getting the vtkActor for the target model display node, I can’t seem to find a way to change the rendering layer.

I also tried modelDisplayManager.GetRenderer().SetLayer(int), but this didn’t effect the 3D view in the expected way.
Any tips?

lassoan · December 5, 2019, 12:31am

You need to add a new renderer, set its layer, and move the actor to that new renderer. You may need to update the camera of the new renderer based on the camera of the default layer’s renderer. This is all very fragile but may allow you to confirm that this kind of 2.5-dimensional visualization is clinically useful and worthy of spending time with implementing properly.

Prashant_Pandey · December 6, 2019, 12:55am

I added a new renderer with the modelDisplayNode as an actor, and then played around with the new and old renderer’s layers, but with no luck. I did get some interesting views but not what I was looking for:

lassoan · December 6, 2019, 1:31am

You can also display line as 2D actor, which may make it easier to ensure it is above the 3D geometry, without introducing extra renderers or layers.

Prashant_Pandey · December 10, 2019, 3:16am

That sounds like a promising approach.

I could use something like

actor2D = vtk.vtkAxisActor2D()
actor2D.SetPoint1(x1,y1)
actor2D.SetPoint2(x2, y2)
renderer = slicer.app.layoutManager().threeDWidget(0).threeDView().renderWindow().GetRenderers().GetFirstRenderer()
renderer.AddActor2D(actor2D)

But is there an easy way to calculate the world to display coordinates for the 3D annotation ruler (x1,y1, x2,y2 for the X,Y,Z ruler coordinates)?

lassoan · December 10, 2019, 3:35am

You can convert from world to display like this:

github.com

Slicer/Slicer/blob/63e781dafd5a353cb37ca3f32358bdaa46bd4163/Modules/Loadable/Markups/VTKWidgets/vtkSlicerMarkupsWidgetRepresentation3D.cxx#L300-L302


this->Renderer->SetWorldPoint(centerPosWorld);
this->Renderer->WorldToDisplay();
this->Renderer->GetDisplayPoint(centerPosDisplay);

Prashant_Pandey · December 10, 2019, 8:19pm

Thanks, got it to work! What’s the correct way of adding an oberver to the renderer node? I’d like the 2D line overlay to change if the camera moves or if the viewport size changes for whatever reason.

lassoan · December 10, 2019, 8:47pm

Probably you just need to add an observer to the camera.

Prashant_Pandey · December 10, 2019, 9:29pm

I ended up adding an observer to the 3D view interactor. Here is the full code for reference:

import numpy as np
layoutManager = slicer.app.layoutManager()
view = layoutManager.threeDWidget(0).threeDView()
renderWindow = view.renderWindow()
renderers = renderWindow.GetRenderers()
renderer = renderers.GetItemAsObject(0)
interactor = slicer.app.layoutManager().threeDWidget(0).threeDView().interactor()

try:
  interactor.RemoveObserver(rendObs)
  renderer.RemoveActor(slicer.lineActor)
  print("Removed Observer")
except:
  pass

slicer.lineActor = vtk.vtkAxisActor2D()
slicer.lineActor.TickVisibilityOff()
slicer.lineActor.SetLabelVisibility(0)
slicer.lineActor.GetProperty().SetLineWidth(2.0)
renderer.AddActor2D(slicer.lineActor)

def onRenderChange(caller, event):
  renderer.SetWorldPoint(56.685, -57.574, -6.521, 1)
  renderer.WorldToDisplay()
  dispPoint1 = renderer.GetDisplayPoint()
  renderer.SetWorldPoint(24.075, -54.898, -8.065, 1)
  renderer.WorldToDisplay()
  dispPoint2 = renderer.GetDisplayPoint()
  viewSize = renderer.GetSize()
  dispPoint1 = list(np.asarray(dispPoint1[:2])/np.asarray(viewSize))
  dispPoint2 = list(np.asarray(dispPoint2[:2])/np.asarray(viewSize))
  slicer.lineActor.SetPoint1(dispPoint1[0], dispPoint1[1])
  slicer.lineActor.SetPoint2(dispPoint2[0], dispPoint2[1])
  print("Render Modified")

rendObs = interactor.AddObserver(vtk.vtkCommand.ModifiedEvent, onRenderChange, 1.0)

lassoan · December 10, 2019, 9:30pm

Great! Can you attach a screenshot of the final result?

Prashant_Pandey · December 10, 2019, 9:42pm

Here are two screenshots and a video

https://drive.google.com/file/d/1CX13XfZ5j3uCAJVVcUEMdfMtW6TuvFYe/view?usp=sharing

lassoan · December 11, 2019, 3:41am

Thank you, these pictures and video look nice and visibility of the current and planned trajectory are good. I guess the main question if the floating lines still offer enough 3D cues. Maybe you can show the trajectories as true 3D lines (tubes) and make the 2D floating lines semi-transparent. This way you would still have strong 3D cues and keep the trajectories always visible. Keep us updated how well it works!