Always display annotation or model on top in 3D view

Support
1

I’m using volume rendering to show a CT volume being cut with a tracked tool. The scene also contains a sugical target, which is represented by a cylinder model and an annotation ruler. Is it possible to show the model and ruler so that they are always visible, even when ‘occluded’ by the rendering?

I’ve attached two images to show what I mean. In the first one the model and annotation ruler are visible because the rendering is clipped ‘behind’ the target. In the second it’s not possible to see the model/ruler becayse the volume is clipped ‘in front’ of the target. I’ve played around with rendering opacity but this does not provide a good visual of the model unless the opacity is very low.

image
image1299×701 182 KB

image
image1271×652 191 KB

2

@lassoan Any ideas on how to get started on this?

3

You can get access to the VTK actor and change its rendering layer. Of course, the application does not expect such low-level manipulation, so there might be problems with picking, placing points, etc.

In general, I would avoid such 2.5D view modes and use proper 3D and 2D views instead (you can tune cropping and transparency settings to make sure everything important is visible).

For aligning tool with a trajectory you can show simplified axial and side views of just the tool and the trajectory (bullseye and progress views).

4

Okay thanks. I know it’s not ideal to do this kind of visualization, but the purpose is actually to compare it to more ‘conventional’ views such as 2D bullseye and progress views which we have already implemented.

From my understanding, 3D views are not very useful for precise navigation but rather rough global navigation (unless you have suggestions for what works well in 3D), so I am trying to make these views more ‘useful’ by augmenting with 2D details such as bullseye or the trajectory projection mentioned in the top post.

Is there a brief description or explanation of how to get access to the vtk actor?

5

You can get the model displayable manager as described in this example in the script repository, then get vtkActor corresponding to a model display node.

6

@Prashant_Pandey another approach might be to provide a custom shader that cuts away the part of the volume that is obscuring the annotation. Something like this example: https://www.youtube.com/watch?v=yiEI_yBMu8k (from these experiments)

2 Likes
7

I got round to trying your suggestions. After getting the vtkActor for the target model display node, I can’t seem to find a way to change the rendering layer.

I also tried modelDisplayManager.GetRenderer().SetLayer(int), but this didn’t effect the 3D view in the expected way.
Any tips?

8

You need to add a new renderer, set its layer, and move the actor to that new renderer. You may need to update the camera of the new renderer based on the camera of the default layer’s renderer. This is all very fragile but may allow you to confirm that this kind of 2.5-dimensional visualization is clinically useful and worthy of spending time with implementing properly.

9

I added a new renderer with the modelDisplayNode as an actor, and then played around with the new and old renderer’s layers, but with no luck. I did get some interesting views but not what I was looking for:

image
image701×781 202 KB

10

You can also display line as 2D actor, which may make it easier to ensure it is above the 3D geometry, without introducing extra renderers or layers.

11

That sounds like a promising approach.

I could use something like

actor2D = vtk.vtkAxisActor2D()
actor2D.SetPoint1(x1,y1)
actor2D.SetPoint2(x2, y2)
renderer = slicer.app.layoutManager().threeDWidget(0).threeDView().renderWindow().GetRenderers().GetFirstRenderer()
renderer.AddActor2D(actor2D)

But is there an easy way to calculate the world to display coordinates for the 3D annotation ruler (x1,y1, x2,y2 for the X,Y,Z ruler coordinates)?

12

You can convert from world to display like this:

1 Like
13

Thanks, got it to work! What’s the correct way of adding an oberver to the renderer node? I’d like the 2D line overlay to change if the camera moves or if the viewport size changes for whatever reason.

14

Probably you just need to add an observer to the camera.

15

I ended up adding an observer to the 3D view interactor. Here is the full code for reference:

import numpy as np
layoutManager = slicer.app.layoutManager()
view = layoutManager.threeDWidget(0).threeDView()
renderWindow = view.renderWindow()
renderers = renderWindow.GetRenderers()
renderer = renderers.GetItemAsObject(0)
interactor = slicer.app.layoutManager().threeDWidget(0).threeDView().interactor()

try:
  interactor.RemoveObserver(rendObs)
  renderer.RemoveActor(slicer.lineActor)
  print("Removed Observer")
except:
  pass

slicer.lineActor = vtk.vtkAxisActor2D()
slicer.lineActor.TickVisibilityOff()
slicer.lineActor.SetLabelVisibility(0)
slicer.lineActor.GetProperty().SetLineWidth(2.0)
renderer.AddActor2D(slicer.lineActor)

def onRenderChange(caller, event):
  renderer.SetWorldPoint(56.685, -57.574, -6.521, 1)
  renderer.WorldToDisplay()
  dispPoint1 = renderer.GetDisplayPoint()
  renderer.SetWorldPoint(24.075, -54.898, -8.065, 1)
  renderer.WorldToDisplay()
  dispPoint2 = renderer.GetDisplayPoint()
  viewSize = renderer.GetSize()
  dispPoint1 = list(np.asarray(dispPoint1[:2])/np.asarray(viewSize))
  dispPoint2 = list(np.asarray(dispPoint2[:2])/np.asarray(viewSize))
  slicer.lineActor.SetPoint1(dispPoint1[0], dispPoint1[1])
  slicer.lineActor.SetPoint2(dispPoint2[0], dispPoint2[1])
  print("Render Modified")

rendObs = interactor.AddObserver(vtk.vtkCommand.ModifiedEvent, onRenderChange, 1.0)
16

Great! Can you attach a screenshot of the final result?

17

Here are two screenshots and a video :slight_smile:

vlcsnap-2019-12-10-13h37m38s145_cr
vlcsnap-2019-12-10-13h37m38s145_cr1478×890 277 KB
vlcsnap-2019-12-10-13h37m23s170
vlcsnap-2019-12-10-13h37m23s1701480×892 346 KB

2DLineOverlay.mp4 - Google Drive

18

Thank you, these pictures and video look nice and visibility of the current and planned trajectory are good. I guess the main question if the floating lines still offer enough 3D cues. Maybe you can show the trajectories as true 3D lines (tubes) and make the 2D floating lines semi-transparent. This way you would still have strong 3D cues and keep the trajectories always visible. Keep us updated how well it works!

19

Here’s an update for you - we published a preliminary study looking at the differences between these views: https://link.springer.com/chapter/10.1007/978-3-030-60946-7_1

Unfortunately the 3D view was still not as promising as some 2D approaches. However when we combined the 3D view with a 2D central bullseye the performance improved. We’re hoping to have a more extensive study completed soon.