GPU-based Projective Texture Mapping in a Loadable Module

Development
,
1

Hello everyone,

I want to create my first Slicer extension. At the moment I am doing research on the design of the extension (mostly reading Slicer API documentation), after this is done I’ll inform you about my plan.

Task

One loadable module that I want to include, requires me to perform Projective Texture Mapping. In short, this is the projection of a 2D image onto a 3D surface. Because the texture and its position in the scene is changing in real time, I decided to implement this algorithm with an GPU-based approach similar to Shadow Mapping. Therefore I need to perform two render passes:

  1. render the 3D surface with a viewport that is aligned with the 2D image in world space to obtain a depth buffer
  2. render the 3D surface with the desired viewport, project its fragments/vertices onto the 2D image to obtain the color and only draw them if their distance to the image plane matches the one in the depth buffer

Problem

Unfortunately implementing such a custom rendering processes isn’t that easy as I thought it would be. If I were to implement this with VTK, I would try to replace the shaders of vtkOpenGLPolyDataMapper as described here.

As far as I know slicer encapsulates VTK, so that there is no direct access to the underlying vtkMappers or vtkActors.

Does anyone have an idea on how to approach this problem?

What I’ve got so far

  • 2D RGB image represented as a vtkMRMLVectorVolumeNode
  • 3D surface represented as a vtkMRMLModelNode
  • orientation and position of the image and the surface represeneted as a vtkMRMLLinearTransformNode
  • obtain a vtkRenderer from the vtkMRMLModelDisplayableManager
  • add a new vtkActor with a vtkPolyDataMapper to the vtkRenderer
  • use custom shaders for the vtkPolyDataMapper

Open questions

  • Are there any easier alternatives?
  • How do i get the vtkMRMLModelDisplayableManager?
  • Do I need to add new Actors and Mappers to the obtained Renderer or can I use existing ones?
  • What is the best way to perform the 2 render passes?

Thank in advance,
Tobias

2

Model displayable manager can already apply texture to a model. See this module as an example: https://github.com/SlicerIGT/SlicerIGT/tree/master/TextureModel

I guess VTK can compute the texture coordinates, so you should be able to implement projective mapping.

If this method is not fast enough or there are other limitations then you have to implement a custom displayable manager. You can use the model displayable manager as a starting point.

1 Like
3

I’d suggest getting this to work in pure VTK first with a simple example and then you can adapt it to work in Slicer.

Did you look at this class?

http://www.vtk.org/doc/nightly/html/classvtkProjectedTexture.html

You can also have a look at how vtk8 with the OpenGL2 back end exposes render passes.

It seems that to do projected textures correctly you probably need to use some shader code - there’s some nice detail here:

4

It seems that this class can compute texture coordinates that the model displayable manager needs for texture display. So, you can get everything working with a short Python script, which sets up the vtkProjectedTexture filter parameters. Of course, this will be slower than a direct shader implementation but may be already usable.