# GPU-based Projective Texture Mapping in a Loadable Module

**URL:** <https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941>\
**Category:** Development\
**Tags:** extensions-manager, rendering\
**Created:** [August 25, 2017, 12:09am UTC](https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941 "2017-08-25T00:09:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![thaenel](https://avatars.discourse-cdn.com/v4/letter/t/9fc348/32.png) [@thaenel](https://discourse.slicer.org/u/thaenel)\
**Post date:** [August 25, 2017, 12:09am UTC](https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941/1 "2017-08-25T00:09:36Z")

</div>

Hello everyone,

I want to create my first Slicer extension. At the moment I am doing research on the design of the extension (mostly reading Slicer API documentation), after this is done I’ll inform you about my plan.

## Task

One loadable module that I want to include, requires me to perform [_Projective Texture Mapping_](https://en.wikipedia.org/wiki/Projective_texture_mapping). In short, this is the projection of a 2D image onto a 3D surface. Because the texture and its position in the scene is changing in real time, I decided to implement this algorithm with an GPU-based approach similar to [_Shadow Mapping_](https://en.wikipedia.org/wiki/Shadow_mapping#Algorithm_overview). Therefore I need to perform two render passes:

1. render the 3D surface with a viewport that is aligned with the 2D image in world space to obtain a depth buffer
2. render the 3D surface with the desired viewport, project its fragments/vertices onto the 2D image to obtain the color and only draw them if their distance to the image plane matches the one in the depth buffer

## Problem

Unfortunately implementing such a custom rendering processes isn’t that easy as I thought it would be. If I were to implement this with VTK, I would try to replace the shaders of [`vtkOpenGLPolyDataMapper`](http://www.vtk.org/doc/nightly/html/classvtkOpenGLPolyDataMapper.html) as described [here](http://www.vtk.org/Wiki/Shaders_In_VTK).

As far as I know slicer encapsulates VTK, so that there is **no direct access to the underlying `vtkMappers` or `vtkActors`**.

Does anyone have an idea on how to approach this problem?

## What I’ve got so far

- 2D RGB image represented as a `vtkMRMLVectorVolumeNode`
- 3D surface represented as a `vtkMRMLModelNode`
- orientation and position of the image and the surface represeneted as a `vtkMRMLLinearTransformNode`
- obtain a `vtkRenderer` from the `vtkMRMLModelDisplayableManager`
- add a new `vtkActor` with a `vtkPolyDataMapper` to the `vtkRenderer`
- use custom shaders for the `vtkPolyDataMapper`

## Open questions

- Are there any easier alternatives?
- How do i get the `vtkMRMLModelDisplayableManager`?
- Do I need to add new Actors and Mappers to the obtained Renderer or can I use existing ones?
- What is the best way to perform the 2 render passes?

Thank in advance,  
Tobias

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 25, 2017, 2:48am UTC](https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941/2 "2017-08-25T02:48:49Z")

</div>

Model displayable manager can already apply texture to a model. See this module as an example: [https://github.com/SlicerIGT/SlicerIGT/tree/master/TextureModel](https://github.com/SlicerIGT/SlicerIGT/tree/master/TextureModel)

I guess VTK can compute the texture coordinates, so you should be able to implement projective mapping.

If this method is not fast enough or there are other limitations then you have to implement a custom displayable manager. You can use the model displayable manager as a starting point.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [August 25, 2017, 1:03pm UTC](https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941/3 "2017-08-25T13:03:16Z")

</div>

I’d suggest getting this to work in pure VTK first with a simple example and then you can adapt it to work in Slicer.

Did you look at this class?

[http://www.vtk.org/doc/nightly/html/classvtkProjectedTexture.html](http://www.vtk.org/doc/nightly/html/classvtkProjectedTexture.html)

You can also have a look at how vtk8 with the OpenGL2 back end exposes render passes.

It seems that to do projected textures correctly you probably need to use some shader code - there’s some nice detail here:

> <https://stackoverflow.com/questions/22732717/opengl-projective-texture-mapping-via-shaders>

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 25, 2017, 1:20pm UTC](https://discourse.slicer.org/t/gpu-based-projective-texture-mapping-in-a-loadable-module/941/4 "2017-08-25T13:20:08Z")

</div>

> [@pieper](#):
>
> Did you look at this class?
> 
> [VTK: vtkProjectedTexture Class Reference](http://www.vtk.org/doc/nightly/html/classvtkProjectedTexture.html)

It seems that this class can compute texture coordinates that the model displayable manager needs for texture display. So, you can get everything working with a short Python script, which sets up the vtkProjectedTexture filter parameters. Of course, this will be slower than a direct shader implementation but may be already usable.