# Upgrading Endoscopy extension with camera interpolation

**URL:** <https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546>\
**Category:** Development\
**Tags:** extensions\
**Created:** [May 23, 2022, 6:43pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546 "2022-05-23T18:43:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jadh4v](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jadh4v/32/65504_2.png) [@jadh4v](https://discourse.slicer.org/u/jadh4v)\
**Post date:** [May 23, 2022, 6:43pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/1 "2022-05-23T18:43:09Z")

</div>

In our work towards developing a camera flythrough visualization for a Slicer based custom application [VPAW](https://github.com/KitwareMedical/vpaw), we are considering keyframe interpolation of camera poses for creating a flythrough.

We considered multiple existing options as discussed in issue [KitwareMedical/vpaw#9](https://github.com/KitwareMedical/vpaw/issues/9), to converge on two extensions that consist most of the functionalities that we require.

_Endoscopy_ is the obvious choice as it provides a camera flythrough over a user-defined curve. However, it doesn’t provide camera pose interpolation over this curve and only follows the curve tangentially.

### Proposed plan

We are thinking of upgrading the Slicer build-in [Endoscopy](https://slicer.readthedocs.io/en/latest/user_guide/modules/endoscopy.html) module to allow defining camera poses over the curve and interpolation using these “keyframes”.

There is an existing C++ extension, [CameraPath](https://github.com/KitwareMedical/Slicer-CameraPath), that contains some of these functions that can either be imported and re-used in _Endoscopy_ or re-implement in it. We would like your opinion on the same.

**Question:** Would it make sense to promote _CameraPath_ as a core Slicer module and then import the logic to upgrade _Endoscopy_? Or Simply re-implement the functions in _Endoscopy_ itself?

In some sense _Endoscopy_ is a special case of _CameraPath_. _CameraPath_ will define a fixed flythrough path and interpolated camera poses along it, while _Endoscopy_ utilizes such a path / camera interpolation for a more interactive navigation and visualization of data.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [May 23, 2022, 7:13pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/2 "2022-05-23T19:13:48Z")

</div>

Glad you are able to work on this @jadh4v 👍

To me it would make sense to add some pure-VTK camera interpolation functionality, perhaps just one class based on [`vtkQuaternionInterpolator`](https://vtk.org/doc/nightly/html/classvtkQuaternionInterpolator.html#details) with some extra code to handle changes in field of view and maybe other parameters. This could go in vtkAddon if it’s C++ or in Endoscopy logic if it could be done in python (I’d think it could).

Then in the Endoscopy module I think it would be pretty straightforward to allow the user to camera keyframes along the path and then they could be interpolated during the flythrough. The exact UI would take a bit of thought, but if we constrain the options it should be usable and not too confusing for the user.

It would also make sense for the Endoscopy module to export to a Sequence, like the Animator does, so that it could be recorded by the ScreenCapture logic.

---

<div class="post-metadata">

**Author:** ![hherhold](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/hherhold/32/12199_2.png) [@hherhold](https://discourse.slicer.org/u/hherhold)\
**Post date:** [May 23, 2022, 7:39pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/3 "2022-05-23T19:39:43Z")

</div>

I’m not familiar with the CameraPath module, and my recent attempts with Endoscopy haven’t gone well, but a few thoughts on what would make this sort of thing useful. I think at a minimum, the Camera position, Camera look point, and Camera orientation (up vector) should be key-framable. Currently (I believe) endoscopy assumes the camera looks down the path, which is not always optimal.

I think a real issue here could be feature creep. It’s very easy to say “well, VG Studio does this or Drishti does that”, which is nice, but I find if I truly need full animation capabilities, I just use Blender. There should be a “sweet spot” of features that will satisfy a large Slicer audience without being onerous or requiring lots of feature additions.

My 2 cents…

---

<div class="post-metadata">

**Author:** ![hherhold](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/hherhold/32/12199_2.png) [@hherhold](https://discourse.slicer.org/u/hherhold)\
**Post date:** [May 23, 2022, 7:40pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/4 "2022-05-23T19:40:53Z")

</div>

Sorry - should have added that yes, I know a number of people (myself included) who would be VERY interested in any work on this!

---

<div class="post-metadata">

**Author:** ![jadh4v](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jadh4v/32/65504_2.png) [@jadh4v](https://discourse.slicer.org/u/jadh4v)\
**Post date:** [May 24, 2022, 2:15pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/5 "2022-05-24T14:15:01Z")

</div>

Thanks for the reply. Using `vtkAddon` to create the interpolation functionality of camera pose sounds good to me.

**To flesh-out the UI part a bit more:**

1. We could use the 3D view camera state itself as a way to insert keyframes. Initially we were considering having a [flight mode](https://github.com/KitwareMedical/Slicer/commit/9af308920309ec8cef890b98078574ef36de3737) for the 3D view to assist in flying around and placing keyframes, but since we can define the path curve first, it is probably not needed. The user can just fly along the path and stop and insert keyframes wherever necessary.

2. Regarding markup / display of keyframes we can simply use a camera mesh model to indicate the position of the interested keyframes along the path, unless there are any existing suitable markups/glyphs for this purpose.

3. Along with camera pos and fov, would it make sense to also interpolate volume property, etc for better local control of visualization?

4. Should we provide lighting fixed on the camera that moves with it?

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [May 24, 2022, 2:20pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/6 "2022-05-24T14:20:58Z")

</div>

> [@jadh4v](#):
>
> Along with camera pos and fov, would it make sense to also interpolate volume property, etc for better local control of visualization?

The Animator module [has some code for this](https://github.com/SlicerMorph/SlicerMorph/blob/master/Animator/Animator.py#L277-L434) and it would be nice to either reuse or generalize this feature. Currently it requires that the transfer functions have the same number of control points at the keyframes.

> [@jadh4v](#):
>
> Should we provide lighting fixed on the camera that moves with it?

Yes, I would think so - a headlight makes sense for an endoscopy simulation.

---

<div class="post-metadata">

**Author:** ![jadh4v](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jadh4v/32/65504_2.png) [@jadh4v](https://discourse.slicer.org/u/jadh4v)\
**Post date:** [May 24, 2022, 4:29pm UTC](https://discourse.slicer.org/t/upgrading-endoscopy-extension-with-camera-interpolation/23546/7 "2022-05-24T16:29:37Z")

</div>

Thanks, that is great to hear. I do appreciate the point of feature creep. While this started as a way to satisfy our requirements of the custom app, the endoscopy extension also feels a little lacking in its current state. As you mentioned, it only flies tangentially to the path. So we are also thinking from the point of view of justification of the name “Endoscopy” for the extension and what are the minimum set of features that are essential to it to be useful.