In our work towards developing a camera flythrough visualization for a Slicer based custom application VPAW, we are considering keyframe interpolation of camera poses for creating a flythrough.
We considered multiple existing options as discussed in issue KitwareMedical/vpaw#9, to converge on two extensions that consist most of the functionalities that we require.
Endoscopy is the obvious choice as it provides a camera flythrough over a user-defined curve. However, it doesn’t provide camera pose interpolation over this curve and only follows the curve tangentially.
Proposed plan
We are thinking of upgrading the Slicer build-in Endoscopy module to allow defining camera poses over the curve and interpolation using these “keyframes”.
There is an existing C++ extension, CameraPath, that contains some of these functions that can either be imported and re-used in Endoscopy or re-implement in it. We would like your opinion on the same.
Question: Would it make sense to promote CameraPath as a core Slicer module and then import the logic to upgrade Endoscopy? Or Simply re-implement the functions in Endoscopy itself?
In some sense Endoscopy is a special case of CameraPath. CameraPath will define a fixed flythrough path and interpolated camera poses along it, while Endoscopy utilizes such a path / camera interpolation for a more interactive navigation and visualization of data.
To me it would make sense to add some pure-VTK camera interpolation functionality, perhaps just one class based on vtkQuaternionInterpolator with some extra code to handle changes in field of view and maybe other parameters. This could go in vtkAddon if it’s C++ or in Endoscopy logic if it could be done in python (I’d think it could).
Then in the Endoscopy module I think it would be pretty straightforward to allow the user to camera keyframes along the path and then they could be interpolated during the flythrough. The exact UI would take a bit of thought, but if we constrain the options it should be usable and not too confusing for the user.
It would also make sense for the Endoscopy module to export to a Sequence, like the Animator does, so that it could be recorded by the ScreenCapture logic.
I’m not familiar with the CameraPath module, and my recent attempts with Endoscopy haven’t gone well, but a few thoughts on what would make this sort of thing useful. I think at a minimum, the Camera position, Camera look point, and Camera orientation (up vector) should be key-framable. Currently (I believe) endoscopy assumes the camera looks down the path, which is not always optimal.
I think a real issue here could be feature creep. It’s very easy to say “well, VG Studio does this or Drishti does that”, which is nice, but I find if I truly need full animation capabilities, I just use Blender. There should be a “sweet spot” of features that will satisfy a large Slicer audience without being onerous or requiring lots of feature additions.
Thanks for the reply. Using vtkAddon to create the interpolation functionality of camera pose sounds good to me.
To flesh-out the UI part a bit more:
We could use the 3D view camera state itself as a way to insert keyframes. Initially we were considering having a flight mode for the 3D view to assist in flying around and placing keyframes, but since we can define the path curve first, it is probably not needed. The user can just fly along the path and stop and insert keyframes wherever necessary.
Regarding markup / display of keyframes we can simply use a camera mesh model to indicate the position of the interested keyframes along the path, unless there are any existing suitable markups/glyphs for this purpose.
Along with camera pos and fov, would it make sense to also interpolate volume property, etc for better local control of visualization?
Should we provide lighting fixed on the camera that moves with it?
The Animator module has some code for this and it would be nice to either reuse or generalize this feature. Currently it requires that the transfer functions have the same number of control points at the keyframes.
Yes, I would think so - a headlight makes sense for an endoscopy simulation.
Thanks, that is great to hear. I do appreciate the point of feature creep. While this started as a way to satisfy our requirements of the custom app, the endoscopy extension also feels a little lacking in its current state. As you mentioned, it only flies tangentially to the path. So we are also thinking from the point of view of justification of the name “Endoscopy” for the extension and what are the minimum set of features that are essential to it to be useful.
