Support for keyframe based animation

[jcfr (Jean Christophe Fillion Robin (Kitware))]

@muratmaga

can benefit from some sort of a keyframe rendering

This reminds me of the following project put together by @agirault and @Andinet_Enquobahrie

- YouTube

Edit: Updated link to video publicly available.

Edit 2: More details are available on this page: Documentation/Labs/FlyThroughNavigation - Slicer Wiki

[agirault (Alexis Girault)]

@jcfr @muratmaga @hherhold I think we should open a separate thread for the keyframe animation so we can focus on answering @Dave_DeMarle question regarding photorealistic rendering, like @lassoan did here.

[pieper (Steve Pieper (Isomics, Inc.))]

iVSPlan looks cool

@muratmaga and I have been talking for a while about the possibility for keyframing support and I had in mind something kind of like what’s shown in the video, but I was also thinking of making it handle potentially any node, not just camera nodes. For example clipping planes and volume rendering transfer functions.

I was thinking of doing something based on the Sequences infrastructure, since that already has multi-node animation abilities. But what’s missing is the way to interpolate various node types between keyframes. I don’t have a specific implementation plan at this point, but I hope to try some experiments and report back with ideas about how to make something with a clean and extensible architecture.

[agirault (Alexis Girault)]

I had in mind something kind of like what’s shown in the video, but I was also thinking of making it handle potentially any node, not just camera nodes

I think that would make a lot of sense. Like you brought up, the interpolation is then the question (it was easy to use vtkSplines for the camera position and orientation). My understanding is that the main use case would be for the camera, so maybe it’s worth not over-engineering a fit-all solution but start building around the camera?

[pieper (Steve Pieper (Isomics, Inc.))]

I agree about not over-engineering - but still, I don’t think it’s enough to do just the camera. I’d thought about adding a virtual method like vtkMRMLDisplayableNode::Interpolate() so that any displayable could expose it’s own interpolation method but I don’t like that for a number of reasons (mainly because it would make MRML depend on Sequences but also because some animations should set up relations between objects - like the camera’s lookat is tied to another object’s location).

Instead I think the place to start is with some custom scripts that handle some specific animation styles, basically as an extension of the current ScreenCapture module.

For reference, here’s a video that @muratmaga suggested as an inspiration:

Here the things we would need to keyframe are:

• Camera parameters (or object transform parameters)
• volume rendering lookup table
• cutting plane or cropping ROI

These are all fairly easy values to interpolate.

[lassoan (Andras Lasso)]

We plan to merge Sequences extension features into Slicer core, so interpolation of any node will be available. If interpolation is implemented for a node then Sequence browser can use it quite easily. The main difficulty is to implement a simple yet powerful user interface. What is in Paraview or MITK seems to have taken a lot of work and still very hard to use.

If we need to change only one parameter at a time then we can easily add those as animation modes to screen capture module.

Camera animation is a special case, as you may not want to specify it by saving current camera positions as key frames, but instead by specifying a curve and maybe a look-at point. This can be managed in the endoscopy module. We should probably rename the module to something that reflects that it is a general “curve to transform” module (Camera path might be good, or maybe something even more general). We should update it to use the new curve widget, add option to export to sequence, etc.

[pieper (Steve Pieper (Isomics, Inc.))]

What’s the timeframe for merging Sequences to the core?

[lassoan (Andras Lasso)]

Sequences should be in its final place in Slicer5. Since Slicer5 is scheduled for next RSNA and we need time for stabilization and testing, Sequences should be moved by the end of summer.

[agirault (Alexis Girault)]

I’m obviously biased, but the interface I created for IVSPlan (linked by @jcfr in the top comment) was really intuitive and easy to use.

[pieper (Steve Pieper (Isomics, Inc.))]

@agirault agreed, it looks nice. Is the code available for reuse?

[agirault (Alexis Girault)]

Now yes!

With the help of @jcfr, we’ve pulled out the code for that module, added a LICENSE and README files, and published it to GitHub:

KitwareMedical/Slicer-CameraPath

JC described it in the README of the repository above, but it depends on an old fork of Slicer with some small changes (which also includes a “flying” as an option that could be toggled in the UI, which worked pretty well):

KitwareMedical/Slicer

Keep in mind that I wrote this as an intern who discovered Slicer 4 years ago

Also, here was the full demo I had created (14min+):

Hth

[jamesobutler (James Butler)]

I’ve seen in the past spam accounts that will copy and paste another user’s response. Likely to gain higher discourse forum privileges by making a post. I’m guessing the post/account was later deleted.

[lassoan (Andras Lasso)]

Yes, it was a spammer. But it accidentally did something useful by reviving the topic.

[pieper (Steve Pieper (Isomics, Inc.))]

Hi @agirault - thanks for sharing the example Just to confirm, this is not expected to compile with current slicer and you aren’t planning to work on it, right? I ask because with @muratmaga and @jcfr we’re trying to get a ballpark idea of the effort involved in getting the keyframing features in place.

[jcfr (Jean Christophe Fillion Robin (Kitware))]

Not expected to build against current Slicer.

I anticipate updating code and backporting changes to Slicer should be straightforward.

We will account for this in project with @muratmaga.

[agirault (Alexis Girault)]

Correct @pieper, this worked with Slicer 4.4, and I don’t have funding to update it to latest Slicer. Are you looking into the effort needed to implement this on top of Sequence, or to simply update it as-is to latest Slicer? I presume the later shouldn’t be much work, there weren’t many functionalities missing in Slicer.

[pieper (Steve Pieper (Isomics, Inc.))]

Thanks @jcfr and @agirault - yes, at this point we’re still trying to scope out all the options so we know what we have to work with and what to build off of. Yes, I’m thinking that we would want to build on top of Sequences but expose a very high level UI to create the specific types of animations described above (not just camera, but also volume rendering transfer functions and cropping) and a simple timeline editor.

[Dave_DeMarle]

I think the way this takes whatever camera positions you have and makes key frames from them for you is important. That is you move the camera the same way you normally do and then when you hit Add Keyframe the details are filled in for you. You have to hunt a bit to find the similar interface in ParaView’s animation view and most people have to confront the awkward “Enter camera positions manually” interface instead.

In both ParaView and Slicer we should make it so that more/all of the controls (not just the camera) that vary over the animation are set up in this user friendly way and then manually editable after the fact.

[jcfr (Jean Christophe Fillion Robin (Kitware))]

@Dave_DeMarle

we should make it so that more/all of the controls (not just the camera) that vary over the animation are set up in this user friendly way and then manually editable after the fact.

Agreed, this is critical for usability.

[jcfr (Jean Christophe Fillion Robin (Kitware))]

To add support for interpolation and associated UI for key frame edition, here is what I was thinking (based on what we discussed during the last project week).

The Key Frame editor

from a high level, I suggest we keep things simple and support the following:

(1) a master timeline to drive the animation. (at first we would only have “Real-time” mode where we set the the duration in seconds).

We could reuse the Drishti widget (MIT License) or the one from Paraview, or an improved version getting the best of both.

• Drishti timeline (I was able to compile it … and it was not straightforward)

• Paraview timeline

(2) three pre-defined tracks:

• camera path

• ROINode (the one allowing to crop volume rendering)

• transfer function

where keyframe can explicitly be added/removed only for ROINode and transfer function

The “camera path” would be defined using an improved version of the endoscopy module (as describe here).

Suggested approach

This section describe a possible way to more forward by updating the sequences modules to be user in Key Frame editor.

See Documentation/Nightly/Extensions/Sequences - Slicer Wiki

Features already available in sequences extension

Given the current sequences implementation where:

• A given vtkMRMLSequenceNode is associated with a collection of data nodes.

• A vtkMRMLSequenceBrowserNode is associated with a collection of vtkMRMLSequenceNode

• The entrypoint for updating all proxy nodes is the function vtkSlicerSequenceBrowserLogic::UpdateProxyNodesFromSequences

• The “time steps” available in a given sequence browser corresponds to the number of “items” associated with a browser node, itself corresponding to the number data nodes in the associated master node.

• if a synchronized sequence node (different from the master sequence) doesn’t have a data node corresponding to the requested “step” (call “indexValue” in the code), the closest data node is used to update the proxy node. This is implemented in

  • vtkMRMLSequenceNode::GetItemNumberFromIndexValue(const std::string& indexValue, bool exactMatchRequired /* =true */)

  • vtkMRMLSequenceNode::GetDataNodeAtValue(const std::string& indexValue, bool exactMatchRequired /* =true */)

We need a way to map:

• the concept of track and key frame with vtkMRMLSequenceNode, vtkMRMLSequenceBrowserNode and data node

• have a container to store time for a key frame, and interpolation function used between two frames

Good news, key frames are data node within a vtkMRMLSequenceNode

Support for interpolation

To support interpolation , I was then thinking of the following:

• update the parameter exactMatchRequired from “GetDataNodeAtValue” function to be something like “interpolationMode” accepting the following values:

  • ExactMatch

  • ClosestMatch

  • ExactMatchOrInterpolated Interpolated

• In case of ExactMatchOrInterpolated mode, if not an exact match GetDataNodeAtValue would have to perform the interpolation (or retrieved a cached value of the interpolated data node)

• the interpolation function would have the following paramerters: fromDataNode, toDataNode, nodeInterpolator, fromTime, toTime and currentTime (or fromStep, toStep and currentStep)

• within each sequence node we would store a vector of “node interpolators”. This would corresponds to the function applies when performing an interpolation between two data nodes.

• update the vtkMRMLNodeSequencer::NodeSequencer base class adding interpolation method : GetInterplatedNode(vtkMRMLNode* from, vtkMRMLNode* to, vtkMRMLNodeSequencer::NodeInterpolator* interpolator, double fromTime, double toTime, double currentTime)

Reusing Paraview code

I was initially thinking to extract some of the code of Paraview and add it to VTK so that we can reuse some in both Slicer and Paraview:

  vtkPVKeyFrame
  vtkPVCompositeKeyFrame
  vtkPVBooleanKeyFrame
  vtkPVSinusoidKeyFrame
  vtkPVExponentialKeyFrame
  vtkPVRampKeyFrame
  vtkPVKeyFrameCueManipulator (and base class vtkPVCueManipulator)
  vtkRealtimeAnimationPlayer
  vtkSequenceAnimationPlayer
  vtkTimestepsAnimationPlayer
  vtkCompositeAnimationPlayer
  vtkSMAnimationScene (investigate if changes can be integrated with vtkAnimationScene)
  Note that vtkAnimationCue and vtkAnimationScene are already part of VTK.

But it seems the API available in the sequences module doesn’t really overlap with what was done in Paraview.