Perk Tutor - Return more than one value in custom metric function

I’ve written a simple metric for Perk Tutor (see code below) to analyze the deviation between the tool and an entry point. Instead of returning the final RMS deviation, I also want to return the R, A, S components of the error. How would I do this? I tried returning multipel values in GetMetric(), but this isn’t allowed.

import math
import vtk
from PythonMetricsCalculator import PerkEvaluatorMetric

class EntryDeviation( PerkEvaluatorMetric ):


  # Static methods
  @staticmethod
  def GetMetricName():
    return "Entry Deviation"
  
  @staticmethod  
  def GetMetricUnit():
    return "mm"
  
  @staticmethod
  def GetTransformRoles():
    return [ "Needle" ]
  
  @staticmethod
  def GetAnatomyRoles():
    return { "Targets": "vtkMRMLMarkupsFiducialNode" }
    
    
  # Instance methods    
  def __init__( self ):
    PerkEvaluatorMetric.__init__( self )
    self.R_deviation = []
    self.A_deviation = []
    self.S_deviation = []
    self.needleTipTargetDistance = []

    
  def SetAnatomy( self, role, node ):   
    if ( role == "Targets" ):
      self.targets = node  
      return True
      
    return False

    
  def AddTimestamp( self, time, matrix, point, role ):  
    currTargetPosition = [ 0, 0, 0 ]
    self.targets.GetNthFiducialPosition(0, currTargetPosition ) #Assume first fid point is the entry point
    currTargetPosition_RAS = [ currTargetPosition[ 0 ], currTargetPosition[ 1 ], currTargetPosition[ 2 ] ]
    
    # Find the needle tip in RAS
    needleTip_RAS = point[0:3]
    
    self.R_deviation.append(currTargetPosition[0] - needleTip_RAS[0])
    self.A_deviation.append(currTargetPosition[1] - needleTip_RAS[1])
    self.S_deviation.append(currTargetPosition[2] - needleTip_RAS[2])

    self.needleTipTargetDistance.append(math.sqrt( vtk.vtkMath.Distance2BetweenPoints( currTargetPosition_RAS, needleTip_RAS ) ))
    
    
  def GetMetric( self ):
    return self.needleTipTargetDistance[-1]

Typically what we do is return a tab delimited string with all the values. See, for example, here:

This makes it easy to copy or load the table of metrics into Excel/Sheets/SPSS/etc. for statistical analysis.

If you let us know what you would like to do with the metric values after you have computed them, I may be able to suggest a better solution.

1 Like

Thanks, that will work for me.

I was also wondering how can I log and save the metric values at each sequence frame, instead of getting summary values only at the end of the sequence?

By default, metric values at each frame are not logged. But you could add a few lines at the end of your AddTimestamp function to write the current metric value to file.

For example, adding something like this line at the end of the AddTimestamp function could work:

with open("my_filename.txt", "a") as myfile:
    myfile.write(str(self.GetMetric()))

But usually I would suggest to do all your computations from within the metric, so you don’t have to do log the values at each frame. What is your particular use case for this? If you let us know, I may be able to suggest a better solution.

I want to get the trajectory of the tracked tool with respect to the target fiducial / annotation ruler. This is why I want time stamped R, A, S coordinates - having this information can let me calculate most other metrics without having to write new metrics in perk tutor.

The trace and visualize trajectory metrics help with visualization, but they don’t provide an easy way to access the trajectory points.

@mholden8 Another issue I’m having is that I can’t seem to import modules other than math and vtk in my metric file.
For example I tried ‘import time’ and ‘from datetime import datetime’ so that I could add date-time to my filenames following on from your previous suggestion. But if I do this the metric does not work.

You should be able to import modules into your metrics just like you would into any other Python script. I tried importing the modules you mentioned into a metric, and it worked for me. When you say “the metric does not work”, what is the error message you get in the log or Python console? If you also send along the latest version of your metric, I can have a look to see what the issue might be.

May I ask why you do not want to write other metrics in Perk Tutor? Is there a way we could make writing metrics in Perk Tutor easier?

That’s strange. Here is my metric file that I’m having import issues with: Entry Deviation Perk Tutor Metric · GitHub

If I comment out the import datetime line (and change line 35), the metric runs fine. If I leave the import in, I can add the metric to Slicer but it does not load the ‘Anatomy’ node selector box properly. Furthermore if I run the analysis, the metric does not return any values. No errors get thrown.

Here is another metric I’m having issue with, but can’t understand why: Trajectory Deviation Perk Tutor metric · GitHub
For this I get the error that the AddTimestamp() method expects 5 inputs but only 4 were given. I couldn’t figure it out. The metric doesn’t run at all.

The reason is because I want to analyse the metrics over time - for example I want velocity per sequence frame, displacement per frame, accuracy per frame. In my application, it is more useful to have a detaile analysis of the tool trajectory over time rather than a summary metric at the end - and from my understanding it’s not possible to do that (unless you output all your data as one string)?

If the solution to that is to write to a .txt or .csv file from the Perk Tutor metric, then I’d rather do it only once from perk tutor and derive the other metrics from the log file.

Thanks for taking the time to help!

Forgot to mention I’m on Windows 10, Slicer 4.10.2

I had a look, and it seems like it was a Python 2 vs Python 3 issue. I’ve pushed a fix to Perk Tutor so that import statements will work in Slicer 4.10.2 with Python 2. It should be available in Slicer 4.10.2 the next time extensions are built for it. Until then, it is already working in the latest nightly.

The issue with your second metric is on line 52:

File "<string>", line 52, in AddTimestamp
TypeError: unsupported operand type(s) for -: 'list' and 'list'

I will work on a fix to make the error messages more useful…

1 Like

Thanks! Somehow missed that error message.

Confirmed that Perk Tutor is working with imports now in Slicer 4.10.2

Thanks for your help!

1 Like

@mholden8 Is there a way to make the ‘Start/Stop’ button within the Transform Recorder module into a shortcut or something that could be accessibe by a footswitch for example?

You can define keyboard shortcuts for any function as described here.

1 Like

@mholden8 Is there a way to access the Sequence Browser node that the metrics are being calculated on, from the PerkEvaluatorClass? I would like to get the name of that node

Yes, given the Perk Evaluator node, you can get the Sequence Browser node using the function:

perkEvaluatorNode.GetTrackedSequenceBrowserNode()

It is defined here: https://github.com/PerkTutor/PerkTutor/blob/master/PerkEvaluator/MRML/vtkMRMLPerkEvaluatorNode.h#L122.

1 Like

Amazing, thank you!!

1 Like