@mholden8 implemented it, but haven’t published the algorithm. You can find the implementation in the source code, here: SlicerIGT/vtkSlicerPivotCalibrationLogic.cxx at master · SlicerIGT/SlicerIGT · GitHub
It is essentially looking for the axis with least variation in the tool motion during calibration. And adds a few heuristics, e.g. for direction in the axis, it assumes that the position sensor is behind the pointer, not in front.
To explain it to someone with no math background: Calculate the rotation of the tool between two time points. That rotation has an axis direction. Then, record a series of rotations of this tool, and take the average direction of the rotation axes. That will be the final axis of rotation. If you have already computed the tip point, then make sure the pointy tip is pointing away from the tracked position, so you may need to flip the direction in the end.