The points should be in mm space
I looked into the Fiducial Registration Wizard code a bit more, and I think the key function being used is provided by vtkThinPlateSplineTransform
(VTK: vtkThinPlateSplineTransform Class Reference). I do pretty much all of my work within the Slicer environment, so I am not very familiar with this, but I believe you can use VTK from python outside of Slicer without any problems (Getting Started - VTK documentation). It looks like you would create a vtkThinPlateSplineTransform, set the source and target landmarks, set the basis to R (because you’re in 3D), and then Update()
to calculate the transform. I’m sure there are other VTK functions which would allow you to apply that transform to a base image to generate a warped image. This might be the best approach for you if you are looking to set up a pipeline which can run automatically outside of Slicer. However, if it turns out you want to use Slicer functionality, you might also like to be aware that it is possible to access Slicer code without running the full GUI in a few different ways, and looking at this discussion might help orient and get you started: Using Slicer and Slicer Modules from Command-Line