Composite transform from ANTsR is read incorrectly into Slicer

Support
1

I have registered a mouse image to a template in ANTsR (R version of ANTs). We have saved the resultant affine and deformable transforms and when these are loaded into Slicer and correctly nested, the resultant image is correct (i.e. consistent with what I obtain in ANTsR).

image
image1320Ă—611 86.7 KB

From the affine and warp field, I can also a create a single composite transform in ANTsR. I know this transform is correctly written to disk, because I if I directly apply (read from disk) it to the image I get the same image in screenshot. Thus the issue is not on ANTsR end.

However, if I load the composite transform and apply it to the image, this is what I get in Slicer:

image
image1423Ă—631 81 KB

Any ideas why this might be the case?

2

If it is helpful, there is this warning generated by Slicer specific to the composite transform:

Warning: In /work/Stable/Slicer-0-build/vtkAddon/vtkOrientedGridTransform.cxx, line 335
vtkOrientedGridTransform (0xbad5f80): InverseTransformPoint: no convergence (0, 34.7143, 224.571) error = 0.491319 after 500 iterations. Further convergence warnings suppressed until transform is modified.

3

@simonoxen do you have any experience with getting composite transforms from ANTs/ANTsX loaded into Slicer? Or have you encountered an issue like this?

4

If you load the composite transform and click “Split” in the transforms module then do you see the same transform components as the ones you loaded separately? (maybe it is easier to compare them if you save to file)
What ITK version is used in your ANTsR?
Does thing work well if you use SlicerANTs?

If you don’t see major issues when you visualize the transform (e.g., using Grid mode) then probably you can ignore this message.

Can you upload somewhere the separate transforms, the composite transform, the fixed and moving image and the image resampled with ANTs?

5

I don’t see this button.

> itkVersion()
[1] "5.3"
> itkTag()
[1] "v5.3rc04"

It is completely wrong. it somehow made into four images as shown in the screenshot.

here it is as an mrb file

6

I forgot to include the ANTsR resampled image. here is the link for that. https://app.box.com/shared/static/yk7vvhlaagcmkwlbl37s8y0hzjiqthss.nrrd

7

Thanks for the data sets. I don’t see any issue on my computer (Windows10, Slicer-5.2.1). The Composite transform and the Warp+Affine transform result in the exact same transformation (which is the same as the Ants-resampled image):

image
image1920Ă—1015 127 KB

Check it on a few different computers to see if you see anything common between those that don’t work correctly.

8

Interesting that’s not what I am seeing on my end on windows 5.2.1

image
image1817Ă—805 115 KB

9

Try unzipping the scene files into a folder and load the .mrml file. Many things can go wrong when you open a 6GB zip file. Or it might be some rendering issue? Have you tried this on different computers/operating systems?

10

This happens to me both on Ubuntu 20.04 and windows 11 with 5.2.1

Can you make sure you zoom in, because if it simply centered on the reference image, you may not notice the deformations in the image. Better yet simply have only the moving image displayed, zoom in and slide across the stack. This is what I see:

image
image2763Ă—727 207 KB

11

I see what you mean now. If you zoom out then then you can some artifacts far, far away from the transform’s domain. The artifacts are due to that it is a grid transform with very large displacement vectors at the edge of the domain. Such artifacts are completely normal and you can safely ignore them.

Why are there artifacts?

The displacement is extrapolated outside the transform’s domain in a way that:

  1. The displacement vectors change changing at the edges of the transform domain (otherwise the transform would not be invertible, which is essential if you want to transform both images and meshes), and
  2. At infinite distance from the transform’s domain the displacement converges to zero.

If a transform has very large displacement at the edge of its domain then the extrapolation may have some large waves before the displacements converge to zero. When you zoom out then you see those large displacement waves far away from the transform’s domain.

How to avoid such displacement extrapolation artifacts?

If you want to avoid such extrapolation artifacts outside the image domain then the best is to keep the large linear rigid bulk transform and the remaining warping grid transform separate. Then the displacement vectors in the grid transform will be small and if there is any extrapolation, the displacements converge to zero very quickly, without any waves.

ITK can save these two transforms into a single transform file (that is a composite transform), but maybe this option is not made available in ANTs. If ANTs cannot save composite transforms then you need to educate users about ignoring extrapolation artifacts or teach them how to load and apply multiple transforms.

12

Yes, that part is fine. When I apply the separate affine + warp transform as nested, I do not see those artifacts at all.

It is available, and that’s what the composite_transform is. That’s the one showing the artifacts when loaded into the Slicer.

13

A composite transform file contains multiple transforms. The transform that is named as “moving-Forward-compositetx” is a simple grid transform that contains the total displacement for each point of the fixed image. You can see this in the “Information” section in Transforms module:

image
image599Ă—555 63.7 KB

You can create a composite transform by hardening the warping transform on the bulk affine transform. The information section will look like this and the “Split” button will appear that allows the user to split the transform to its components:

image
image634Ă—688 110 KB

14

This is the command in ANTsR (and ANTs proper as well) to concatenate transforms into a single transforms. that’s what I used to generate the composite transform.

antsApplyTransforms(ref, moving, transformlist = c("moving-Warp.nii.gz", "moving-Affine.mat"), compose='./moving-Forward-compositetx')

Your method of hardening separate transforms in Slicer should be equivalent to this at the end. But the fact that I am seeing deformation artifacts in composite from ANTsR but not from the composite transform by hardening in Slicer tells me that something is going wrong when the original composite transform is applied to the image. I am not going to chase this any further, but I wanted to bring this up in case someone encounters this.

15

Composite transform refers to a very specific thing in ITK: a transform that contains multiple transforms inside. Let’s use it only for this meaning to avoid misunderstandings. The antsApplyTransforms command does not create a composite transform.

The antsApplyTransform simply crates a grid transform by modifying a transform by another one. It is an irreversible, lossy operation, which fundamentally changes the properties of the transform (it may not be feasible to robustly invert or extrapolate it anymore), and may hugely increase the storage size (if the input contained warping transform, such as bspline or tps). There are some meaningful uses of this transform (it allows resampling an image in the transform’s domain without requiring someone to implement ITK transforms; or get a displacement field for statistical analysis), but in general, the antsApplyTransform command should be avoided for grid transforms.

It seems that in your workflow ANTs is not used properly. “General registration (ANTs)” in Slicer (provided by SlicerANTs extension) provides a composite transform as a result. You can have a look at SlicerANTs source code to see how to achieve this.

2 Likes