Bug in ImportLabelmapToSegmentationNode if multiple parent transforms

Support
1

A labelmap imported to a segmentation using ImportLabelmapToSegmentationNode is transformed incorrectly if there are multiple parent transforms soft-applied. I have been using ImportLabelmapToSegmentationNode successfully for a long time with a transformed labelmap with no problem, but I just ran a case where there were multiple parent transforms (i.e. the parent transform had a parent transform applied) and the converted segmentation node put the segments in the wrong place. With a little testing, it appears that each transform is at least partially applied, but the total transformation is incorrect.

I reproduced this effect with SampleData and saved the scene, it is available here: https://drive.google.com/file/d/1fXu8EHHY4jzL3COEn8sjXXz1FmgdGdiq/view?usp=sharing

In that scene, I created a segmentation with 3 segments and exported that to a labelmap (these align perfectly). Then I created two transforms which each apply a rotation around the AP axis and a SI translation. Then I soft-applied the first transform to the labelmap and the original segmentation, and then soft-applied the second transform to the first transform. Lastly, I ran ImportLabelmapToSegmentationNode on the transformed labelmap, creating the segmentation node labeled SegmentationFromLabelMap in the scene. With both the SegmentationFromLabelMap and LabelMapFromSegmentation shown, you can see that they do not align the way they should. It looks like an equal amount of rotation has been applied, but not an equal total translation. So, it’s not the case that only the first parent transform has been applied, because then the rotation amounts would not be equal. If I combine the two transforms into one (clone the first, apply the second, and harden) and use that, then the output segmentation is aligned with the transformed labelmap (correct behavior). Is it possible that the parent transforms are being applied in the wrong order?

Using Slicer 5.10.0 on Windows 11. The conversion was carried out using the following convenience wrapper around ImportLabelmapToSegmentationNode

def labelMapToSeg(labelmapNode, createClosedSurfaces=True):
    """Converts an input labelmap node to a segmentation node"""
    segNode = slicer.mrmlScene.AddNewNodeByClass("vtkMRMLSegmentationNode")
    slicer.modules.segmentations.logic().ImportLabelmapToSegmentationNode(
        labelmapNode, segNode
    )
    if createClosedSurfaces:
        segNode.CreateClosedSurfaceRepresentation()
    return segNode

To recreate:

# Load scene linked above, then run
labelNode = getNode('LabelMapFromSegmentation')
segNode = labelMapToSeg(labelNode)
segNode.SetName('NewSegmentationFromLabelMap')

image
image570×491 101 KB

2

Thanks for the detailed report. Probably the underlying code is only taking the partent transform into account and not the full transform to world. This should probably be a one-line change to use the right call. Did you look at the code? I’ll bet this is something fixable, either by looking at the code ones self or with the help of an LLM, probably easier with the slicer-skill.

3

This was my initial guess also, but does not appear to be the case. If that were so, the amount of rotation as well as translation would have been incorrect, since each of the two parent transforms apply a rotation.

I didn’t. I have more limited ability to follow the C++ code, and have never successfully set up a Slicer build environment (which would be needed to test a corrected solution, I think). I was hoping that a clear, reproducible example with the bug report would be enough for someone with these things already set up to take a look and hopefully see the problem and fix. I too suspect that the fix will be straightforward once identified.

I am looking forward to getting this working and trying it out for development and for problems such as these in the future. I can’t do it on my work computer because of sensible corporate security concerns, but I will be trying to get Claude with the slicer-skill set up on a home computer as soon as I can manage to get the time.

4

The relevant section of the code appears to be:

So, something is going wrong with either getting the transform or applying it. Both of those functions are clearly intended to handle general transformations, so they should be set up to properly handle a hierarchy of linear transforms.

5

This is an excellent example of where we are today in terms of coding tools.

Just to give you a sense of what you can expect, I prompted my local Claude to use the slicer-skill like so:

Can you read this forum post and suggest an answer to Mike’s question? https://discourse.slicer.org/t/bug-in-importlabelmaptosegmentationnode-if-multiple-parent-transforms/46524

It too quite some time looking around, examining source code, git logs, etc.

The reasoning is quite interesting to skim. It’s too long to post here but you can look at it here: reasoning for this question: https://discourse.slicer.org/t/bug-in-importlabelmaptosegmentationnode-if-multiple-parent-transforms/46524/3 · GitHub

In the end it came up with this suggested answer which I didn’t check but sounds awfully good to me.

:robot: This post contains AI-generated content.

Now I have a clear picture of the bug. Here’s the answer I’d suggest posting to Mike’s forum question:

Root Cause Found

The bug is in vtkOrientedImageDataResample::IsTransformLinear in vtkOrientedImageDataResample.cxx around line 987.

What happens:

When ImportLabelmapToSegmentationNode handles parent transforms, it calls GetTransformBetweenRepresentationAndSegmentation → vtkMRMLTransformNode::GetTransformBetweenNodes, which correctly traverses the full transform hierarchy and builds a vtkGeneralTransform using PostMultiply mode. With PostMultiply, transforms are appended to the end of the pipeline and applied in order (T1 first, T2 second).

So for your setup (labelmapNode → T1 → T2 → world), GetTransformBetweenNodes correctly builds T2 * T1 (T1 applied first, then T2).

Then TransformOrientedImage calls IsTransformLinear, which extracts the chained transforms into a vtkTransform using its default PreMultiply mode. In PreMultiply mode, each new Concatenate(M) does T = T_old * M, which reverses the effective application order. So it builds T1 * T2 instead of T2 * T1 — the transforms are applied backwards.

Why single transforms work: With only one transform in the chain, there’s no ordering issue. The first (and only) transform is correct regardless of pre/post multiply mode.

Why you see “equal rotations but wrong translations”: If T1 and T2 both rotate around the same axis, their rotations commute (T1_rot * T2_rot = T2_rot * T1_rot), so the rotation result looks the same either way. But translations do NOT commute under rotation, so the translation accumulation is incorrect with the reversed order.

The Fix

In vtkOrientedImageDataResample.cxx, in IsTransformLinear around line 1006, add one line to switch to PostMultiply before the concatenation loop:

vtkGeneralTransform* generalTransform = vtkGeneralTransform::SafeDownCast(inputTransform);
if (generalTransform)
{
  outputLinearTransform->PostMultiply();  // <-- add this line
  for (int transformIndex = 0; transformIndex < generalTransform->GetNumberOfConcatenatedTransforms(); ++transformIndex)
  {
    ...
    outputLinearTransform->Concatenate(currentLinearTransform);
  }
  return true;
}

Immediate Workaround

Until this is fixed, you can harden the transforms before importing:

labelmapNode.HardenTransform()
segNode = labelMapToSeg(labelmapNode)

This would be a good candidate for a PR against Slicer/Libs/vtkSegmentationCore/vtkOrientedImageDataResample.cxx. Note the fix would also affect vtkSegmentation::ApplyLinearTransform which uses the same function.

Should I post this response or refine it further? I should mention this analysis is based on reading the code in your local Slicer checkout — it would be good to add a small Python test script to verify before submitting the PR.

6

Much appreciated @pieper ! I had wondered if transform order might have been the issue, but it might have taken me quite some time to realize that the the issue was that the default concatenation multiply order was pre-multiply and the missing piece was to specify post-multiplication. With it pointed out, I agree that this makes good sense, is a plausible way for the bug to have entered the code, and that this is the likely fix.

If this diagnosis correct, does this mean that segmentations under two parent transforms will also appear in the wrong place? That’s actually a more serious bug. I’m used to needing to think about whether an export or conversion process in Slicer is going to respect or ignore parent transforms, but I do not worry about arbitrary transform hierarchies soft-applied to basic MRML nodes like segmentation nodes. I assume those always “just work”

For moving forward, I can mark the AI solution as the solution for this thread, but that won’t get the fix into Slicer. Should I make that change in the cxx file and submit a PR? Open an issue on github? As mentioned earlier, I don’t have the build system set up, so I can’t actually compile and test the solution, it would just be based on the above “seeming reasonable”. What would you (or others) recommend?

7

I agree we should file an issue on github with a link to this thread. I also think we should have @lassoan weigh in since he knows this code well. Then we can work on a PR to fix this and any related issues. We can convert your example into a test that will confirm the behavior. Thanks again for your careful analysis and report :+1:

Regarding a local build, if you are on an apple silicon mac, even a macbook air, the builds are remarkably fast, like under an hour. (They are still a bit buggy but would work for this kind of debugging). I’m not sure what the compile times are these days on other platforms but be sure you have a fast disk. Historically builds on windows were overnight processes.

1 Like