A colleague gave me a volume which she aligned in a particular orientation she want the data to be collected. This a linear transform and it is hardened.
When I open this volume in Segment Editor, there is the little warning about “Slice views are not aligned with segmentation and warns me that striping artifact may appear”.
As it happens I want to paint a single slice in this orientation and export this slice. And when I do this and yes, stripping effects do appear in slice above and below. I can align the slice views with respect to the segmentation, and the stripping effects disappear, but then I am not in the orientation i want to be in to extract the data.
I find this behavior confusing and it is a common complaint among our users (samples are often scanned in random orientations and one of the common things they do is to reorient the sample in approximate anatomical planes before doing segmentation).
Why can’t we rotate the segmentation to the slice view as opposed to doing the other way around? It defeats the purpose of aligning the volume in the first place.
If you want to segment a single slice in an orientation that is not aligned with the acquisition you need to resample the volume, not just harden it. We could probably make this easier to do, but normally you want to segment a 3D region so this isn’t really an issue most of the time.
Even in 3D it is an issue, because it confuses users. Where it rotates the slices back to segmentation orientation, would it be possible to give user the resampling the volume option too?
Since some users will want to segment in a reformat/reslice slice view, I think the improvement here will be needing to improve the understanding between reslicing and resampling.
Reslicing to align to the anatomical plane is not resampling. Therefore that should be a first step of the user prior to segmenting for the voxels to be aligned in this new direction. Then they can utilize effects like “Fill between slices” as painting won’t be across slices which would prevent this in the non resampled case. When resampling the user is going to have to be mindful of whether they save this new volume as an additional volume, overwrite the original, or don’t save it and only get the statistics out of the segmentation.
It could make sense to add an option to the Segmentation Geometry dialog to resample so that the image grid aligns with the current orientation directions of the volume. Or there could be a module that makes it simple to resample acquisitions to anatomical space (e.g. click on the nose, center of head, and one ear) if people have resolution to spare and prefer better painting over preserving resolution.
This is also one of the issues: Harden does two different things depending on the transform. It actually resamples the volume when you choose a non-linear transform, but it only modifies the directional matrix, if it is a linear transform. THe point is, it is not clear to the user the which action it is going to happen by choosing “harden transform”, if they do not know this difference.
We can try to work on this, with some guidance from you or @lassoan. However, I still think as opposed to doing a patch to the segment editor, I think the language and functionality around the transforms and hardening can be improved to make the communication better.
Another typical complaint is that hardened volumes do not show up as in the orientation the user thought they saved in programs like ImageJ, which does not read the directional matrix in the header.
I think the root cause of the issue is that segmentation on oblique slices of a binary labelmap looks simple, and users don’t realize just how complex this problem is.
One possible solution is to educate users about complex things. That’s why I’ve added this page, which explains why this is hard and two possible solutions. Do your users know about this page? Does it solve their problem of segmentation on oblique slices?
Another potential improvement could be to make it easier to align the ROI markup with slices. Currently, when an ROI is placed on a 2D slice, its axes are aligned with the world coordinate system axes by default (and so the user needs to enable rotation interaction and then align the ROI manually). Instead of this, we could align the ROI axes with the slice axes by default (similarly to how it is done for plane markups). This way, the steps for resampling the volume would be:
draw a new ROI on a slice view
click specify geometry button in segment editor
select the ROI
click OK
This takes about 6-8 mouse clicks, but since this resampling permanently damages the image, it should not be performed frequently anyway. Would this help?
We cannot fix 2D imaging software, such as ImageJ. They have other problems to worry about, such as dealing with extremely large images, so probably they just don’t want the complexity of arbitrarily positioned and oriented 3D images.
Users need to accept that ImageJ, Adobe PhotoShop (that some people use for DICOM segmentation), etc. are just not suitable for 3D imaging and they should not trust these software when they work with 3D images.
Harden does one thing: permanently applies a transform to the node. The process to achieve this is chosen automatically and non-technical users don’t need to know the difference: the application handles this transparently for them. If they open the resulting image in ImageJ then they can see unexpected results - as for any other 3D data sets. If the user rotated the image so that oblique slices can be edited then that was a mistake: the image position or orientation should never be corrupted just to allow editing in oblique slices, but instead the slice views should be rotated.
I don’t really like the “harden” term, but at least it is unique enough that there are no misunderstandings it you can easily google search for its meaning in Slicer. In Blender it is called just “apply” which is just too generic. Similar words for persistently applying some modifications are “burn in” for annotations, “bake” for textures, but none of these seem significantly better than “harden”.
Any suggestions for improving wording in documentation or user interface are very welcome.
This page does a good job of explaining the complexity of why the way the Slicer works the way it works, but does not necessarily help them very much. Here is what I get following the suggestion of resampling via CropVolume:
This is confusing because, the only way I can see my newly resampled volume in the rotated orientation is when I choose the “reformat” axis. If I click on the sagittal, it goes back to the untransformed orientation. What I am going to do when I wanted to see the new volume what I think is the axial plane? Click axial, and expand the dialog box and click rotate volume to plane? This is too much in my opinion. Not only this is still confusing, it doesn’t resolve the issue of exported volume will still not be read in the anticipated direction in ImageJ or others, even though the user thinks they resampled the volume (It just looks like noisy volume).
I understand why you guys are doing this, Slicer is a predominantly for medical imaging, and you don’t want to mess with the acquired data randomly. But that’s not common for our datasets (again it is very common for scans to be in totally random orientation). So what we need is a tool (as part of SlicerMorph, not in core Slicer) to resample the volume with the warning to the user that this will potentially degrade their volume, particularly if they keep doing it (transform ->resample, then further transform and resample). But it is a need.
It’s true that bad metadata about patient orientation is rare in most medical imaging (MR, CT…), but not all (US). I think having some easy ways to define and resample the data along standard reference planes makes sense. It could start in SlicerMorph and migrate to the core if generally useful.
Crop volume module does something very similar to what is needed. Of course a dedicated Python module wrapped over it could simplify the GUI and workflow.
I think the only missing feature is assigning image axes to arbitrary RAS axes. Current modules strive to preserve the original image position and orientation (because once they are lost, they cannot be recovered), but if the scan contains a bucket of specimens in random orientation then it is safe to discard the original orientation. There is an “Orient Scalar Volume” module that reorders the voxels, which is useful, but it still preserves the original axis directions and the GUI is very user-unfriendly. The small Python module could take care of these, too (discard the original orientation and show immediate preview of the new image orientation).
In recent Slicer versions, if you click the eye icon in data module for a volume then it will take care of all this: it automatically resets the slice views to the default (axial, sagittal, coronal) directions, and then rotates them to the volume axes. Of course, this is only relevant if the goal is to preserve the original image orientation.
The module switches to four-up view and enables interactive slice intersections
Rotate slice intersections using drag-and-drop to show axial slice in red and coronal slice in green
Click “Align” button
The module rotates volume to make the slice planes aligned with anatomical axes. The result is that the volume is oriented correctly in physical space, except there may be flips.
Click Flip LR, Flip AP, or Flip IS button if needed to fix any inverted axis
The module “flips” the volume along the chosen direction (actually it rotates by 180 degrees)
Define clipping ROI (optional, if not specified then the entire volume is kept)
Click “Finish” button
The module uses Crop volume module to crop and resample the volume
Even without the automations, the workflow takes about 1.5 minutes:
With the module it should take less than 1 minutes. It could be automated further by asking the user to specify a threshold and generate a segmentation from that, get principal axes, and use that to initialize the slice orientations. The segmentation can be also used for automatic cropping.
These code snippets were used in the video (that would be in the custom scripted module):
def rotateVolumeToSlicePlanes(volumeNode):
sliceViewNames = ["Yellow", "Green", "Red"] # sagittal, coronal, axial
# Rotate slice planes to standard anatomical planes
roiToWorld = vtk.vtkMatrix4x4()
for axisIndex, sliceViewName in enumerate(sliceViewNames):
# Set roiToWorld column to slice normal
sliceToRas = slicer.app.layoutManager().sliceWidget(sliceViewName).mrmlSliceNode().GetSliceToRAS()
for component in range(3):
roiToWorld.SetElement(component, axisIndex, sliceToRas.GetElement(component, 2))
# Invert matrix
worldToRoi = vtk.vtkMatrix4x4()
vtk.vtkMatrix4x4.Invert(roiToWorld, worldToRoi)
# Apply transform to volume
volumeNode.ApplyTransformMatrix(worldToRoi)
slicer.modules.volumes.logic().CenterVolume(volumeNode)
# Reset view axes to default
for sliceViewName in sliceViewNames:
slicer.app.layoutManager().sliceWidget(sliceViewName).mrmlSliceNode().SetOrientationToDefault()
# Reset field of view in slice views
slicer.util.setSliceViewerLayers(fit=True)
def flipVolumeOrientation(volumeNode, axis):
transform = vtk.vtkTransform()
if axis == 0:
transform.RotateX(180)
if axis == 1:
transform.RotateY(180)
if axis == 2:
transform.RotateZ(180)
volumeNode.ApplyTransform(transform)
volumeNode = getNode('809_1-resample_sample')
rotateVolumeToSlicePlanes(volumeNode)
flipVolumeOrientation(volumeNode, 0)
#flipVolumeOrientation(volumeNode, 1)
#flipVolumeOrientation(volumeNode, 2)
