Nrrd vs nii regarding affine data and transforms

Hi everyone,

I’ve noticed that after preforming brain registration task, if I save the resulting output file (after transform hardening) as a nrrd file and afterwards load the saved file to slicer, the loaded file is transformed and looks as it did before saving as I would expect. But if I try to save the registered output file as a .nii format, (which I need for further processing) after loading the nii file back to slicer I can see that the affine data doesn’t match with the one that I was saving. Are there any differences with affine data saving for nrrd and nii formats?

Thanks

Can you provide a specific example with the steps to reproduce the issue? As far as Slicer is concerned nrrd and nii should save and reload the same (we prefer nrrd because nii has a history of different software interpreting the standard differently).

Sure:
I am performing CT to MRI registration task. I am using BRAINS registration module. I set the MRI volume to be fixed, CT - moving. After registration is done, I check if it looks ok. Then I harden the transforms and save the transformed CT volume. If I save it as nrrd and load it again to the scene with the used MRI volume - everything is fine. If I save it as nii, when I load it to slicer with the used MRI volume, they don’t fit.

Interesting - yes, I can reproduce that, thanks for pointing it out. Hardening the transform is applying the affine (possibly shear) matrix to the slice directions and it seems that’s preserved in the nrrd header but the directions are orthogonalized in the nii version.

The workaround for this case would be to resample the images before saving (e.g. with BRAINS Resample choosing a reference with orthogonal directions).

@lassoan and others: what do you think of enforcing orthogonality of the IJKToRAS directions?

It would mean, for example, resampling when applying a linear transform with shear E.g. here:

It’s a good idea, I’ve added a ticket to track this idea: https://issues.slicer.org/view.php?id=4499.

Note that ApplyNonLinearTransform would need some improvements, as currently it resamples the image on the grid of the original volume, so the output may be heavily cropped (https://issues.slicer.org/view.php?id=4433).

If I understand the proposal correctly, I vote against. This is an issue with a specific (and arguably obsolete) file format. Resampling is a lossy operation.

Both NIFTI-1 and NIFTI-2 support affine transforms. Would there be a problem if we export using that option?

It is not just about how to save the file. There are many other places in the Slicer and its dependencies where image axes are assumed to be orthogonal.

For example, image data axes must be orthogonal in VTK and ITK. In VTK, you cannot even store image direction in vtkImageData (just origin and spacing). In ITK, “columns of the Direction matrix are expected to form an orthogonal right handed coordinate system. But this is not checked nor enforced in itk::ImageBase” (source).

I guess I’m still not convinced. Sure, there are some poorly behaved algorithms which would benefit from resampling to orthogonal. But if I understand correctly, this proposal would exclude the option for lossless computation by well-behaved algorithms.

The slicer volume node design allows non-orthogonal directions but they never really happened much. Hardening a shear transform is the first user-level case I can think of where this has come up, and difference in output that @NaglisR reported is a valid concern.

@gcsharp perhaps well behaved algorithms should accept transforms (linear or nonlinear) instead of expecting it to be in the direction cosines? The in-between case where algorithms don’t accept nonlinear transforms but do accept non-orthogonal directions is pretty rare (even handling orientation at all is pretty rare).

It’s more about convincing ITK and VTK algorithm developers. Even just oriented image data is a hard sell sometimes, as it makes many algorithms more complicated and sometimes slightly slower. That’s why VTK still does not support oriented image data - but we’ll try to change this.

However, making algorithm work correctly on images with non-orthogonal axes is a different story. It is often much more difficult (and probably not always feasible) to change algorithms to work directly on images with non-orthogonal axes. As the cost is high and the benefit is relatively low (as we have a quite good workaround of treating shearing transforms as non-linear transforms), direct processing of sheared images will probably not be generally supported in ITK, VTK, or in applications that are built on these toolkits.

@gcsharp “This is an issue with a specific (and arguably obsolete) file format.” Which format is obsolete in this case? .nii is used way more widely than nrrd in my opinion. Thanks everyone for your feedback.

I see .nii used extensively in neuroimaging community, but not much elsewhere. Nifti has important neuroimaging-specific features, but suffers from several problems, for example:

  • there seems to be a confusion around axis orientations (different variants may use different conventions)
  • storage of multidimensional (4D, 5D, …) is limited
  • header is not user-readable, it is difficult to read/write without a parsing library (non-user-readable binary header has some advantages, too, but most often flexibility and ease of use is more important than storage efficiency or completely lossless encoding of numerical values)
  • no built-in support for compression (you can only zip the entire file, losing its original file extension, lossy compression cannot be supported this way)
  • ITK reader ignores patient orientation tag

NRRD format has none of these limitations. I don’t use nifty, so maybe not all of these limitations are true anymore or there may be workarounds, but overall nifti is the file format of choice for neuroimaging but does not seem to be an ideal file format for general research use.

2 Likes

Sorry to resurrect this thread. My lab just ran into this issue, leading to some significant re-work of slightly mis-registered Nifti files. I would strongly suggest at least warning the user when this occurs. It is completely reasonable for a user to assume that if they harden a transform and then save, the saved file will accurately reflect the whole transform, including any shear. In fact, this is the case when saving as NRRD.

I believe the NIFTI header is fully capable of storing general affine transforms, including shear. This is necessary with, for example, CTs acquired with a tilted gantry. Why does the Nifti writer orthogonalize the transform matrix when saving?

Where would you want the warning - maybe when you select an output format (say nii, but could be others) we should have a pre-save step where it inspects the data and reports if the save operation will be lossy.

I believe this is because ITK images need to have orthogonal direction vectors and we cast the vtkImageData to itk::Image as part of the save process.

I think a warning when saving a non-orthogonal volume to Nifti would be appropriate, either when selecting “nii” or “nii.gz” from the list, or when clicking “save”.

Do you know of a way to convert from NRRD (or another format that Slicer handles better than Nifti) to Nifti while preserving the full affine spatial orientation information, including any shear?

Agreed, a warning would be good. It would require a bit of effort, so if someone wants to volunteer to work on it and test it out then probably several of us could suggest ways to implement that. There are lots of other examples of potentially lossy file formats, like saving models to STL.

To answer your specific question, perhaps you could use dcm2niix, which has recently added nrrd conversion options. @Chris_Rorden may have additional suggestions.

Also, standard warning, nifi format has several ways of encoding orientation (qform and sform modes) and it’s never clear to me which methods any particular tool will or won’t support, so you may run into other software that trips up on shear transforms. At least for Slicer we defer to ITK’s implementation in the hopes that it will do things right as often as possible, but clearly that approach didn’t work for this particular case. I’d think other common tools (maybe ITK-snap?) would also have trouble with sheared orientation info. It would be interesting to hear if people know what the general support is for this feature of nifti.

This silent orthogonalization of image axes in the image writer should be probably fixed in ITK. I’ve asked about this on the ITK forum: https://discourse.itk.org/t/saving-non-orthogonal-volume-in-nifti-format/2760.

@hbraunDSP the NIfTI header contains two spatial transforms- the quaternion-based QForm and the Matrix-based SForm. It is worth noting that Quaternions can not store shears, so it is worth thinking of them as a 9-degree of freedom transform (translation, rotation, zoom in each of 3 dimensions). On the other hand, the Matrix can store shears and is 12 DoF (translation, rotation, zoom and shear). By design, the QForm was designed for the scanner native space, with the SForm initially identical to the QForm but capable of having a 12-DoF coregistration applied.

This has proved problematic in two ways. First, CT scans are often acquired with shear (gantry tilt). Second, ITK does not allow volumes to have shears, and therefore ITK based tools give precedence to the QForm and if the QForm is not set will orthogonalized the SForm. This is different from other tools, that give precedence to the SForm. Historically, this has caused a lot of issues, e.g. SPM gives precedence to SForm and ANTs gives precedence to QForm, so one tool has vastly different starting estimate to the other.

This explains one bizarre aspect of my dcm2niix tool. In most cases, it goes to great lengths to not interpolate the data, but it will interpolate to remove shear.

@lassoan correctly details many of the weaknesses of NIfTI. I have no dog in this fight, but I do think there are tradeoffs, and the popularity of this format reflects Darwinian selection. In defense of NIfTI I would argue:

  1. It is very simple and explicit. This makes it very easy for scientists to access the data rapidly and with minimal code.
  2. In general, the spatial transforms are very clear, and it does allow shears. One could argue that ITK simply does not support the NIfTI format as defined. This is a limitation in ITK, not the NIfTI format. NRRD does not resolve this issue either. The core issue is that ITK requires volumes to not have shear.
  3. The while file compression is not an issue, as the header fits in the first 352 bytes. Therefore, you can rapidly extract the first 352 bytes to read the header. From my perspective, this is superior to NRRD compression that uses text at the start that prevents people from using high performance compression and decompression algorithms.
  4. The header is simple and explicit, so there are great tools for reading them (fslhd, 3dinfo, etc), and it is simple to write your own in your favorite language.
  5. Not sure exactly what is meant by ITK ignoring patient orientation, but again this is a weakness of ITK not the format.
  6. I agree that storage of multidimensional data is limited, but this also makes it very easy to make strong assumptions regarding the data across all volumes stored in the 7 dimensions. This makes writing analysis and processing tools much simpler as the input is necessarily constrained, e.g. one knows from the outset that all volumes have consistent resolution.

Its nice to have different formats that each fit a niche. While NIfTI is far less flexible than DICOM, it also avoids the complication. Seen from the perspective of DICOM, NRRD and NIfTI are very similar. They have slightly different niches, but are typically used in the same role. So while some advocate medical imaging should rely on a single format my sense is one should choose the best tool for the job.

I am not sure whether @gcsharp is referring to NRRD or NIfTI as obsolete. In my field, NIfTI is far more widely used (SPM, FSL, AFNI, BIDS). NRRD is convenient since you can create a header with a text editor, but beyond the basic tags there seems to be no consensus on how to store a lot of meta information. They are far more similar than different.

1 Like

There is a large gap between simplest NRRD and hyper-complex DICOM, which domain-specific file formats, such as Nifti can fill. The main problem with Nifti is that it is so poorly specified (or the community misuses/misinterprets so often for some other reasons) that it is really hard to write software that can use it “correctly”.

Not for everybody. After reading the specification, I was still not sure what the role of qform and sform transforms are. Discussion in the related ITK forum topic shows that regular users of Nifti and core ITK developers are uncertain, too.

If you know what the correct behavior would be for ITK, it would be great if you could describe it in this ITK forum topic.

@lassoan often the SForm and QForm are the same. A good approach would be to use whichever has the larger NIFTI1_XFORM_CODE, and use the SForm if they are tied. In general, I would only expect these to differ if the acquired data could not reported in the QForm (e.g. CT scan with shear) or if the user coregistered the image so that the raw transform is still in the QForm and the subsequent coregistration uses the SForm. I am pretty sure recent releases of SPM and FSL attempt to set both the QForm and SForm to the same value (to the extent possible) to reduce issues with ITK based tools like ANTs. In this way, modern versions of SPM sacrifice the benefit of two representations (patient space and standard space) in order to ensure ITK compatibility.

In the interest of full disclosure, before the NIfTI format was finalized I explicitly criticized the inclusion of these two spatial mappings. The desire for backward compatibility with the Analyze format made it hard to store two matrices (as they would not fit in the header). The QForm was seen as sufficient for raw MRI and the SForm allowed 12-DoF coregistration without reslicing the data. It does mean that every tool that wants to support this format needs to support these two ways of representing space, e.g. nifti_quatern_to_mat44 / nifti_mat44_to_quatern. So not only do I feel your pain, but part of me feels that this was an issue that was vocally identified prior to the format being finalized. I feel very similar to the BIDS format, where I also raised concerns. In any democratic discussion of a format one will always feel there was a compromise. However, this niggling issue is small compared to the overall benefits for these unified formats.

I did contact the ITK team about a decade ago about adapting code to behave similar to other tools. I think there was a concern regarding unintended consequences of tools that assumed ITK volumes always have orthogonal dimensions. As I understand it, ITK can not abide by non-rectangular volumes, so while the SForm in NIfTI is particularly clumsy for ITK.

To be clear, this is an limitation with ITK not NIfTI. One could just as easily create a NRRD header where space directions defines a shear and ITK will treat the volume as rectangular. Likewise, there are many DICOM CT scans with shear.

Further, while NIfTI is complicated by having both quaternion and matrix representations, the NRRD format is complicated because it allows the user to specify the frame of reference. This makes it really easy to write a NRRD header, as you can use the same frame of reference as the source data (e.g. NIfTI LAS or DICOM RPS). However, a fully compliant NRRD reader must handle all the possible transforms. Form my perspective, NIfTI headers are trickier to write than NRRD, but they are easier to read.

As an aside, matrices tend to be very familiar to developers, as they are useful for so many problems (e.g. GLM statistics). They seem like a natural way to represent space in a manner that is intuitive. Therefore, I think I was not alone when I had an initial distaste for the alien quaternions. However, they do have some remarkable properties. They allow us to interpolate between two orientations gracefully and provide a nice way to avoid gimbal lock. In addition, the inability to store shear can actually be an asset, as it avoids rounding errors from accidentally inducing shears. Therefore, I certainly appreciate quaternions, but I maintain that including both matrices and quaternions into an otherwise simple format has had repercussions.

I do not think NIfTI is poorly specified. I think the ITK developers made an explicit decision to use the representation they could fully encode. This was a different choice than other tools. Nowadays most tools attempt to store the same value in the Q and S Form (to the fullest extent possible). Beyond this one issue, the format is remarkably simple helping developers focus on the unique processing of their tools with simple and efficient I/O routines for interchange.

2 Likes