Slicer 5.0.3 - UKF Tractography Resource Issue

Support
1

Hi all,

I am trying to run UKF Tractography on my DTI data, and am encountering problems that seem to be related to memory use.
Note: this always worked with the same data and parameters on a previous release of Slicer (4.10.2), so it would appear something has changed in the new release?

OS: Windows 10
Slicer version: 5.0.3
System details: Precision workstation, i9-9940X 14 core CPU, 128GB RAM, M.2 SSD, RTX2080
Input data: DTI data with 1 B0 and 64 directions, 220x220x156 voxels, 1mm³ resolution

Behavior:
UKF Tractography runs, using significant resources (over 35 GB of RAM) for a few hours (see image below).
Upon completion, the created fiber bundle file is empty. However, there are no reported errors (UKF Tractography completes “without errors”).

1mm_resources
Blue= memory use (GB), Pink = CPU use (%)

When I resample my data to 1.2mm isotropic voxels, the tractography runs fine (see figure below for resource use).
Similarly, it runs fine if I significantly crop my data (by excluding some background, brainstem and cerebellum).
In both these cases, memory used by UKF is reduced to <32GB.

1.2mm_resources
Blue= memory use (GB), Pink = CPU use (%)

I noticed this has been an issue in the past for some users (see UKF tractography fails for large scans (~1.5mm iso)), and that the suggested fix was to add more physical/virtual memory to ensure the process is not running out of resources. However, we definitely have 128GB available, so this does not seem to be the issue in our case.

I have noted, that compared to the previous version we used (Slicer 4.10.2), the memory usage of UKF tractography has more than doubled for processing of the same data.
Could this be related to the switch from VTK 8.2 to VTK 9 in Slicer 5 / Some internal format change? e.g. from integer to floating point?

Apart from what has caused that increased memory usage, from what I can tell, it appears as though when memory use of the UKF Tractography process exceeds 32GB, then no output file is created / the process crashes. Can this be some internal limitation? Some maximum size of something specified somewhere? Explicitly or given by some variable’s format?
Interestingly, the UKF tractography process completes “without errors”, so maybe the UFK-extension is actually fine and the problem occurs just in the very final stage when the resulting tract file is assembled and written out?

Is this something that is known/has been reported before? I am happy to provide a sample dataset to replicate this issue if this would help.

Thanks in advanced for your support!

2

Thanks for reporting this - does this happen on all datasets or just on a few cases? E.g. do you see the same issue on the DWI sample data that comes with Slicer in the SampleData module?

3

Thank you for your reply!

I do not observe this on the sample data, or on any other data where the resolution is not as high / where there are less directions. So I guess it is related to the size of the data?

4

Interesting. Okay, then yes, please provide a sample dataset and I hope someone can have a look.

5

I would really appreciate that, thank you!

Here is a sample set with the DTI data and the mask for tractography.

6

Thanks for sharing the data - someone from the DMRI team should be able to look into this. Can you help by providing the exact command line (you can find the exact parameters in the log file if you ran from within Slicer).

7

Of course! Here is the full command line:

[DEBUG][Qt] 12.10.2022 17:08:09 [] (unknown:0) - UKF Tractography command line: 

C:/ProgramData/NA-MIC/Slicer 5.0.3/NA-MIC/Extensions-30893/UKFTractography/lib/Slicer-5.0/cli-modules/UKFTractography.exe --dwiFile C:/Users/Administrator/AppData/Local/Temp/Slicer/JAIE_vtkMRMLDiffusionWeightedVolumeNodeB.nhdr --seedsFile C:/Users/Administrator/AppData/Local/Temp/Slicer/JAIE_vtkMRMLLabelMapVolumeNodeB.nhdr --labels 1 --maskFile C:/Users/Administrator/AppData/Local/Temp/Slicer/JAIE_vtkMRMLLabelMapVolumeNodeB.nhdr --tracts C:/Users/Administrator/AppData/Local/Temp/Slicer/JAIE_vtkMRMLFiberBundleNodeB.vtp --seedsPerVoxel 1 --seedingThreshold 0.18 --stoppingFA 0.12 --stoppingThreshold 0.1 --numThreads -1 --numTensor 2 --stepLength 0.3 --Qm 0 --recordLength 0.9 --maxHalfFiberLength 200 --recordFA --recordTensors --Ql 0 --Qw 0 --Qkappa 0.01 --Qvic 0.004 --Rs 0 --sigmaSignal 0 --maxBranchingAngle 0 --minBranchingAngle 0 --minGA 10000

Thank you!

1 Like
8

Hi @pieper - Do you happen to know if there has been any progress made from the DMRI team on this front? Any updates would be welcome! Cheers!

9

I know it’s being investigated but I don’t think it’s been resolved. Hopefully soon…

1 Like
10

Hello @cnot, we have been testing the UKFTractography module with the data you have provided. We have noticed that you had an additional seed file that was used when you ran it yourself. Looking at the two files that are within the sample data, there is a Sample_DWI_Volume_4D_1mm.nrrd, and a Sample_Mask_1mm.nrrd, and there is not any JAIE_vtkMRMLLabelMapVolumeNodeB.nhdr.

We are currently testing with our own seedFile which was created using the segmentation editor, but we also tested without any seedFile and have some mixed results.

Another quick note: When running UKFTractography with the initial mask file you provided we would get the following error:

implementation only accepts masks of type ‘signed char’, ‘unsigned char’, ‘short’, and ‘unsigned short’

Convert your mask using ‘unu convert’ and rerun.

So, we ended up using unu convert to convert the mask you provided to a short data type to perform the following tests.

First, we ran the UKFTractography on your data on our HPC compute nodes outside of slicer just to verify that the data would get results. We did this to make sure the issue was not in the data at all, and we got good results doing that.

Next, we tested data with the newest slicer that was not your sample data just to see if we could get results with any data and we were able to get results using that different data inside the newest slicer.

Then we Used your data without and seedFile using the Slicer GUI and during this test we were not able to generate a usable tracts file (We are still investigating what is causing this issue within the GUI)

Additionally, we were able to use your data without a seedFile using the newest Slicer through the command line and that generated a good tracts file. The command I used for this is as follows:

C:\Users\ryanz>“D:\ryanz\AppData\Local\NA-MIC\Slicer 5.0.3\Slicer.exe” -launch “D:\ryanz\AppData\Local\NA-MIC\Slicer 5.0.3\NA-MIC\Extensions-30893\UKFTractography\lib\Slicer-5.0\cli-modules\UKFTractography.exe”

–dwiFile D:\SlicerTest\Sample_DWI_Volume_4D_1mm.nrrd

–maskFile D:\SlicerTest\Sample_Mask_1mmShort.nrrd

–tracts D:\SlicerTest\newTractTest1.vtk

–seedsPerVoxel 1

–seedingThreshold 0.18

–stoppingFA 0.12

–stoppingThreshold 0.1

–numThreads -1

–numTensor 2

–stepLength 0.3

–Qm 0

–recordLength 0.9

–maxHalfFiberLength 200

–recordFA

–recordTensors

–Ql 0 --Qw 0 --Qkappa 0.01 --Qvic 0.004 --Rs 0

–sigmaSignal 0

–maxBranchingAngle 0

–minBranchingAngle 0

–minGA 10000

Link to the tracts file is here: Dropbox - newTractTest1.vtk - Simplify your life

image
image772×600 163 KB

It is a 5GB tract file so incase you did not want to DL to view this is a stapshot of the tracts.

Currently, we are in the process of running the UKFTractography with a seedFile using the GUI to see if we are able to produce the bug with a seedFile as well.

It takes so long to run without the seed that trying to debug it running on the data without a seedFile will be very time consuming so if we can reproduce it with faster runs that will allow us to generate debug data at a much faster rate. This is the current goal.

Sorry about the delayed response, and thanks you for your patience while we sort out the issue.

11

If you run a CLI module using Slicer GUI then you can find the command-line parameters in the application logs. You might find that the difference is due to computing the tracts with slightly different parameters.

1 Like
12

Thanks, we will check the logs and see what we can find then. We appreciate the suggestions.

13

Hi @RyanZurrin, thank you so much for your detailed update! I really appreciate it.

Sorry I was not clear with this - I was using essentially the same as the mask as my seeds file, so I was basically telling it to seed from each voxel in the brain.

Ah, yes, I realize I also did this from within Slicer as I encountered the same error. I must have not saved the converted mask to send you, really sorry about this.

This is interesting to know, thank you! I will try to reproduce this.

And this all sounds excellent, thanks again so much for all your help and for figuring this out. I look forward to hearing the next update!

14

Hi @RyanZurrin, I hope you had a good start to the New Year! Do you happen to have any updates on this front? Cheers.

15

Hi, how did you Convert your mask using ‘unu convert’? I am getting the same error message regarding the mask datatype. Thank you