TotalSegmentator ERROR failed to compute result - returned non-zero exit status 120

Support
1

When I run totalsegmentator on 3D slicer using the CTA Abdomen (panoramix) sample data it comes with the following error. I have tried uninstalling 3D slicer/PyTorch/totalsegmentator numerous times with no luck.

Operating system: macOS Monterey 12.5.1 (Apple M1 chip)
Slicer version: 5.2.2
Expected behavior: totalsegmentator
Actual behavior: ERROR when running apply

"Processing started
Writing input file to /private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-05-23_10+25+00.631/total-segmentator-input.nii
Creating segmentations with TotalSegmentator AI…
Total Segmentator arguments: [‘-i’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-05-23_10+25+00.631/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-05-23_10+25+00.631/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]
/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py:116: UserWarning: torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.
warnings.warn(“torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.”)
/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/cuda/amp/autocast_mode.py:118: UserWarning: torch.cuda.amp.autocast only affects CUDA ops, but CUDA is not available. Disabling.
warnings.warn(“torch.cuda.amp.autocast only affects CUDA ops, but CUDA is not available. Disabling.”)
Traceback (most recent call last):
File “/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator”, line 93, in
main()
File “/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator”, line 86, in main
totalsegmentator(args.input, args.output, args.ml, args.nr_thr_resamp, args.nr_thr_saving,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/totalsegmentator/python_api.py”, line 173, in totalsegmentator
seg = nnUNet_predict_image(input, output, task_id, model=model, folds=folds,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/totalsegmentator/nnunet.py”, line 255, in nnUNet_predict_image
nnUNet_predict(tmp_dir, tmp_dir, task_id, model, folds, trainer, tta)
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/totalsegmentator/nnunet.py”, line 106, in nnUNet_predict
predict_from_folder(model_folder_name, dir_in, dir_out, folds, save_npz, num_threads_preprocessing,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/inference/predict.py”, line 668, in predict_from_folder
return predict_cases_fastest(model, list_of_lists[part_id::num_parts], output_files[part_id::num_parts], folds,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/inference/predict.py”, line 493, in predict_cases_fastest
res = trainer.predict_preprocessed_data_return_seg_and_softmax(d, do_mirroring=do_tta,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/training/network_training/nnUNetTrainerV2.py”, line 211, in predict_preprocessed_data_return_seg_and_softmax
ret = super().predict_preprocessed_data_return_seg_and_softmax(data,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/training/network_training/nnUNetTrainer.py”, line 516, in predict_preprocessed_data_return_seg_and_softmax
ret = self.network.predict_3D(data, do_mirroring=do_mirroring, mirror_axes=mirror_axes,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/network_architecture/neural_network.py”, line 147, in predict_3D
res = self._internal_predict_3D_3Dconv_tiled(x, step_size, do_mirroring, mirror_axes, patch_size,
File “/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/nnunet/network_architecture/neural_network.py”, line 348, in _internal_predict_3D_3Dconv_tiled
gaussian_importance_map[gaussian_importance_map == 0] = gaussian_importance_map[
RuntimeError: “min_all” not implemented for ‘Half’
Exception ignored in: <totalsegmentator.libs.DummyFile object at 0x1b765f5b0>
AttributeError: ‘DummyFile’ object has no attribute ‘flush’

If you use this tool please cite: https://doi.org/10.48550/arXiv.2208.05868

No GPU detected. Running on CPU. This can be very slow. The ‘–fast’ option can help to some extend.
Using ‘fast’ option: resampling to lower resolution (3mm)
Resampling…
Resampled in 4.13s
Predicting…"

2

Could you please go to Pytorch Utils module and copy here the installed pytorch version?

3

Pytorch version 1.8.1

4

Could you also write us the NVIDIA driver version (line below PyTorch version in PyTorch Utils module).

The problem is that TotalSegmentator requires pytorch>=2 (see more information here). You can fix this by installing pytorch manually as described here - just use PythonSlicer executable instead of pip3.

5

NVIDIA driver: not found

6

Thanks for the information. I was wondering if pytorch-1.8.1 is enforced by some old GPU driver, but apparently not.

Anyway, until we figure out why an old pytorch version is installed on some computers, you can install pytorch-2.x in your Slicer to make TotalSegmentator work.

7

I still don’t seem to be able to upgrade PyTorch on 3D slicer. Could you help troubleshoot?

8

I’ve submitted a pull request to make it easier to select pytorch version in PyTorch Utils module. Hopefully it will be merged today or tomorrow. You can track its status here:

You need to use the lateset Slicer Stable Release or latest Slicer Preview Release to get the update (the update is available automatically in the Extensions Manager the day after the pull request is merged).

9

Despite install PyTorch via Anaconda it still fails. When I then tried to run totalsegmentator it advised that I needed to install PyTorch which is then did however it installed version 1.8.1.

See screenshots.

Screenshot 2023-06-05 at 18.33.30.png874×340 83.3 KB
Screenshot 2023-06-05 at 18.33.45.png874×556 97.3 KB
Screenshot 2023-06-05 at 18.34.03.jpg1420×928 198 KB

Here’s a copy of the console:

Collecting torch
Using cached https://download.pytorch.org/whl/cpu/torch-2.0.1-cp39-none-macosx_10_9_x86_64.whl (143.4 MB)
Collecting torchvision
Using cached https://download.pytorch.org/whl/torchvision-0.9.1-cp39-cp39-macosx_10_9_x86_64.whl (13.1 MB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting filelock
Using cached filelock-3.12.0-py3-none-any.whl (10 kB)
Requirement already satisfied: typing-extensions in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torch) (4.6.0)
Collecting sympy
Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
Requirement already satisfied: networkx in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torch) (3.1)
Collecting torch
Using cached https://download.pytorch.org/whl/cpu/torch-1.8.1-cp39-none-macosx_10_9_x86_64.whl (119.6 MB)
Requirement already satisfied: pillow>=4.1.1 in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torchvision) (9.4.0)
Requirement already satisfied: numpy in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torchvision) (1.23.4)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.1.3-cp39-cp39-macosx_10_9_x86_64.whl (13 kB)
Collecting mpmath>=0.19
Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Installing collected packages: torch, torchvision
WARNING: The scripts convert-caffe2-to-onnx and convert-onnx-to-caffe2 are installed in ‘/Applications/Slicer.app/Contents/lib/Python/bin’ which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed torch-1.8.1 torchvision-0.9.1

[notice] A new release of pip is available: 23.0 → 23.1.2
[notice] To update, run: python-real -m pip install --upgrade pip
[Python] Failed to compute results.
[Python] Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-06-05_18+35+07.399/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-06-05_18+35+07.399/segmentation’, ‘–ml’, ‘–task’, ‘total’]’ returned non-zero exit status 120.
Traceback (most recent call last):
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 264, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 762, in process
self.logProcessOutput(proc)
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 655, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-06-05_18+35+07.399/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/n4/_8hf03v543x_q3b0k017wbfm0000gp/T/Slicer-mosc7cj2/__SlicerTemp__2023-06-05_18+35+07.399/segmentation’, ‘–ml’, ‘–task’, ‘total’]’ returned non-zero exit status 120.

10

It appears that for some reason torchvision 0.9.1 is getting installed which is compatible with torch 1.8. See chart at GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer Vision.

I’m not sure why it is not getting the latest torchvision, but that does appear to explain why torch 1.8.x would get installed.

11

It’s interesting, maybe it is related to torchvision. Anyway, if the pull request gets merged (hopefully tomorrow) then it’ll be really easy to upgrade/downgrade pytorch.

12

Have you tried this page with the various script installer options?

image
image1132×743 54.6 KB

13

Yes, indeed, @cmjones if you don’t want to wait a day then you can run the command that you get from the official pytorch website that @rbumm showed above. You just need to use PythonSlicer instead of pip3.

14

I’ve installed PyTorch using Conda but when I try to import or install in Slicer it seems to only install version 1.8.1.

import PyTorchUtils

torch = PyTorchUtils.PyTorchUtilsLogic().torch

Collecting torch

Using cached https://download.pytorch.org/whl/cpu/torch-2.0.1-cp39-none-macosx_10_9_x86_64.whl (143.4 MB)

Collecting torchvision

Using cached https://download.pytorch.org/whl/torchvision-0.9.1-cp39-cp39-macosx_10_9_x86_64.whl (13.1 MB)

Collecting sympy

Using cached sympy-1.12-py3-none-any.whl (5.7 MB)

Collecting jinja2

Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)

Requirement already satisfied: networkx in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torch) (3.1)

Requirement already satisfied: typing-extensions in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torch) (4.6.3)

Collecting filelock

Using cached filelock-3.12.0-py3-none-any.whl (10 kB)

Requirement already satisfied: pillow>=4.1.1 in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torchvision) (9.4.0)

Requirement already satisfied: numpy in /Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages (from torchvision) (1.24.2)

Collecting torch

Using cached https://download.pytorch.org/whl/cpu/torch-1.8.1-cp39-none-macosx_10_9_x86_64.whl (119.6 MB)

Collecting MarkupSafe>=2.0

Using cached MarkupSafe-2.1.3-cp39-cp39-macosx_10_9_x86_64.whl (13 kB)

Collecting mpmath>=0.19

Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)

Installing collected packages: torch, torchvision

WARNING: The scripts convert-caffe2-to-onnx and convert-onnx-to-caffe2 are installed in ‘/Applications/Slicer.app/Contents/lib/Python/bin’ which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Successfully installed torch-1.8.1 torchvision-0.9.1

[notice] A new release of pip is available: 23.0.1 → 23.1.2

[notice] To update, run: python-real -m pip install --upgrade pip

15

slicer.util.pip_install("-U torch torchvision") in python console worked for me

16

Thank you. I have finally managed to get it to work by running:

pip.main([‘install’, ‘–upgrade’, ‘torch’])
pip.main([‘install’, ‘–upgrade’, ‘torchvision’])

and then install PyTorch via the Extensions Manager

Thank you for your help

17

I have a Total segmentator running errors.
My GPU is RTX 3070 (VRAM 8Gb). Cuda version 12.1. Slicer ver 5.2.2
I have installed TotalSegmentator extension (ver. 9340dd7) and PyTorch (ver. fe2e92e) through Extension Manager.
When I press Apply button of Total Segmentator with Fast option off, the Slicer 5.2.2 become frozen.
With Fast option on, this error message appeared.
What can I do to fix this issue?

Traceback (most recent call last):
File “C:\Users\Anatomy-Kim\AppData\Local\NA-MIC\Slicer 5.2.2\bin\Python\slicer\util.py”, line 2967, in tryWithErrorDisplay
yield
File “C:/Users/Anatomy-Kim/AppData/Local/NA-MIC/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 264, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “C:/Users/Anatomy-Kim/AppData/Local/NA-MIC/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 762, in process
self.logProcessOutput(proc)
File “C:/Users/Anatomy-Kim/AppData/Local/NA-MIC/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 655, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[‘C:/Users/Anatomy-Kim/AppData/Local/NA-MIC/Slicer 5.2.2/bin/…/bin\PythonSlicer.EXE’, ‘C:\Users\Anatomy-Kim\AppData\Local\NA-MIC\Slicer 5.2.2\lib\Python\Scripts\TotalSegmentator’, ‘-i’, ‘C:/Users/Anatomy-Kim/AppData/Local/Temp/Slicer/__SlicerTemp__2023-06-13_14+50+31.409/total-segmentator-input.nii’, ‘-o’, ‘C:/Users/Anatomy-Kim/AppData/Local/Temp/Slicer/__SlicerTemp__2023-06-13_14+50+31.409/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.

18

@pssi_92 You have returned non-zero exit status 120 in the log, which means that the installed PyTorch version is not compatible with TotalSegmentator. You can uninstall the current version of Pytorch and install a compatible version using PyTorch Utils module:

  • Go to PyTorch Utils module
  • Click Uninstall PyTorch
  • Restart Slicer
  • Go to PyTorch Utils module
  • Type >=2 in PyTorch version requirement (this forces installation of PyTorch version >=2.0.0, which is compatible with TotalSegmentator)
  • Click Install PyTorch
20

Thank you for advice.
In my computer today, the total segmentator worked fine. I don’t know what happened exactly.
I installed total segmentator same in other computer of my colleague, but it won’t work well.
The error message was same “returned non-zero exit status 120”.

So I followed your advice but the result was same.
Before I reinstall PyTorch, I checked the version but it was 2.0.1+cu118.
I think it is higher version than 2.0.0.

The GPU was GTX 1070, Cuda version 12.1, Slicer 5.2.2. PyTorch verison 2.0.1+cu118.

21

Maybe the installation on your colleague’s computer is corrupted (most likely due to insufficient disk space or network error during installation).

I would recommend to reinstall Slicer from scratch into an empty folder. Also remove the downloaded TotalSegmentator model from your user profile folder. When installing TotalSegmentator and its dependencies make sure to have at least 20GB of free disk space. If there is any error then