It seems that you have an old driver and CUDA version installed, which may cause incompatibility issues.
Please install the latest CUDA version that pytorch supports - currently CUDA 11.6 or 11.7 - and then uninstall and install pytorch.
Already update the drivers, but still got the error message:
the message:
Traceback (most recent call last):
File “C:\Users\ACER\AppData\Local\NA-MIC\Slicer 5.2.1\bin\Python\slicer\util.py”, line 2961, in tryWithErrorDisplay
yield
File “C:/Users/ACER/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 258, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “C:/Users/ACER/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 715, in process
self.logProcessOutput(proc)
File “C:/Users/ACER/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 624, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[‘C:/Users/ACER/AppData/Local/NA-MIC/Slicer 5.2.1/bin/…/bin\PythonSlicer.EXE’, ‘C:\Users\ACER\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Scripts\TotalSegmentator’, ‘-i’, ‘C:/Users/ACER/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_14+12+33.314/total-segmentator-input.nii’, ‘-o’, ‘C:/Users/ACER/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_14+12+33.314/segmentation’, ‘–ml’, ‘–task’, ‘total’]’ returned non-zero exit status 120.
How much memory your GPU has? Do you have issues with all data sets or just some of them?
Probably the best is if you remove Slicer and reinstall it. All the experimentation with installing various pytorch versions may have messed things up.
1 Like
On my borderline laptop, I get a “less than 7 GB” warning for a “total” TotalSegmentator run with the CTChest sample dataset.
If I select “No” (no FAST mode) →
Processing completed in 226.51 seconds
System Settings:
NVIDIA System Information report created on: 01/03/2023 09:17:56
System name: LAPTOP-xxxxxxxxxx
[Display]
Operating System: Windows 10 Home, 64-bit
DirectX version: 12.0
GPU processor: NVIDIA GeForce GTX 1060
Driver version: 526.98
Driver Type: DCH
Direct3D feature level: 12_1
CUDA Cores: 1280
Core clock: 1404 MHz
Memory data rate: 8.01 Gbps
Memory interface: 192-bit
Memory bandwidth: 192.19 GB/s
Total available graphics memory: 14278 MB
Dedicated video memory: 6144 MB GDDR5
System video memory: 0 MB
Shared system memory: 8134 MB
Video BIOS version: 86.06.3A.00.0F
IRQ: Not used
Bus: PCI Express x16 Gen3
Device Id: 10DE 1C20 114E1025
Part Number: 2914 0030
Could you post your system information? See NVIDIA Control Panel → System Information
I am having the same issue with “returned non-zero exit status 120”
I have a clean install of Slicer 5.2.1. Am running with 128Gigs RAM and a 3090Ti with 24Gigs.
PyTorch is 1.13.1+cu117
I have noticed that the file that is saved in the …/fold_0 folder during processing is only called model_final_checkpoint.model - it does not have the .pkl extension as well - not sure if that is the issue?
TotalSegmentator window:
Processing started
Writing input file to C:/Users/282745H/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_17+18+42.605/total-segmentator-input.nii
Creating segmentations with TotalSegmentator AI…
Total Segmentator arguments: [‘-i’, ‘C:/Users/282745H/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_17+18+42.605/total-segmentator-input.nii’, ‘-o’, ‘C:/Users/282745H/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_17+18+42.605/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]
Traceback (most recent call last):
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Scripts\TotalSegmentator”, line 201, in
main()
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Scripts\TotalSegmentator”, line 179, in main
seg = nnUNet_predict_image(args.input, args.output, task_id, model=model, folds=folds,
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\totalsegmentator\nnunet.py”, line 232, in nnUNet_predict_image
nnUNet_predict(tmp_dir, tmp_dir, task_id, model, folds, trainer, tta)
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\totalsegmentator\nnunet.py”, line 106, in nnUNet_predict
predict_from_folder(model_folder_name, dir_in, dir_out, folds, save_npz, num_threads_preprocessing,
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\nnunet\inference\predict.py”, line 668, in predict_from_folder
return predict_cases_fastest(model, list_of_lists[part_id::num_parts], output_files[part_id::num_parts], folds,
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\nnunet\inference\predict.py”, line 468, in predict_cases_fastest
trainer, params = load_model_and_checkpoint_files(model, folds, mixed_precision=mixed_precision, checkpoint_name=checkpoint_name)
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\nnunet\training\model_restore.py”, line 140, in load_model_and_checkpoint_files
trainer = restore_model(join(folds[0], “%s.model.pkl” % checkpoint_name), fp16=mixed_precision)
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\nnunet\training\model_restore.py”, line 56, in restore_model
info = load_pickle(pkl_file)
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Lib\site-packages\batchgenerators\utilities\file_and_folder_operations.py”, line 49, in load_pickle
with open(file, mode) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\282745H\.totalsegmentator\nnunet\results\nnUNet\3d_fullres\Task256_TotalSegmentator_3mm_1139subj\nnUNetTrainerV2_ep8000_nomirror__nnUNetPlansv2.1\fold_0\model_final_checkpoint.model.pkl’
Exception ignored in: <totalsegmentator.libs.DummyFile object at 0x000002BC5A99A730>
AttributeError: ‘DummyFile’ object has no attribute ‘flush’
Using ‘fast’ option: resampling to lower resolution (3mm)
Resampling…
Resampled in 0.88s
Predicting…
Small popup error window:
Traceback (most recent call last):
File “C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\bin\Python\slicer\util.py”, line 2961, in tryWithErrorDisplay
yield
File “C:/Users/282745H/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 258, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “C:/Users/282745H/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 715, in process
self.logProcessOutput(proc)
File “C:/Users/282745H/AppData/Local/NA-MIC/Slicer 5.2.1/NA-MIC/Extensions-31317/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 624, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[‘C:/Users/282745H/AppData/Local/NA-MIC/Slicer 5.2.1/bin/…/bin\PythonSlicer.EXE’, ‘C:\Users\282745H\AppData\Local\NA-MIC\Slicer 5.2.1\lib\Python\Scripts\TotalSegmentator’, ‘-i’, ‘C:/Users/282745H/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_17+18+42.605/total-segmentator-input.nii’, ‘-o’, ‘C:/Users/282745H/AppData/Local/Temp/Slicer/__SlicerTemp__2023-01-03_17+18+42.605/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.
Already re-install my slicer, instal totalsegmentator and the pytorch again, but the same result
This is my system information:
NVIDIA System Information report created on: 01/03/2023 16:36:34
System name: PREDATOR
[Display]
Operating System: Windows 10 Home Single Language, 64-bit
DirectX version: 12.0
GPU processor: NVIDIA GeForce RTX 3070 Laptop GPU
Driver version: 527.56
Driver Type: DCH
Direct3D feature level: 12_1
CUDA Cores: 5120
Resizable BAR Yes
Max-Q Technologies 3rd Gen
Dynamic Boost 2.0 Yes
WhisperMode 2.0 Yes
Advanced Optimus No
Maximum Graphics Power 100 W
Core clock: 1290 MHz
Memory data rate: 12.00 Gbps
Memory interface: 256-bit
Memory bandwidth: 384.06 GB/s
Total available graphics memory: 16299 MB
Dedicated video memory: 8192 MB GDDR6
System video memory: 0 MB
Shared system memory: 8107 MB
Video BIOS version: 94.04.2B.40.2B
IRQ: Not used
Bus: PCI Express x16 Gen3
Device Id: 10DE 249D 14421025
Part Number: 4735 0010
Sorry for the errors, hope we can fix them. The two systems indicated above should definitely be capable of doing complete TotalSegmentator GPU runs.
Please understand that the 3D Slicer extension is a wrapper of the TotalSegmentator AI tool.
Have you tried something simple as running Slicer in Admin mode?
You could also install TotalSegmentator in a local Python installation and run it from the command line, which is what we did before implementing the tool wrapper in 3D Slicer.
Could you uninstall PyTorch completely with the PyTorch extension.
Use the restart option and repeat until you get a uninstall success message.
Then paste
slicer.util.pip_install("torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117")
it into the Python Interactor and press return and wait until it is installed.
This looks like a real error. Maybe the model download have been interrupted. Please remove the entire .totalsegmentator folder and retry.
Laptop with Optimus technology (switchable integrated graphics/discrete GPU) are always problematic. As a quick test, disable Optimus (choose to always use the GPU for all software) and see if it fixes the issue. If you still get some error then you can contact me in a private message to set up a call with screen sharing so that I can have a closer look.
1 Like
Thank you so much for the response @lassoan! Deleting the .totalsegmentator folder is the only thing I hadn’t thought of, all is now working perfectly. You are an absolute legend!
1 Like
Experienced the same “status 120” error on a fresh AWS EC2 instance today after installation of PyTorch via the extension and running TotalSegmentator.
It happened upon the first run.
Restarted Slicer and “Upgraded” once which resolved the error.
Is your problem solved by now?
@rbumm This is very useful information. I’ve updated TotalSegmentator extension following fixes in the Python package. We don’t need the SimpleITK installation hack and git installation anymore. Could you try if Slicer-5.2.1 installation with latest TotalSegmentator still runs into status 120 on a fresh EC2 instance?
1 Like
@lassoan Sure. Would I need to fork the extension or just install it from the extension manager?
Just install Slicer-5.2.1 from scratch and install TotalSegmentator extension from the Extensions Manager.
1 Like
@lassoan Now it works. On the fresh EC2 instance (g5.xlarge), PyTorch installs completely from the extension and recognizes the new NVIDIA A10G drivers.
TotalSegmentator installs upon the first run, but takes a while to complete:
Processing completed in 558.95 seconds
The second run:
Processing completed in 110.23 seconds
Awesome! Thanks for testing.
First run takes long time because the model weight files (about 1.6GB) have to be downloaded. If this delay causes a problem (slow, unreliable, etc) then we found have a look at where the files are hosted and if adding mirrors would make sense.
2 Likes
Hi I am having a similar problem. I have m2 MacBook. I have tried all the above including updating PyTorch/total seg as well as downloading git. I have also changed from GPU to CPU in edit-application setting–> volume rendering–>VTK CPU (from GPU). I am using sample data. It works for the MRI head. but when I use on CTAP or CT liver or even my own DICOM CT Chest it brings up error messages.
This is my error message
Traceback (most recent call last):
File “/Applications/Slicer.app/Contents/bin/Python/slicer/util.py”, line 2973, in tryWithErrorDisplay
yield
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 264, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 799, in process
self.logProcessOutput(proc)
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 692, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/9l/6p4fv9qs1gb_22w9xm8_fm2r0000gn/T/Slicer-omarel-taji/__SlicerTemp__2023-04-30_02+17+21.212/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/9l/6p4fv9qs1gb_22w9xm8_fm2r0000gn/T/Slicer-omarel-taji/__SlicerTemp__2023-04-30_02+17+21.212/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.
Thanks
Please start all over again.
It would be great if you could read the troubleshooting section of the TotalSegmentator extension in the beginning.
- Uninstall 3D Slicer and delete the installation directory.
- Install 3D Slicer stable in a new directory.
- Install the Pytorch extension.
- In the Pytorch extension, select CPU support just for a test.
- Install Pytorch.
- Restart 3D Slicer.
- Install the Totalsegmentator extension.
- Load the CT Chest dataset
- Run TotalSegmentator (will take longer because of using CPU).
If not working, please post the results of the log and/or complete content of progress window here.
We will handle GPU support later, probably by someone who has an M2 Macbook at hand. Not sure GPU support works in this case.
I am having a similar problem too.
I installed pytorch module and totalsegmentator in extensions manager，and installed NVIDIA CUDA11.8 and GIT.
But there was still some errors when running the program.Need help to solve this，thx！
Here are the error infos。
first try:
Traceback (most recent call last):
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 2967, in tryWithErrorDisplay
yield
File “C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 255, in onApplyButton
self.logic.setupPythonRequirements()
File “C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 660, in setupPythonRequirements
slicer.util.pip_install(self.totalSegmentatorPythonPackageDownloadUrl)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3578, in pip_install
_executePythonModule(‘pip’, args)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3540, in _executePythonModule
logProcessOutput(proc)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3509, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command '‘C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/bin/…/bin\PythonSlicer.EXE’, ‘-m’, ‘pip’, ‘install’, '[https://github.com/wasserth/TotalSegmentator/archive/b38eb449ad8652a987878a925203cbfa354e9b85.zip’]’ returned non-zero exit status 1.
second try：
Traceback (most recent call last):
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 2967, in tryWithErrorDisplay
yield
File “C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 255, in onApplyButton
self.logic.setupPythonRequirements()
File “C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/NA-MIC/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 660, in setupPythonRequirements
slicer.util.pip_install(self.totalSegmentatorPythonPackageDownloadUrl)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3578, in pip_install
_executePythonModule(‘pip’, args)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3540, in _executePythonModule
logProcessOutput(proc)
File “C:\Users\sain\Desktop\3DSilcer\Slicer 5.2.2\bin\Python\slicer\util.py”, line 3509, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command '‘C:/Users/sain/Desktop/3DSilcer/Slicer 5.2.2/bin/…/bin\PythonSlicer.EXE’, ‘-m’, ‘pip’, ‘install’, '[https://github.com/wasserth/TotalSegmentator/archive/b38eb449ad8652a987878a925203cbfa354e9b85.zip’]’ returned non-zero exit status 2.
Could you try with the latest Slicer Preview Release?
1 Like