Failed to fetch models from remote server. ..Make sure server address is correct and <server_uri>/v1/models is accessible in browser

Support
1

when I use Nvidia AIAA，error happen. feedback like below.
could you do me a favor to tell me how to resolve this problem?

Failed to fetch models from remote server. Make sure server address is correct and <server_uri>/v1/models is accessible in browser
Traceback (most recent call last):
  File "/Applications/Slicer.app/Contents/Extensions-31734/NvidiaAIAssistedAnnotation/lib/Slicer-5.3/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py", line 198, in fetchAIAAModels
    models = self.logic.list_models()
  File "/Applications/Slicer.app/Contents/Extensions-31734/NvidiaAIAssistedAnnotation/lib/Slicer-5.3/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py", line 1076, in list_models
    return aiaaClient.model_list(label)
  File "/Applications/Slicer.app/Contents/Extensions-31734/NvidiaAIAssistedAnnotation/lib/Slicer-5.3/qt-scripted-modules/NvidiaAIAAClientAPI/client_api.py", line 190, in model_list
    status, response = AIAAUtils.http_method('GET', self._server_url, selector)
  File "/Applications/Slicer.app/Contents/Extensions-31734/NvidiaAIAssistedAnnotation/lib/Slicer-5.3/qt-scripted-modules/NvidiaAIAAClientAPI/client_api.py", line 626, in http_method
    conn.request(method, selector)
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
2

Developers of NVIDIA AIAA extension deprecated it and switched over to developing MONAILabel extension instead. MONAILabel is not only for running existing models but you can use it to train your own models.

For segmenting almost anything on CT (all major organs, bones, vessels, etc.) you can now use the TotalSegmentator extension (100+ structures, fully automatically in 2 minutes, even without a GPU).

3

Thank you for help!
Your answer help me a lot!
I try to use total segments extension.I installed successfully.
But errors happened.
my computer is MacBook Air m1. 16gb
slicer version is 5.2.2
Could you tell me how to reslolve this problem?

errors feedback :
failed to compute results.

Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_15+52+24.819/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_15+52+24.819/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.

raceback (most recent call last):
File “/Applications/Slicer.app/Contents/bin/Python/slicer/util.py”, line 2967, in tryWithErrorDisplay
yield
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 264, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 799, in process
self.logProcessOutput(proc)
File “/Applications/Slicer.app/Contents/Extensions-31382/TotalSegmentator/lib/Slicer-5.2/qt-scripted-modules/TotalSegmentator.py”, line 692, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_15+52+24.819/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_15+52+24.819/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.

4

I reinstalled the slicer again，preview version 5.3.0.
and install PyTorch for cpu ,install total segment extension again.
errors happened again.

what can I do to resolve this problem?
feedback:
Failed to compute results.

Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_21+25+33.298/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_21+25+33.298/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.

Traceback (most recent call last):
File “/Applications/Slicer.app/Contents/bin/Python/slicer/util.py”, line 2973, in tryWithErrorDisplay
yield
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 264, in onApplyButton
self.logic.process(self.ui.inputVolumeSelector.currentNode(), self.ui.outputSegmentationSelector.currentNode(),
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 799, in process
self.logProcessOutput(proc)
File “/Applications/Slicer.app/Contents/Extensions-31734/TotalSegmentator/lib/Slicer-5.3/qt-scripted-modules/TotalSegmentator.py”, line 692, in logProcessOutput
raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command ‘[’/Applications/Slicer.app/Contents/bin/…/bin/PythonSlicer’, ‘/Applications/Slicer.app/Contents/lib/Python/bin/TotalSegmentator’, ‘-i’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_21+25+33.298/total-segmentator-input.nii’, ‘-o’, ‘/private/var/folders/xm/g5xx81wd55b1rvslqnrdgwv80000gn/T/Slicer-cy/__SlicerTemp__2023-05-12_21+25+33.298/segmentation’, ‘–ml’, ‘–task’, ‘total’, ‘–fast’]’ returned non-zero exit status 120.

5

I’m having trouble with ‘Total segmentator’ on Linux too, since recently, don’t know if mines are related to yours.

....
....

    ret = self.network.predict_3D(data, do_mirroring=do_mirroring, mirror_axes=mirror_axes,
  File "/home/user/programs/Slicer/lib/Python/lib/python3.9/site-packages/nnunet/network_architecture/neural_network.py", line 147, in predict_3D
    res = self._internal_predict_3D_3Dconv_tiled(x, step_size, do_mirroring, mirror_axes, patch_size,
  File "/home/user/programs/Slicer/lib/Python/lib/python3.9/site-packages/nnunet/network_architecture/neural_network.py", line 348, in _internal_predict_3D_3Dconv_tiled
    gaussian_importance_map[gaussian_importance_map == 0] = gaussian_importance_map[
RuntimeError: "min_all" not implemented for 'Half'
Exception ignored in: <totalsegmentator.libs.DummyFile object at 0x7fd3d45f8c10>
AttributeError: 'DummyFile' object has no attribute 'flush'

I noticed that it happens when the input is a whole volume, for instance, CTA-cardio. If it is cropped down to the abdomen or the thorax, ‘Total segmentator’ completes nicely. And the cropped volume must not be too big. If it includes abdomen and thorax, then it fails likewise.

Using Fast mode with AMD GPU on laptop and desktop.

6

Error 120 means PyTorch is not installed correctly or the version is incompatible with your hardware. Restarting your computer and installing pytorch manually may fix th problem.

What is your current pytorch version?
Do you use cuda or cpu version?

To solve this issue, you probably need to upgrade to pytorch>=2. Please uninstall Pytorch restart Slicer, and then force installation using pip_install('torch>=2')

7

I was using 1.8.1+cpu.

After installing Torch 2 as you advised, ‘Total segmentator’ works normally with a full CTA-cardio volume as test.

My machines do not run with NVidia GPUs, so I’m compelled to use CPU only;

Thank you very much.

8

On linux, you can try to utilize your AMD GPU by using the ROCm edition of PyTorch. See manual install instructions here. Let us know if it works and if the performance is better then with CPU.

9

I’m using SlicerPyTorch to install Torch. It always install 1.8.1. Is there another helper module that would install the latest version ? I can of course note the command and type it whenever needed.

10

I don’t know why on some systems why not the latest pytorch is installed by default. Maybe because an older version of ltt is installed, which does not know about torch-2.x yet? If you install Slicer in a new folder and install TotalSegmentator then which torch version gets installed?

11

Using Slicer preview 5.3.0-2023-05-11 r31736 / 10e85ca in a new folder, ‘Total segmentator’ installs Torch 1.8.1+cpu.

12

I installed this ROCm edition in Slicer’s python console using their instructions :

pip_install(“torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2”)

The installation was flawless : 2.0.2+rocm5.4.2 in ‘PyTorch utils’. But ‘Total segmentator’ can’t use the GPU on my laptop featuring an ‘AMD Ryzen 5 2500U with Radeon Vega Mobile Gfx’ APU with shared VRAM.

I suppose it’s because it’s an integrated GPU. I’ll be able to test on my work machine tomorrow or on Monday. It has a Radeon RX 6700 XT video card, it should hopefully be seen and used by Torch. I’ll keep you updated of course. Thanks for your input.

13

I’m not sure why not the latest pytorch is installed. I’ve submitted an issue:

I’m not surprised that it does not work with an integrated GPU. Looking forward to seeing if it works with the discrete AMD GPU.

14

Thanks again for your answer!
According to your suggestion,I uninstalled the PyTorch (version 1.8.1),install PyTorch (version 2.0.1).
This time, total segmentor works!
I succeed in segmenting abdominal organ in fast mode，although fail in normal mode（taking about 50 minutes，cpu mode）.
Thank you very much for your help!
Wish you happy weekend!

15

Installed the rocm5.4.2 version on my desktop with discrete ‘RX 6700 XT’ video card using pip_install("torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2"). Things got worse.

It does not propose GPU at all. And fails to process a cropped CTA-cardio volume, while this would succeed with Torch < 2.

Uninstalled rocm5.4.2 and installed torch with pip_install('torch>=2'). Torch version is 2.0.1+cu117. GPU is not used as expected. But processing a full CTA-cardio succeeds.

16

It seems that nnunet developers did not make their framework compatible with AMD. They only use CUDA on Linux, so TotalSegmentator has to use a special version that Jakob modified to be able to run on CPU and Windows. So, it is not surprising that AMD does not work without some fixes.

17

Yes, everything I tried failed. It’s not a big issue as I work with cropped volumes very often, so CPU mode is decently fast. It’s good to know there’s an effort being made towards AMD GPUs.