# MONAI label does not utilize Tesala GPU

**URL:** <https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695>\
**Category:** Support\
**Created:** [June 3, 2022, 8:39pm UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695 "2022-06-03T20:39:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ron](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/ron/32/10835_2.png) [@Ron](https://discourse.slicer.org/u/Ron)\
**Post date:** [June 3, 2022, 8:39pm UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/1 "2022-06-03T20:39:05Z")

</div>

I have been working with Andres to train a MONAI label spine segmentation on a windows 10 pro workstation (Lenovo p610, 64 core CPU AMD Threadripper pro, Nvidia Tesla V100, Quadro 1000 GPU, 256Gb Ram). We have tried several times to run the model, and although it runs and trains, it appears to run on the CPU only. Running Nvidia SMI recognizes both GPUs. I have installed the server driver version of the Tesla card (V100). I wonder if anyone had the same issue/found a solution.  
Thanks, Ron

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [June 4, 2022, 2:07am UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/2 "2022-06-04T02:07:09Z")

</div>

In my experience, this is almost always due to incorrect CUDA installation and/or insufficient environmental variable settings. Nvidia-smi can only confirm that the Nvidia driver is installed correctly and that GPU is operational. CUDA set up is different.  
First you have to confirm that:

1. You are using a torch that is cuda enabled
2. Torch is indeed using the GPU (incorrect environment setup can block this).

See this answer, and follow the steps and see if you are seeing your V100 being reported.

> <https://stackoverflow.com/questions/48152674/how-to-check-if-pytorch-is-using-the-gpu>

---

<div class="post-metadata">

**Author:** ![Ron](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/ron/32/10835_2.png) [@Ron](https://discourse.slicer.org/u/Ron)\
**Post date:** [June 4, 2022, 11:39pm UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/3 "2022-06-04T23:39:18Z")

</div>

Dear Murat  
Thanks for the suggestion, I will work with Andres on Monday and see if we can find and fix what is the issue. Will post our solution. Thanks, Ron

---

<div class="post-metadata">

**Author:** ![mau\_igna\_06](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/mau_igna_06/32/9056_2.png) [@mau\_igna\_06](https://discourse.slicer.org/u/mau_igna_06)\
**Post date:** [June 5, 2022, 10:55am UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/4 "2022-06-05T10:55:48Z")

</div>

We were able to use gpus on docker enviroments using virtualgl with vglrun…  
I have read that this methos was also used for AI.  
So you need to set up a docker image test.

Maybe @lassoan has some experience woth this

Hope it helps

---

<div class="post-metadata">

**Author:** ![RafaelPalomar](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/rafaelpalomar/32/1436_2.png) [@RafaelPalomar](https://discourse.slicer.org/u/RafaelPalomar)\
**Post date:** [June 5, 2022, 12:24pm UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/5 "2022-06-05T12:24:58Z")

</div>

Hello @Ron,

a while ago I asked the same question in the MONAI Label discussions ([Generic deepgrow -- High CPU usage and low GPU usage · Discussion #282 · Project-MONAI/MONAILabel · GitHub](https://github.com/Project-MONAI/MONAILabel/discussions/282)). My feeling was that some operators in some MONAI Label applications were GPU-based while others were only CPU-based. The answer that I got points to that direction. I hope this helps.

---

<div class="post-metadata">

**Author:** ![mau\_igna\_06](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/mau_igna_06/32/9056_2.png) [@mau\_igna\_06](https://discourse.slicer.org/u/mau_igna_06)\
**Post date:** [June 5, 2022, 1:00pm UTC](https://discourse.slicer.org/t/monai-label-does-not-utilize-tesala-gpu/23695/7 "2022-06-05T13:00:05Z")

</div>

sohuld work on gpu—