PyTorch3D in 3D Slicer

Development
1

As part of SlicerSALT, the SlicerDentalModelSeg extension used PyTorch3D to run segment and number teeth from a vtkMRMLModelNode of an intraoral scan. There was some difficulty in getting PyTorch3D to install in Slicer, and it currently only works on Linux.

Given that there may be more projects in the future that could benefit from PyTorch3D, and it would be good if SlicerDentalModelSeg could run on Windows and Mac as well as Linux, I am wondering if an extension can be made that will take care of the install. I know that SlicerPyTorch exists to ease the use of PyTorch in 3D Slicer, which leads to few couple questions:

  1. Would it make sense to create a SlicerPyTorch3D extension that handled the install? If so, what is the relation to SlicerPyTorch?
  2. Would it instead make sense to have SlicerPyTorch (optionally) install PyTorch3D?
  3. Has anyone already installed PyTorch3D in Slicer such that it works for all platforms, or even some of them?
2

Agreed, PyTorch3D sounds like something we’d want to have available as widely as possible, although I have not tried it myself… I think the idea of adding it to SlicerPyTorch makes sense since PyTorch is a core dependency of PyTorch3D.

3

@pieper @jcfr
I am working to deploy a Slicer extension or intraoral scans that uses PyTorch3D. It is working properly, but I was able to install it only in Slicer on Linux. I tried SlicerPyTorch but it does not install PyTorch3D in Slicer on Windows.
Back in Sep 2022, Steve had suggested that it made sense to add PyTorch3D to SlicerPyTorch. Can one of you please guide me how to install PyTorch3D in Slicer on Windows? If code development needs to be done, I am willing to work on contributing to the code, but I would really appreciate your guidance on what I need to do. Thank you.

4

@allemangd @lassoan
I am a new user and I was not able to tag David and Andras in my previous message ( it said there was a limit of reply to only two users) so I am resending to you.
I am working to deploy a Slicer extension or intraoral scans that uses PyTorch3D. It is working properly, but I was able to install it only in Slicer on Linux. I tried SlicerPyTorch but it does not install PyTorch3D in Slicer on Windows.
Back in Sep 2022, Steve had suggested that it made sense to add PyTorch3D to SlicerPyTorch. Can one of you please guide me how to install PyTorch3D in Slicer on Windows? If code development needs to be done, I am willing to work on contributing to the code, but I would really appreciate your guidance on what I need to do. Thank you.

5

PyTorch Slicer extension is only needed because PyTorch Python package is over the maximum size limit of PyPI and therefore it requires special mechanisms to download and install. For all other Python packages, you should be able to use slicer.util.pip_install().

The problem with PyTorch3D is that it does not have binary packages on PyPI. Many people complain about this - see the 40 open issues about installation in their tracker. Considering that having packages on PyPI is such a basic requirement and PyTorch3D developers don’t care about it, the only recommendation I can make is avoid using PyTorch3D.

6

@lassoan @pieper

Thank you. Can you please suggest another framework that can replace PyTorch3D?

7

Sorry, I don’t have any suggestion. Others might.
If you find alternatives then please report it back here.

8

I am planning to investigate issues with Slicer on Windows installation using as possible alternative frameworks, Kaolin, Tensorflow3D, Tensorflow graphics and Unity Barracuda.

If you have any indication on which one to start with, I appreciate any guidance.

1 Like
9

The field is evolving very quickly, so it’s hard to know which will work best for your research. As a general approach tough, I’d suggest using an external python environment with miniconda to isolate a package with all the pytorch3d dependencies and then communicate with Slicer through files or network connections.

10

Thank your for your answer @pieper.
I am going to try your suggestion.

11

@pieper @jcfr
I investigated some technics to find a solution.

First, I tried to use docker in Python with the docker library. The docker library is an API that uses docker on the desktop. It is a problem because that means the users have to install docker, this idea is not an option.

The error I got is below.

client = docker.from_env()
client.containers.run(‘hunath/pythontest’,volumes=[‘path/TestDocker:/app/’] )
raise DockerException(docker.errors.DockerException: Error while fetching server API version:(‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

The error was obtained on Windows without docker installed.

The second test was to use the subprocess library with Python environment create by miniconda or the venv library.
The different difficulties I meet were:

  • using subprocesses on Windows because python couldn’t open my python file

command_line =[os.path.normpath(‘C:/Users/…/Python/Python36/python’),os.path.normpath(‘C:/Users/…/testpython.py’)]
subprocess.run(command_line,env=slicer.util.startupEnvironment())

CompletedProcess(args=[‘C:\Users\…\Python\Python36\python’, ‘C:\Users\…\testpython.py’], returncode=2)
[FD] C:\Users.…\Python\Python36\python: can’t open file ‘C:\Users.…\testpython.py’: [Errno 2] No such file or directory

  • I don’t know how to install packages in a specific environment with pip. I suppose I should use the argument target of pip but I don’t know which folder I should target.
  • For miniconda, I tried to use the conda.cli.python_ali library, but it was impossible to import in Slicer. I suppose it’s the same problem as docker because this library it’s also an API.

If you have an idea how to solve any of these problems, that would be very helpful. If you have links or Python libraries that might be interesting to investigate, I would be interested.

12

Many Slicer extensions use PyTorch, for example TotalSegmentator and HDBrainExtraction. You can use them as examples for installing and using PyTorch in an extension.

1 Like
13

Thanks so much Andras! Our issue is that at least 3 Slicer extensions that our group developed, DentalModelSeg, ALIIOS and ARegIOS use Pytorch 3D. @juanprietob suggested using Docker which Nathan has not been able to install in Windows. Can someone please provide guidance on how to install Docker in Windows?
@pieper suggested using an external python environment with miniconda to isolate a package with all the pytorch3d dependencies and then communicate with Slicer through files or network connections.
Can one of you please suggest how to connect miniconda with Slicer through files or network connections?

14

If you want to control the Pytorch3D process from Slicer, then using QProcess is a good option. You can send commands and data to the miniconda instance using stdin and get results via stdout, like what’s done in the Processes module of the ParallelProcessing extension. In addition to stdin/stdout, if the processes share a filesystem they can read and write files to exchange data.

Another option is to create a server, like in MONAI Label, or use OpenIGTLink.

15

suggested using an external python environment with miniconda to isolate

See https://slicer.readthedocs.io/en/latest/developer_guide/script_repository.html#launch-external-process-in-startup-environment

And if using the subprocess.check_output referenced above is not enough … you could achieve similar outcome leveraging QProcess.

process = qt.QProcess()
process.setProcessChannelMode(qt.QProcess.ForwardedChannels)

processEnv = qt.QProcessEnvironment()
for var_name, var_value in slicer.util.startupEnvironment().items():
    processEnv.insert(var_name, var_value)

process.setProcessEnvironment(processEnv)

process.setProgram("python3.11")
process.setArguments(["-c", "import sys; print(sys.version)"])

process.start()
process.waitForStarted()

communicate with Slicer through files

When invoking the external process, you would indicate what should be the input/output files/directories.

communicate with Slicer through […] network connections

I suggest to first looking into a file based integration

17

Hello !

Just an update, I managed to run some code using pytorch3d on windows.
To do this I used WSL. I installed miniconda on it and in a new environment I was able to download pytorch3d and all the other libraries needed for my code. The code is then able to run correctly using pytorch3d.
I’m still trying to automate the wsl installation to simplify it. :slight_smile:

1 Like
18

Hi
I have problem in install pytorch

the totalsegmentor need to download pytorch and after download it , it erroer in installion

19

Download and install the PyTorch extension, then install PyTorch from there.

20

If you have a failed Pytorch install attempt (you have run out of disk space or there was network instability during installation) then you may need to uninstall Pytorch first (using the extension), restart Slicer, and then install Pytorch again. Note that you may need 20GB free disk space during installation (the final amount of used disk space will be less, but you need a lot of space during installation).