Can I use multiprocessing in a Python module

(Langderyos) #1

Hello,
I am trying to use python multiprocessing in Slicer to deal with a large amount of calculations. I tried ThreadPool but it brings little effect.Then I tried Pool and Process,but they just can’t work. Could you tell me how to use multiprocessing in a Python module.
I am using Slicer4.8.0 in Windows7.

thank you

(Jean Christophe Fillion Robin) #2

Hi @langderyos,

Thanks for your report.

Could you share an example of python code allowing to reproduce the problem ?

(Csaba Pinter) #3

I tried various approaches around two years ago, and what I found out is that you cannot use the threads to extend the CPU “allowance” of Slicer. So it will share the resources of the main thread which runs on a single core.
Of course this may have changed, just one thing I thought is good to expect.

(Andras Lasso) #4

All the underlying libraries (VTK, ITK, numpy, etc) are highly optimized and using multiple threads, some functions may even use GPU. Instead of implementing any low-level processing in Python, I could almost always find a combination of existing methods that do what I need. In the extremely rare case there is some computationally intensive task that I cannot find implemented in any of the libraries, I create a C++ loadable module, and implement the algorithm in a VTK class and put it in the logic component of the module. The class gets Python-wrapping automatically, so I can use it from Python.

1 Like
(Langderyos) #5

Hi, jcfr

Thanks for your reply!

In fact, I just tried a few lines of code to test if multiprocessing works in a python module.I am not good at programming so maybe there is something wrong in the code.

Example 1

import os
from multiprocessing.dummy import Pool as ThreadPool
def onepool(x):
  return(x**2)
a=range(100)
pool = ThreadPool(9)
b=pool.map(onepool,a)
pool.close()
pool.join()

This example works.

Example 2

import os
from multiprocessing import Pool
def onepool(x):
  return(x**2)
a=range(100)
pool = Pool()
b=pool.map(onepool,a)
pool.close()
pool.join()

This example doesn’t work. Slicer has no respond.

(Langderyos) #6

Hi, cpinter
Thanks for your reply!

(Langderyos) #7

Hi, lassoan
Thanks for your reply!
Your reply is very help.Maybe I should find an alternate way.

(Alex Vergara) #8

I have tried several methods, in particular your example 1 and still have the same behaviour, Slicer uses only 1 processor and share all resources and threads in that processor. The rest of processors remain idle. I have tried with several examples that releases GIL, joblib with threads, pathos, multiprocessing… Nothing seems to work properly, sequential codes are still faster than the threaded ones.
Is there a way you can actually launch several threads in python in Slicer??

(Andras Lasso) #9

I’m not sure if you can run several threads but you can certainly run several processes.

You can use Slicer’s CLI module interface to synchronize execution and pass data between the main process and the parallel processes. CLI modules can be implemented in Python or C++.

We plan to have native Python3 support and ability to install any Python packages within a few weeks. This will allow you to use any of the native Python multi-processing methods. It won’t make things faster or simpler, the only difference is that you can use pure Python mechanisms instead of Slicer-specific ones.

(Alex Vergara) #10

Really good news!!! Thanks! This will be a major jump in Slicer. Is there anyway I can help??