Can I use multiprocessing in a Python module

Development
1

Hello，
I am trying to use python multiprocessing in Slicer to deal with a large amount of calculations. I tried ThreadPool but it brings little effect.Then I tried Pool and Process,but they just can’t work. Could you tell me how to use multiprocessing in a Python module.
I am using Slicer4.8.0 in Windows7.

thank you

2

Hi @langderyos,

Thanks for your report.

Could you share an example of python code allowing to reproduce the problem ?

3

I tried various approaches around two years ago, and what I found out is that you cannot use the threads to extend the CPU “allowance” of Slicer. So it will share the resources of the main thread which runs on a single core.
Of course this may have changed, just one thing I thought is good to expect.

4

All the underlying libraries (VTK, ITK, numpy, etc) are highly optimized and using multiple threads, some functions may even use GPU. Instead of implementing any low-level processing in Python, I could almost always find a combination of existing methods that do what I need. In the extremely rare case there is some computationally intensive task that I cannot find implemented in any of the libraries, I create a C++ loadable module, and implement the algorithm in a VTK class and put it in the logic component of the module. The class gets Python-wrapping automatically, so I can use it from Python.

1 Like
5

Hi, jcfr

Thanks for your reply!

In fact, I just tried a few lines of code to test if multiprocessing works in a python module.I am not good at programming so maybe there is something wrong in the code.

Example 1

import os
from multiprocessing.dummy import Pool as ThreadPool
def onepool(x):
  return(x**2)
a=range(100)
pool = ThreadPool(9)
b=pool.map(onepool,a)
pool.close()
pool.join()

This example works.

Example 2

import os
from multiprocessing import Pool
def onepool(x):
  return(x**2)
a=range(100)
pool = Pool()
b=pool.map(onepool,a)
pool.close()
pool.join()

This example doesn’t work. Slicer has no respond.

6

Hi, cpinter
Thanks for your reply!

7

Hi, lassoan
Thanks for your reply!
Your reply is very help.Maybe I should find an alternate way.

8

I have tried several methods, in particular your example 1 and still have the same behaviour, Slicer uses only 1 processor and share all resources and threads in that processor. The rest of processors remain idle. I have tried with several examples that releases GIL, joblib with threads, pathos, multiprocessing… Nothing seems to work properly, sequential codes are still faster than the threaded ones.
Is there a way you can actually launch several threads in python in Slicer??

9

I’m not sure if you can run several threads but you can certainly run several processes.

You can use Slicer’s CLI module interface to synchronize execution and pass data between the main process and the parallel processes. CLI modules can be implemented in Python or C++.

We plan to have native Python3 support and ability to install any Python packages within a few weeks. This will allow you to use any of the native Python multi-processing methods. It won’t make things faster or simpler, the only difference is that you can use pure Python mechanisms instead of Slicer-specific ones.

10

Really good news!!! Thanks! This will be a major jump in Slicer. Is there anyway I can help??

11

Hi all,

I am new to the Slicer CLI world as well and was looking for any insight on the following:

Currently, I am processing some extensive vtkModifiedBSPTree line intersections (iterating over about 6000) in a python slicer module and am currently iterating through them in the Logic portion of my module. As pointed out this can be quite slow as only one cpu core is being used. As far as I know, I should move these calculations to a python-based CLI module and interface with it to speed up this process. My question is; is this correct? Will this speed up this process (perhaps if I use multiprocessing within the cli module)? Or will this only allow this process to be run asynchronously and thus allow the user to interact with slicer will it is being done.

Many thanks,
Evan

12

CLI modules run asynchronously, without blocking the main thread. So, moving the computationally intensive part of your module to a CLI would allow you to keep the application responsive.