# Slicer crash when call multiprocessing

**URL:** <https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044>\
**Category:** Support\
**Tags:** python\
**Created:** [September 13, 2017, 8:29pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044 "2017-09-13T20:29:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![anqiang-12sigma](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anqiang-12sigma](https://discourse.slicer.org/u/anqiang-12sigma)\
**Post date:** [September 13, 2017, 8:29pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/1 "2017-09-13T20:29:18Z")

</div>

I want to use python multiprocessing, but it makes slicer crash.

```auto
Traceback (most recent call last):
  File "C:\Program Files\Slicer 4.7.0-2017-08-16\lib\Python\Lib\multiprocessing\process.py", line 249, in _bootstrap
    sys.stdin.close()
AttributeError: 'PythonQtStdInRedirect' object has no attribute 'close'

```

Does slicer have plans to fully support Python3 and PyQt5?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 13, 2017, 8:38pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/2 "2017-09-13T20:38:38Z")

</div>

You can already build Slicer with Qt5, probably we’ll update to Qt5 in a couple of weeks.

We plan to move to Python3 when all Slicer’s dependencies support Python3.

PyQt uses GPL license, which is incompatible with Slicer’s more permissive BSD-type license (we use PythonQt, which is LGPL).

---

<div class="post-metadata">

**Author:** ![anqiang-12sigma](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anqiang-12sigma](https://discourse.slicer.org/u/anqiang-12sigma)\
**Post date:** [September 25, 2017, 3:09pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/3 "2017-09-25T15:09:23Z")

</div>

I found PythonQt too many bugs, why not replace it with PySide? It’s also LGPL.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 25, 2017, 6:49pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/4 "2017-09-25T18:49:25Z")

</div>

PySide seems to be only for Qt4 ([https://wiki.qt.io/PySide](https://wiki.qt.io/PySide)). Is PySide2 complete and stable, supporting Python3 and latest Qt5?

We don’t experience any issues with Qt wrapping using PythonQt, so unless we run into some major problems in the future, it is not likely Slicer would switch.

Do you know what would be some advantages of using PySide2 compared to PythonQt? What “bugs” you experienced with PythonQt? Are you sure you using it correctly? Have you reported the errors to PythonQt?

---

<div class="post-metadata">

**Author:** ![anqiang-12sigma](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anqiang-12sigma](https://discourse.slicer.org/u/anqiang-12sigma)\
**Post date:** [September 26, 2017, 12:57am UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/5 "2017-09-26T00:57:52Z")

</div>

1. multiprocsssing can not work
2. QAbstractItemModel createIndex overide error
3. setItemDelegateForColumn crash
4. Delegate can not work normally

The code runs fine on PyQt

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 26, 2017, 2:28am UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/6 "2017-09-26T02:28:40Z")

</div>

The issues that you describe do not seem to be difficult to address; except the last one, which I don’t understand. I would suggest to submit your questions (with more specific description of the problem) to [PythonQt / Discussion](https://sourceforge.net/p/pythonqt/discussion/).

There are some differences compared to PySide (see [http://pythonqt.sourceforge.net/Features.html](http://pythonqt.sourceforge.net/Features.html)), but there doesn’t seem to be any major limitation in PythonQt. The beauty of PythonQt is that it is so small and simple that it is easy to develop and maintain it, even for a small team. PySide has some advantages, mainly the larger user community, but I don’t think that would justify the huge work that would be required to migrate Slicer and all extensions. It is probably a better investment to spend that time with fixing issues or adding more features to Slicer.

> The code runs fine on PyQt

PyQt is irrelevant for us due to its restrictive license.

---

<div class="post-metadata">

**Author:** ![anqiang-12sigma](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anqiang-12sigma](https://discourse.slicer.org/u/anqiang-12sigma)\
**Post date:** [September 26, 2017, 2:52am UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/7 "2017-09-26T02:52:56Z")

</div>

I have already submitted a question on sourceforge.  
[Model/View/Delegate some problems!](https://sourceforge.net/p/pythonqt/discussion/631393/thread/4529adde/)

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 26, 2017, 2:57am UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/8 "2017-09-26T02:57:04Z")

</div>

To increase the chance that your questions will be answered on sourceforge, post only one question in a topic and provide much more details including a complete example that reproduces the problem.

---

<div class="post-metadata">

**Author:** ![John\_DiMatteo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/john_dimatteo/32/3168_2.png) [@John\_DiMatteo](https://discourse.slicer.org/u/John_DiMatteo)\
**Post date:** [March 19, 2019, 7:30pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/9 "2019-03-19T19:30:16Z")

</div>

You can work around the Slicer `PythonQtStdInRedirect` multiprocessing crash by temporarily changing sys.stdin. For example, the below multiprocessing works in slicer:

```
from multiprocessing import Pool
import sys, os
import math

original_stdin = sys.stdin
sys.stdin = open(os.devnull)
try:
  p = Pool(5)
  print(p.map(math.sqrt, [4, 8, 16]))
finally:
  sys.stdin.close()
  sys.stdin = original_stdin
```

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 7, 2020, 6:37pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/10 "2020-02-07T18:37:38Z")

</div>

Thanks @John_DiMatteo for sending the `Pool` recipe. Did you also get it working for `multiprocessing.Process`? I’m looking for something that will work robustly on all platforms and I’m thinking I’ll use `subprocess` with `PythonSlicer`.

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 8, 2020, 8:03pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/11 "2020-02-08T20:03:55Z")

</div>

This trick does not work on Windows, and will never work. Although it works perfectly on Mac and linux.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 8, 2020, 8:17pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/12 "2020-02-08T20:17:20Z")

</div>

Thanks for the info @Alex_Vergara. Yes, I did some testing and now I’m thinking I’ll use `QProcess` since it has a very clean API and is already integrated with the signals/slots event loop. Turns out it’s pretty straightforward to start a `PythonSlicer` process and send pickled data back and forth through stdin/stdout. Will share some sample code if I get something working well on all platforms.

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 10, 2020, 10:52am UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/13 "2020-02-10T10:52:42Z")

</div>

> [@pieper](#):
>
> QProcess

You will end up by facing the same problem in Windows:

- [python - What's the best way to duplicate fork() in windows? - Stack Overflow](https://stackoverflow.com/questions/23397/whats-the-best-way-to-duplicate-fork-in-windows)
- [What's the equivalence of os.fork() on Windows with Python? - Stack Overflow](https://stackoverflow.com/questions/2738809/whats-the-equivalence-of-os-fork-on-windows-with-python)

There is simply not solution available to Windows. The multiprocessing shall be done in a separate script with the `if self. __name__ == "main"` trick

- [python - What does if \_\_name\_\_ == "\_\_main\_\_": do? - Stack Overflow](https://stackoverflow.com/questions/419163/what-does-if-name-main-do)

In this last case the script itself must be called in a way that it is the main module, otherwise the multiprocessing spawn will just create several non functioning processes. This is a real fault in windows conception. The best explanation is [this](https://stackoverflow.com/a/52476456/10059284)

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 10, 2020, 1:51pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/14 "2020-02-10T13:51:32Z")

</div>

Agreed, Windows and Unix took very different approaches years ago and it’s caused a lot of complexity.

For my particular use case the QProcess approach is working well for me on both Windows and Mac. In particular we have some computationally intense single threaded python code we want to apply to a set of data already loaded in Slicer. So launching a PythonSlicer instances and sending them the data will work. I don’t exactly want to fork the whole Slicer process anyway and may at some point want to run these processes on another machine, e.g. via ssh. I’m still working on complete examples and then I’ll push some code.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 10, 2020, 8:41pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/15 "2020-02-10T20:41:00Z")

</div>

Here’s the example using QProcess:

> **[pieper/SlicerProcesses](https://github.com/pieper/SlicerProcesses)**
>
> Slicer modules for running subprocesses to operate on data in parallel - pieper/SlicerProcesses

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 11, 2020, 2:37pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/17 "2020-02-11T14:37:31Z")

</div>

I have successfully translated your code into mine with some caveats:

- Basically you need to decode the stdout to the correct data type, it is not enough to ask pickle to do it
- You need to deepcopy your input data
- You can add a progress bar 😉

```python
import qt
import json
import pickle
import slicer
import copy
import numpy as np

from Logic import logging, utils
from slicer.ScriptedLoadableModule import *

#
# ProcessesLogic
#

class ProcessesLogic(ScriptedLoadableModuleLogic):

    def __init__ (self, parent = None, maximumRunningProcesses=None, completedCallback=lambda : None):
        super(). __init__ (parent)
        self.results = {}
        if not maximumRunningProcesses:
            cpu_cores, cpu_processors, lsystem = utils.getSystemCoresInfo()
            self.maximumRunningProcesses = cpu_cores
        self.completedCallback = completedCallback

        self.QProcessStates = {0: 'NotRunning', 1: 'Starting', 2: 'Running',}

        self.processStates = ["Pending", "Running", "Completed"]
        self.processLists = {}
        for processState in self.processStates:
            self.processLists[processState] = []

        self.canceled = False
        self.ProgressDialog = slicer.util.createProgressDialog(
                parent=None, value=0, maximum=100)
        labelText = "Processing ..."
        self.ProgressDialog.labelText = labelText
        self.ProgressDialog.show()
        slicer.app.processEvents()

    def setMaximumRunningProcesses(self, value):
        self.maximumRunningProcesses = value

    def saveState(self):
        state = {}
        for processState in self.processStates:
            state[processState] = [process.name for process in self.processLists[processState]]
        self.getParameterNode().SetAttribute("state", json.dumps(state))

    def state(self):
        return json.loads(self.getParameterNode().GetAttribute("state"))

    def addProcess(self, process):
        self.processLists["Pending"].append(process)
        self.ProgressDialog.maximum = len(self.processLists["Pending"])

    def run(self):
        while len(self.processLists["Pending"]) > 0:
            if len(self.processLists["Running"]) >= self.maximumRunningProcesses:
                break
            process = self.processLists["Pending"].pop()
            process.run(self)
            self.processLists["Running"].append(process)
            self.saveState()

    def onProcessFinished(self,process):
        self.processLists["Running"].remove(process)
        self.processLists["Completed"].append(process)
        self.ProgressDialog.value += 1
        slicer.app.processEvents()
        self.saveState()
        if len(self.processLists["Running"]) == 0 and len(self.processLists["Pending"]) == 0:
            for process in self.processLists["Completed"]:
                k, v = process.result[0], process.result[1]
                self.results[k] = v
            self.ProgressDialog.close()
            self.completedCallback()
        if self.ProgressDialog.wasCanceled:
            self.canceled = True
        else:
            self.run()

class Process(qt.QProcess):
    """TODO: maybe this should be a subclass of QProcess"""

    def __init__ (self, scriptPath):
        super(). __init__ ()
        self.name = "Process"
        self.processState = "Pending"
        self.scriptPath = scriptPath
        self.debug = False
        self.logger = logging.getLogger("Dosimetry4D.qprocess")

    def run(self, logic):
        self.connect('stateChanged(QProcess::ProcessState)', self.onStateChanged)
        self.connect('started()', self.onStarted)
        finishedSlot = lambda exitCode, exitStatus : self.onFinished(logic, exitCode, exitStatus)
        self.connect('finished(int,QProcess::ExitStatus)', finishedSlot)
        self.start("PythonSlicer", [self.scriptPath,])

    def onStateChanged(self, newState):
        self.logger.info('-'*40)
        self.logger.info(f'qprocess state code is: {self.state()}')
        self.logger.info(f'qprocess error code is: {self.error()}')

    def onStarted(self):
        self.logger.info("writing")
        if self.debug:
            with open("/tmp/pickledInput", "w") as fp:
                fp.buffer.write(self.pickledInput())
        self.write(self.pickledInput())
        self.closeWriteChannel()

    def onFinished(self, logic, exitCode, exitStatus):
        self.logger.info(f'finished, code {exitCode}, status {exitStatus}')
        stdout = self.readAllStandardOutput()
        self.usePickledOutput(stdout.data())
        logic.onProcessFinished(self)

class ConvolutionProcess(Process):
    """This is an example of running a process to operate on model data"""

    def __init__ (self, scriptPath, initDict, iteration):
        super(). __init__ (scriptPath)
        self.initDict = copy.deepcopy(initDict)
        self.iteration = iteration
        self.name = f"Iteration {iteration}"

    def pickledInput(self):
        return pickle.dumps(self.initDict)

    def usePickledOutput(self, pickledOutput):
        output = np.frombuffer(pickledOutput, dtype=float)
        #output = pickle.loads(pickledOutput)
        self.result = [self.iteration, output]

```

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 11, 2020, 2:40pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/18 "2020-02-11T14:40:29Z")

</div>

Remaining question, Have you tested this in Windows??

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [February 11, 2020, 3:00pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/19 "2020-02-11T15:00:49Z")

</div>

> [@Alex\_Vergara](#):
>
> it is not enough to ask pickle to do it

It should be possible to define custom pickling methods for VTK and MRML classes. It would be nice if you could look into this. It may be more elegant than implementing some custom serialization solution for each project.

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 11, 2020, 3:05pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/20 "2020-02-11T15:05:00Z")

</div>

In my case I just output a numpy array in the cpu intensive calculation. But you are right, standard pickling methods would be great, specially for generic intensive computations.

This algorithm can be made generic for any process that basically a class that accepts dictionaries as input (any kind of data wrapped into a dictionary), watch my convolution class:

```python
import numpy as np
import pickle
import sys
import Logic.logging as logging

class Convolution:
    '''convolution class to be runned either sequentially or parallel
    for parallel execution a wrapper class must be implemented as:
        def multi_run_wrapper(args):
            convolution = Convolution(*args) 
            return convolution.convolute()
    where args must be a tuple with
        args = (indexes, p0, DVK, distance_kernel, dens_array, act_array, num, boundary)
    '''
    def __init__ (self, indexes, p0, DVK, distance_kernel, dens_array, act_array, num, boundary):
        self.logger = logging.getLogger('Dosimetry4D.convolution')
        self.indexes=indexes
        self.p0=p0
        self.DVK=DVK
        self.distance_kernel=distance_kernel
        self.dens_array=dens_array
        self.act_array=act_array
        self.num=num

    def convolute(self):
        ''' Non homogeneous convolution in a list of voxel (vectorized)
            Variable density correction by distance
        '''

        res = np.zeros_like(self.indexes, dtype=float)
        ..... Computation .....

        return res

try:
    pickledInput = sys.stdin.buffer.read()
    input = pickle.loads(pickledInput)

    convolution = Convolution(**input) # Making the convolution pickable
    output = convolution.convolute() # Invoking the function is now thread safe

    sys.stdout.buffer.write(pickle.dumps(output))
except Exception as e:
    print(e)

```

And the calling method is

```python
                    logic = ProcessesLogic()
                    scriptPath = self.script_path / "Logic" / "Convolution.py"

                    chunks = 100 # to avoid overheading 
                    iteration = 0
                    args1 = { # wrap arguments for multithreading
                            'p0':p0, 
                            'DVK':self.DVK, 
                            'distance_kernel':self.distanceKernel, 
                            'dens_array':self.densityArray, 
                            'act_array':self.activityArray,
                            'num': num, 
                            'boundary':'repeat'
                    } 
                    while acc < length1:
                        stepsize = max(1, min(int(length1/chunks), length1-acc))
                        lindex = copy.deepcopy(args1)
                        lindex['indexes'] = np.array(indexes[acc:acc + stepsize])
                        convolutionProcess = ConvolutionProcess(scriptPath, lindex, iteration)
                        logic.addProcess(convolutionProcess)
                        acc += stepsize
                        iteration += 1
                    
                    results=[]
                    self.logger.info(f"Launching convolution in {self.CpuCores} CPU cores")
                    slicer.app.processEvents()

                    try:
                        logic.run()
                        canceled = logic.canceled
                    except Exception as e:
                        print(e)
                        return False

```

---

<div class="post-metadata">

**Author:** ![Alex\_Vergara](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/alex_vergara/32/15205_2.png) [@Alex\_Vergara](https://discourse.slicer.org/u/Alex_Vergara)\
**Post date:** [February 11, 2020, 3:24pm UTC](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044/21 "2020-02-11T15:24:28Z")

</div>

For reference, when you use the pickle method to decode the output you will usually get the following error:

```bash
  File "/Users/alexvergaragil/Documents/GIT/dosimetry4d/Dosimetry4D/Logic/Process.py", line 131, in usePickledOutput
    output = pickle.loads(pickledOutput)
_pickle.UnpicklingError: could not find MARK

```

This only means pickle does not know how to read the output

[Next page](https://discourse.slicer.org/t/slicer-crash-when-call-multiprocessing/1044.md?page=2)