Slicer crashes when using QThreadPool to start QRunnable in python module

Support
1

I Would like to move some jobs to another QThread to avoid the frozen of the main thread (GUI).

I tried python ThreadPoolExecutor to create a thread. However, the child thread is too slow. According to the suggestion of multithreading-in-extension, I tried QTimer. It also frozen GUI.

Now I am trying QThreadPool by following multithreading in pyqt example:

import vtk, qt, ctk, slicer
from slicer.ScriptedLoadableModule import *

class Worker(qt.QRunnable):
    '''
    Worker thread
    '''

    def __init__(self):
        super(Worker, self).__init__()

    def run(self):
        '''
        Your code goes in this function
        '''
        print("Thread start")
        time.sleep(5)
        print("Thread complete")

class TestWidget(ScriptedLoadableModuleWidget):
    def __init__(self):
        self.thread_pool = qt.QThreadPool()

    def setup(self):
         ScriptedLoadableModuleWidget.setup(self)
         parametersCollapsibleButton = ctk.ctkCollapsibleButton()
         parametersCollapsibleButton.text = "Parameters"
         self.layout.addWidget(parametersCollapsibleButton)

         parametersFormLayout = qt.QFormLayout(parametersCollapsibleButton)


         self.listenButton = qt.QPushButton("Test")
         self.listenButton.toolTip = "Test"
         self.listenButton.enabled = True
         self.listenButton.connect('clicked(bool)', self.onListenButton)
         parametersFormLayout.addRow(self.listenButton)

    def onListenButton(self):
        self.worker = Worker() 
        self.thread_pool.start(self.worker)

If I click the button to test it, slicer crashes with the following error:
"
Received signal 11 SEGV_MAPERR 559d00000002
#0 0x7f58faea558f
#1 0x7f58f98d785d
#2 0x7f58faea5a9e
#3 0x7f58e49ff890
#4 0x7f58eaddfe2d QThreadPoolThread::run()
#5 0x7f58eade9554 QThreadPrivate::start()
#6 0x7f58e49f46db start_thread
#7 0x7f58d877e88f clone
"
I have no idea what is wrong.
Could anyone point me how to create QThread in Slicer with Python? Thanks.

2

Python multi-threading is really messy in general and it is further complicated by using a Python interpreter embedded in an application.

I would recommend running background processing in a Python CLI module (which runs processing in a separate process) or implement it in a C++ loadable module (where you can use QThread).

3

Thank you very much!

4

you can do the following:

    def onListenButton(self):
        self.worker = Worker() 
        original_stdin = sys.stdin # Unlock SlicerPython GIL
        sys.stdin = open(os.devnull)
        try:
            self.thread_pool.start(self.worker)
        except Exception as e: # Is something wrong happens, force to terminate the pool
            self.thread_pool.terminate()
        finally:
            self.thread_pool.join()
            sys.stdin.close() # Restores SlicerPython GIL
            sys.stdin = original_stdin

This shall work :wink:

1 Like
5

Do you know what are the side effects of this?

6

yep, if you try to do this more than once slicer crashes, but if you wait until the end it works pretty well. So you shall enforce some kind of lock yourself.
Oh, and this only works with python3, I forgot to mention this.

1 Like
7

To elaborate more on this: SlicerPython is attached to the console which is locked by slicer itself, this procedure liberates temporarily the console input and sets a virtual input with no owner (no GIL). If you try to use slicer console while it is calculating, then SlycerPython will not read from it (not that bad). If you try to execute the pool again while is being executed then the system will see colliding threads and will kill slicer.

8

Thank you very much. I just tried the new code of onListenButton function.
I got the new errors as the following"

Received signal 11 SEGV_MAPERR 0000000000a0
#0 0x7f7f1721558f
#1 0x7f7f15c4785d
#2 0x7f7f17215a9e
#3 0x7f7f00d6f890
#4 0x7f7f0c259d70 tupledealloc
#5 0x7f7f0c2473c6 _PyCFunction_FastCallDict
#6 0x7f7f0c1eaa9e _PyObject_FastCallDict
#7 0x7f7f0c21045b PyFile_WriteObject
#8 0x7f7f0c21053b PyFile_WriteString
#9 0x7f7f0c326e7a PyTraceBack_Print
#10 0x7f7f0c31abcf print_exception_recursive
#11 0x7f7f0c31bb48 PyErr_Display
#12 0x7f7f0c3237c8 sys_excepthook
#13 0x7f7f0c2473ad _PyCFunction_FastCallDict
#14 0x7f7f0c1eaa9e _PyObject_FastCallDict
#15 0x7f7f0c31bcdc PyErr_PrintEx
#16 0x7f7f10b19a54 PythonQt::handleError()
#17 0x7f7f10bc0de1 PythonQtSignalTarget::call()
#18 0x7f7f10bc0e06 PythonQtSignalTarget::call()
#19 0x7f7f10bc1625 PythonQtSignalReceiver::qt_metacall()
#20 0x7f7f0733c504 QMetaObject::activate()
#21 0x7f7f0824e1f2 QAbstractButton::clicked()
#22 0x7f7f0824e3f4 QAbstractButtonPrivate::emitClicked()
#23 0x7f7f0824ff8e QAbstractButtonPrivate::click()
#24 0x7f7f082500e5 QAbstractButton::mouseReleaseEvent()
#25 0x7f7f10e73a05 PythonQtShell_QPushButton::mouseReleaseEvent()
#26 0x7f7f081994c8 QWidget::event()
#27 0x7f7f10e726d7 PythonQtShell_QPushButton::event()
#28 0x7f7f0815cdac QApplicationPrivate::notify_helper()
#29 0x7f7f08164833 QApplication::notify()
#30 0x7f7f1eee6536 qSlicerApplication::notify()
#31 0x7f7f073114e8 QCoreApplication::notifyInternal2()
#32 0x7f7f0816348f QApplicationPrivate::sendMouseEvent()
#33 0x7f7f081b301d QWidgetWindow::handleMouseEvent()
#34 0x7f7f081b5913 QWidgetWindow::event()
#35 0x7f7f0815cdac QApplicationPrivate::notify_helper()
#36 0x7f7f08163e57 QApplication::notify()
#37 0x7f7f1eee6536 qSlicerApplication::notify()
#38 0x7f7f073114e8 QCoreApplication::notifyInternal2()
#39 0x7f7f0793fe37 QGuiApplicationPrivate::processMouseEvent()
#40 0x7f7f07941d35 QGuiApplicationPrivate::processWindowSystemEvent()
#41 0x7f7f0791bb7b QWindowSystemInterface::sendWindowSystemEvents()
#42 0x7f7ee9fceb8b QPAEventDispatcherGlib::processEvents()
#43 0x7f7f0730fe4a QEventLoop::exec()
#44 0x7f7f07318850 QCoreApplication::exec()
#45 0x7f7f1d90d2aa qSlicerCoreApplication::exec()
#46 0x55bfa73abc87 main
#47 0x7f7ef49eeb97 __libc_start_main

9

instead of using terminate and join, try using quit and wait respectively as they are the recommended with QThreadPool.

Anyways, if you want to try the multiprocessing module

from multiprocessing import Pool

def worker_wrapper(args):
    worker = Worker(*args)
    return worker.run()

original_stdin = sys.stdin # Unlock SlicerPython GIL
sys.stdin = open(os.devnull)
args = tuple('your arguments here, can be empty so you can omit args')
p = Pool(self.CpuCores)
try: # Start producing results
    iresults = p.imap(worker_wrapper, args) # This is an iterator pointer
    p.close()
    for i, result in enumerate(iresults):
        print("performed {} threads".format(i)) # Here you can follow your threads progress ;)

except Exception as e: # Is something wrong happens, force to terminate the pool
    canceled = True # Necesary, no "return False" allowed here
    p.terminate()
    logging.error(e)

finally:
    p.join()
    sys.stdin.close()
    sys.stdin = original_stdin
10

This sounds very fragile. Python CLI or C++ module options are much more reliable. Currently, Slicer’s scheduler runs Python CLIs one by one, but it would be possible to change this so that tasks that do not depend on completion of other tasks could all run in parallel.

11

Thank you very much! After I try it and get back to you.

12

We previously had a discussion (Running multiple CLIs at once) about this, but didn’t take any action yet.

13

Update:

I have sub-process working with ProcessPoolExecutor.

14

May you please add your final solution?

15

Sure.

import vtk, qt, ctk, slicer
from slicer.ScriptedLoadableModule import *
from concurrent.futures import ProcessPoolExecutor

def task():
   print("sub process start")
   time.sleep(5)
   print("sub process done")

def task_done(future):
    if future.cancelled():
        print("task is cancelled!")
    elif future.done():
        error = future.exception()
        if error:
            print("task error")
        else:
            print(" task is done")

class TestWidget(ScriptedLoadableModuleWidget):
    def __init__(self):
        self.process_executor = ProcessPoolExecutor(max_workers=3)

    def setup(self):
         ScriptedLoadableModuleWidget.setup(self)
         parametersCollapsibleButton = ctk.ctkCollapsibleButton()
         parametersCollapsibleButton.text = "Parameters"
         self.layout.addWidget(parametersCollapsibleButton)

         parametersFormLayout = qt.QFormLayout(parametersCollapsibleButton)


         self.listenButton = qt.QPushButton("Test")
         self.listenButton.toolTip = "Test"
         self.listenButton.enabled = True
         self.listenButton.connect('clicked(bool)', self.onListenButton)
         parametersFormLayout.addRow(self.listenButton)

    def onListenButton(self):
        original_stdin = sys.stdin
        sys.stdin = open(os.devnull)
        try:
            ex = self.process_executor.submit(task)
            ex.add_done_callback(task_done)
        finally:
            sys.stdin.close()
            sys.stdin = original_stdin
3 Likes
16

Remember to add a mutex to prevent duplicated calls to onListenButton, if you call it again while it is being executed it will make Slicer to crash.

17

Thanks for the suggestion.

I found this poster here talking about using multiprocessing.Manager. Due to the stdin problem, it does not working in Slicer.
I simply set a flag before call the subprocess and reset it in callback function.

1 Like