I have been building a custom slicer extension that allows me to do localised modifications in a CT scan using some AI models running in a Flask server that I access with requests.
The extension I implemented follows a structure similar to this MedSAM extension here.
My extension works well most of the time, but, at times, for reasons I have not been yet able to determine, I am encountering that Slicer suddenly crashes.
Things I have checked:
CPU and RAM usage before crash is not larger than usual.
Following this other issue documented in this community I have increased the cache size to 10000MBs. This seemed to decrease the frequency of the crashes but they still happen occasionally.
Crashes can happen right before/during I write some DICOM data to qt.QTemporaryDir() for later sending it to my server via requests. Although, they can also happen right after incorporating the predictions of the model into the scene.
Checking the logs after the crash in a new Slicer session are not informative. There aren’t any specific errors, just simply blank after my last debugging print.
The crashes don’t happen always but I need to understand how to avoid them completely. Do you have any ideas?
The easiest would be to get a stack trace to see where exactly the crash occurs. It is even better if you run the code step by step in a debugger, then you can see exactly what leads to the issue. Also, make sure you use the matest main version of the source code, as we keep improving the code.
If you are interested in interactive AI segmentation then I would recommend to try NNInteractive extension. Based on my experience, it is way better than any SAM variants: it is natively 3D, fast, and accurate, and you can use it to edit or refine existing 3D segmentations. I would say NNInteractive is the first practically useful interactive medical AI segmentation tool.
Thanks for the fast response! I have been investigating possible causes and I wonder if my sporadic crashes are due to the fact that I am using the threading python lib in my extension’s logic module.
Considering I am running a python function in my ScriptedLoadableModuleLogic class which performs multiple different API calls to a server. For one of these requests due it taking beyond a minute to complete I am using threading lib to run it in a different thread and still be able to update a progress bar (slicer.util.createProgressDialog) based on the processing time that has passed.
I have since realised from the SlicerNNUnet extension one can use qt.QProcess() to run a CLI program in a different process. Do you think running all my API calls in a different process could help minimise the crashes? Finally, is there any added benefit to using qt.QProcess() for that rather than python’s multiprocessing?
Yes, absolutely. Threading is complex and it is easy to cause crashes. For example, if you make any modifications to any nodes in a worker thread then the application will crash. You can make modifications to a copy of the data on a worker thread then signal to the main thread that new data is available, for example using a Python queue. The main thread can regularly check the queue using a QTimer and if it finds that there is updated data, it can update nodes in the scene.
Using a subprocess for running tasks in parallel is a good way to isolate the background operations from the main thread, so crashes cannot occur. This is good for any tasks that take longer than a couple of seconds (because then the overhead of starting a new process is negligible). This is also good if the processing is resource-intensive (e.g., uses lots of memory) or the implementation is not robust (it may crash) because in case of a crash, the main application does not crash. You can also terminate the subprocess anytime (while you can just nicely ask a thread to terminate, but it may or may not actually stop).
You can find examples for both running a background process and using background threads to not block the GUI in MONAIAuto3DSeg extension.
Following the advice I have restructured my extension to use qt.Process to run the longest API request (few minutes) in a subprocess to not clog Slicer and hopefully avoid the crashes I am having entirely.
After doing so I still get these crashes. My Mac’s problem report for slicer shows the following massive log which is not much different than what I got before running the long API call in the subprocess:
I am not really sure what to make of this. I do still have some API calls running in the main process, however they take only a few seconds (< 5 sec) to run to completion. Do you think running a subprocess for these calls might help or is my problem likely elsewhere?
I am thinking it might be due to something else considering Slicer at times also crashes upon loading regular DICOM data into the scene and while no functionality from my extension has been used. Could it be due to some issue while initialising my extension? Perhaps something to do with the python libraries it depends on? I am open to ideas.