What kind of images are you working with? If they are large, you might want to crop them first (search discourse for CropVolume).
Also does this happen more if you have used Slicer for a while? If so maybe there’s a memory leak we need to investigate. If you can provide specific details to replicate the issue (what data, what effects, etc) it would really help.
Otherwise maybe the easiest is just to get an extra big disk.
While the suggested answer may solve this particular case, I’m still interested about the general principle of stopping a process in 3D slicer.
If I have a custom pushbutton that calls a python function, is there proper way to stop that call mid-process, similar to a ctrl+c in a regular python interpreter?
To the best of my knowledge, there is no general purpose method of halting an in-progress operation in slicer. Some filters can be canceled and things like that, but there is no “global ctrl-C”, I think.
There are a number of tricks that can reduce your memory footprint, including cropping or resampling your dataset, and you can even set the number of undo states to zero with a little python.
But looking at the force quit dialog - Slicer is using 133GB? This is massive. I have to echo what @pieper is saying - what type of data are you looking at, and do you need this resolution?
To make this work you need to be able to integrate your custom code into the Slicer event loop. You can put up a progress dialog and have the cancel signal set a variable that you check periodically during processing. By calling slicer.app.processEvents periodically the variable will get set and you can break out of your processing.
There are several examples of this pattern in the source code, like this one: