Subprocess call in Python interpreter results in memory corruption

Support
1

I have a C++ program that I can call from my system-installed Python using the subprocess module without causing any problems.

However, if I do so from within Slicer it results in a memory corruption error. The problem occurs when the program reaches this line of code:

itk::Statistics::MersenneTwisterRandomVariateGenerator::Pointer rndgen = \
  itk::Statistics::MersenneTwisterRandomVariateGenerator::New();

In fact, the problem can be reproduced by calling a program that consists of that one, single line.

Can anyone explain why this is happening and how it can be circumvented?

2

Hi @Zoe_Goey,

Could you that the program you are calling is linked against its own version of ITK ?

4

Yes, the program is linked against its own version of ITK.

5

Great. If you can wait few days, I have to implement a solution to allow calling a subprocess with a ā€œcleanā€ environment (aka without Slicer paths) for a different project.

Once done, you should have an easy way to achieve this.

6

OK, thanks. It would be great to get this working without replacing the random number generator. Please keep me posted about your solution.

7

Any news on this yet? I am now using docker to achieve what I need. I would still be interested in a more elegant solution, though.

8

Probably you can simply launch your process using custom environment. We do this when we launch Elastix. Example:

  1. Create custom environment
  2. Launch process using subprocess.Popen

Note that in case of Elastix, we added more directories to the path/LD path. In your case you probably want to either add your external programā€™s paths to the beginning of relevant environment variables; or remove all Slicer-specific paths from environment variables.

10

Yes, you are right. I did not know that subprocess offered options to configure the environment. Thank you!

11

Then, as soon as we are done implementing a more general solution, you would be able to do:

from  subprocess import check_output
check_output(
    ["/path/to/program", "arg1", ...],
    env=slicer.util.startupEnvironment())

similarly, it will be possible to start CLI excluding the slicer environment. (edit: This last part is NOT yet implemented.)

2 Likes
12

Thanks, this will be useful. Please consider giving a more descriptive name and only use positive statements in names (try to describe what it is, instead of what it is not). For example, systemEnv(), defaultEnv(), externalEnv(), or startupEnv() names would be better.

13

Good point. Thanks for the suggestions :+1: . I will most likely go with systemEnv startupEnvironment.

14

Here is a work-in-progress topic

1 Like
15

With the upcoming feature, it will be possible to easily invoke processes by excluding the Slicer environment.:

  • from c++ (using app->startupEnvironment()), or;
  • from python (using slicer.util.startupEnvironment()

Here are two examples:

Without using the startup environment, this first example fails (as expected):

>>> from subprocess import check_output
>>> check_call(["/usr/bin/python3", "-c", "print('hola')"])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/jcfr/Projects/Slicer-2-build/python-install/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/python3', '-c', "print('hola')"]' returned non-zero exit status -6

Now, using the sanitized environment, this one succeeds:

>>> from subprocess import check_output
>>> check_output(
  ["/usr/bin/python3", "-c", "print('hola')"], 
  env=slicer.util.startupEnvironment())
'hola\n'
16

The topic is now read for final review

601: Test timeout computed to be: 1800
601: Number of registered modules: 1 
601: Number of instantiated modules: 1 
601: Number of loaded modules: 1 
601: -------------------------------------------
601: path: ['/home/jcfr/Projects/Slicer-2-build/Slicer-build/Applications/SlicerApp/Testing/Python', '/home/jcfr/Projects/Slicer-2/Base/Python/slicer/tests']
601: testname: test_slicer_environment
601: -------------------------------------------
601: test_slicer_app_environment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: test_slicer_app_startupEnvironment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: test_slicer_util_startupEnvironment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: 
601: ----------------------------------------------------------------------
601: Ran 3 tests in 0.002s
601: 
601: OK
601: vtkDebugLeaks has found no leaks.
1/1 Test #601: py_nomainwindow_test_slicer_environment ...   Passed    1.23 sec
17

Starting with r26351, a new public API allowing to get the startup environment is available.

  • In c++: qSlicerCoreApplication::startupEnvironment()
  • In python: slicer.util.startupEnvironment()

We are still discussing some of the internals but that will not affect the public API.

18

@dzenanz You can now start a process with the original environment using something like this:

from subprocess import check_output
check_output(
  ["/usr/bin/python3", "-c", "print('hola')"], 
  env=slicer.util.startupEnvironment())
'hola\n'
3 Likes