# Subprocess call in Python interpreter results in memory corruption

**URL:** <https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919>\
**Category:** Support\
**Tags:** python\
**Created:** [August 22, 2017, 1:03pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919 "2017-08-22T13:03:10Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zoe\_Goey](https://avatars.discourse-cdn.com/v4/letter/z/898d66/32.png) [@Zoe\_Goey](https://discourse.slicer.org/u/Zoe_Goey)\
**Post date:** [August 22, 2017, 1:03pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/1 "2017-08-22T13:03:11Z")

</div>

I have a C++ program that I can call from my system-installed Python using the subprocess module without causing any problems.

However, if I do so from within Slicer it results in a memory corruption error. The problem occurs when the program reaches this line of code:

```auto
itk::Statistics::MersenneTwisterRandomVariateGenerator::Pointer rndgen = \
  itk::Statistics::MersenneTwisterRandomVariateGenerator::New();

```

In fact, the problem can be reproduced by calling a program that consists of that one, single line.

Can anyone explain why this is happening and how it can be circumvented?

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 22, 2017, 1:05pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/2 "2017-08-22T13:05:16Z")

</div>

> [@Zoe\_Goey](#):
>
> Can anyone explain why this is happening and how it can be circumvented?

Hi @Zoe_Goey,

Could you that the program you are calling is linked against its own version of ITK ?

---

<div class="post-metadata">

**Author:** ![Zoe\_Goey](https://avatars.discourse-cdn.com/v4/letter/z/898d66/32.png) [@Zoe\_Goey](https://discourse.slicer.org/u/Zoe_Goey)\
**Post date:** [August 22, 2017, 3:45pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/4 "2017-08-22T15:45:08Z")

</div>

Yes, the program is linked against its own version of ITK.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 22, 2017, 6:38pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/5 "2017-08-22T18:38:57Z")

</div>

Great. If you can wait few days, I have to implement a solution to allow calling a subprocess with a “clean” environment (aka without Slicer paths) for a different project.

Once done, you should have an easy way to achieve this.

---

<div class="post-metadata">

**Author:** ![Zoe\_Goey](https://avatars.discourse-cdn.com/v4/letter/z/898d66/32.png) [@Zoe\_Goey](https://discourse.slicer.org/u/Zoe_Goey)\
**Post date:** [August 23, 2017, 4:47pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/6 "2017-08-23T16:47:17Z")

</div>

OK, thanks. It would be great to get this working without replacing the random number generator. Please keep me posted about your solution.

---

<div class="post-metadata">

**Author:** ![Zoe\_Goey](https://avatars.discourse-cdn.com/v4/letter/z/898d66/32.png) [@Zoe\_Goey](https://discourse.slicer.org/u/Zoe_Goey)\
**Post date:** [September 3, 2017, 11:05pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/7 "2017-09-03T23:05:58Z")

</div>

Any news on this yet? I am now using docker to achieve what I need. I would still be interested in a more elegant solution, though.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 3, 2017, 11:36pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/8 "2017-09-03T23:36:43Z")

</div>

Probably you can simply launch your process using custom environment. We do this when we launch Elastix. Example:

1. [Create custom environment](https://github.com/lassoan/SlicerElastix/blob/777c5259eaa5eb45d46b97ce83c67f865096ec55/Elastix/Elastix.py#L423-L434)
2. [Launch process using subprocess.Popen](https://github.com/lassoan/SlicerElastix/blob/777c5259eaa5eb45d46b97ce83c67f865096ec55/Elastix/Elastix.py#L460-L466)

Note that in case of Elastix, we added more directories to the path/LD path. In your case you probably want to either add your external program’s paths to the beginning of relevant environment variables; or remove all Slicer-specific paths from environment variables.

---

<div class="post-metadata">

**Author:** ![Zoe\_Goey](https://avatars.discourse-cdn.com/v4/letter/z/898d66/32.png) [@Zoe\_Goey](https://discourse.slicer.org/u/Zoe_Goey)\
**Post date:** [September 4, 2017, 9:39am UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/10 "2017-09-04T09:39:25Z")

</div>

Yes, you are right. I did not know that subprocess offered options to configure the environment. Thank you!

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 4, 2017, 9:08pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/11 "2017-09-04T21:08:59Z")

</div>

Then, as soon as we are done implementing a more general solution, you would be able to do:

```python
from subprocess import check_output
check_output(
    ["/path/to/program", "arg1", ...],
    env=slicer.util.startupEnvironment())

```

similarly, it will be possible to start CLI excluding the slicer environment. (edit: This last part is **NOT** yet implemented.)

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 4, 2017, 10:34pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/12 "2017-09-04T22:34:11Z")

</div>

Thanks, this will be useful. Please consider giving a more descriptive name and only use positive statements in names (try to describe what it _is_, instead of what it _is not_). For example, `systemEnv()`, `defaultEnv()`, `externalEnv()`, or `startupEnv()` names would be better.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 5, 2017, 4:28am UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/13 "2017-09-05T04:28:46Z")

</div>

Good point. Thanks for the suggestions 👍 . I will ~~most likely~~ go with ~~`systemEnv`~~ `startupEnvironment`.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 5, 2017, 3:37pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/14 "2017-09-05T15:37:37Z")

</div>

Here is a work-in-progress topic

> <https://github.com/Slicer/Slicer/issues/787>
>
> This issue was created automatically from an original Mantis Issue. Further discussion may take place here.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 5, 2017, 8:37pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/15 "2017-09-05T20:37:21Z")

</div>

With the upcoming feature, it will be possible to easily invoke processes by excluding the Slicer environment.:

- from c++ (using `app->startupEnvironment()`), or;
- from python (using `slicer.util.startupEnvironment()`

Here are two examples:

Without using the `startup` environment, this first example fails (as expected):

```auto
>>> from subprocess import check_output
>>> check_call(["/usr/bin/python3", "-c", "print('hola')"])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/jcfr/Projects/Slicer-2-build/python-install/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/python3', '-c', "print('hola')"]' returned non-zero exit status -6

```

Now, using the sanitized environment, this one succeeds:

```auto
>>> from subprocess import check_output
>>> check_output(
  ["/usr/bin/python3", "-c", "print('hola')"], 
  env=slicer.util.startupEnvironment())
'hola\n'

```

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 5, 2017, 10:20pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/16 "2017-09-05T22:20:35Z")

</div>

The topic is now read for final review

```auto
601: Test timeout computed to be: 1800
601: Number of registered modules: 1 
601: Number of instantiated modules: 1 
601: Number of loaded modules: 1 
601: -------------------------------------------
601: path: ['/home/jcfr/Projects/Slicer-2-build/Slicer-build/Applications/SlicerApp/Testing/Python', '/home/jcfr/Projects/Slicer-2/Base/Python/slicer/tests']
601: testname: test_slicer_environment
601: -------------------------------------------
601: test_slicer_app_environment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: test_slicer_app_startupEnvironment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: test_slicer_util_startupEnvironment (test_slicer_environment.SlicerEnvironmentTests) ... ok
601: 
601: ----------------------------------------------------------------------
601: Ran 3 tests in 0.002s
601: 
601: OK
601: vtkDebugLeaks has found no leaks.
1/1 Test #601: py_nomainwindow_test_slicer_environment ... Passed 1.23 sec

```

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 6, 2017, 3:23am UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/17 "2017-09-06T03:23:01Z")

</div>

Starting with [r26351](http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=26351), a new public API allowing to get the startup environment is available.

- In c++: `qSlicerCoreApplication::startupEnvironment()`
- In python: `slicer.util.startupEnvironment()`

We are still [discussing](https://github.com/Slicer/Slicer/pull/787#issuecomment-327361381) some of the internals but that will not affect the public API.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [September 7, 2017, 8:14pm UTC](https://discourse.slicer.org/t/subprocess-call-in-python-interpreter-results-in-memory-corruption/919/18 "2017-09-07T20:14:45Z")

</div>

@dzenanz You can now start a process with the original environment using something like this:

```auto
from subprocess import check_output
check_output(
  ["/usr/bin/python3", "-c", "print('hola')"], 
  env=slicer.util.startupEnvironment())
'hola\n'

```