Slicer and jupyter kernel dies repeatedly, problem with connection.

When I install SlicerJupyter and run it, it tries to connect and the dies. The messsage ist: “The kernel appears to have died. It will restart automatically.” It then restarts and dies again.
This happens both when I’m using standalone Slicer/Jupyter or Docker image.
I feel hopeless.

The other message is: “A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.”

I’m using newst Slicer app and Ubuntu 18

What I found to be strange is that Slicer app repeatedly starts and closes. After I close it, and shutdown jupyter I still have like 4 Slicer processes.

It’s strange to me that the same error is when I’m using docker and standalone app.

The docker image works fine - you can try it in binder: https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master

Do you see any errors in Slicer application log files? (in Slicer, menu: Help / Report a bug -> check for logs of other sessions)

So below is the application log. I noticed that there’s a message ‘device not open’:

  • QIODevice::read (QFile, “/home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/share/Slicer-4.11/qt-loadable-modules/JupyterKernel/Slicer-4.11/kernel-template.json”): device not open

apart from that, standard message about abnormal exit:

[CRITICAL][FD] 09.12.2020 19:33:29 [] (unknown:0) - error: [/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/bin/SlicerApp-real] exit abnormally - Report the problem.
[CRITICAL][FD] 09.12.2020 19:33:30 [] (unknown:0) - [I 19:33:30.948 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports

more logs below:

[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Session start time …: 2020-12-09 19:27:11
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Slicer version …: 4.11.20200930 (revision 29402 / 002be18) linux-amd64 - installed release
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Operating system …: Linux / 4.15.0-126-generic / #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 - 64-bit
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Memory …: 7861 MB physical, 10239 MB virtual
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - CPU …: GenuineIntel Intel® Core™ i3-3217U CPU @ 1.80GHz, 2 cores, 4 logical processors
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - VTK configuration …: OpenGL2 rendering, Sequential threading
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Qt configuration …: version 5.15.1, with SSL, requested OpenGL 3.2 (core profile)
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Developer mode enabled …: no
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Prefer executable CLI …: yes
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Application path …: /home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/bin
[DEBUG][Qt] 09.12.2020 19:27:11 [] (unknown:0) - Additional module paths …: /home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/lib/Slicer-4.11/qt-loadable-modules, /home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/lib/Slicer-4.11/qt-scripted-modules
[DEBUG][Python] 09.12.2020 19:27:15 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/pydicom/datadict.py:432) - Reversing DICOM dictionary so can look up tag from a keyword…
[DEBUG][Python] 09.12.2020 19:27:24 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: Annotations
[DEBUG][Python] 09.12.2020 19:27:25 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: SegmentEditor
[DEBUG][Python] 09.12.2020 19:27:25 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: SegmentStatistics
[DEBUG][Qt] 09.12.2020 19:27:25 [] (unknown:0) - Switch to module: “Welcome”
[DEBUG][Qt] 09.12.2020 19:27:37 [] (unknown:0) - Switch to module: “JupyterKernel”
[WARNING][Qt] 09.12.2020 19:27:37 [] (unknown:0) - QIODevice::read (QFile, “/home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/share/Slicer-4.11/qt-loadable-modules/JupyterKernel/Slicer-4.11/kernel-template.json”): device not open
[DEBUG][Python] 09.12.2020 19:28:32 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/jupyter_client/kernelspec.py:333) - Installing kernelspec in /home/mikolaj/.local/share/jupyter/kernels/slicer-4.11
[INFO][Python] 09.12.2020 19:28:32 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/jupyter_client/kernelspec.py:342) - Removing existing kernelspec in /home/mikolaj/.local/share/jupyter/kernels/slicer-4.11
[INFO][Python] 09.12.2020 19:28:32 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/jupyter_client/kernelspec.py:346) - Installed kernelspec slicer-4.11 in /home/mikolaj/.local/share/jupyter/kernels/slicer-4.11
[WARNING][Qt] 09.12.2020 19:28:32 [] (unknown:0) - QIODevice::read (QFile, “/home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/share/Slicer-4.11/qt-loadable-modules/JupyterKernel/Slicer-4.11/kernel-template.json”): device not open
[INFO][Stream] 09.12.2020 19:28:32 [] (unknown:0) - Removing existing kernelspec in /home/mikolaj/.local/share/jupyter/kernels/slicer-4.11
[INFO][Stream] 09.12.2020 19:28:32 [] (unknown:0) - Installed kernelspec slicer-4.11 in /home/mikolaj/.local/share/jupyter/kernels/slicer-4.11
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [I 19:28:34.300 NotebookApp] Serving notebooks from local directory: /home/mikolaj/codete/git_repos/medtransfer-ml/slicerdemo
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [I 19:28:34.300 NotebookApp] Jupyter Notebook 6.1.5 is running at:
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [I 19:28:34.300 NotebookApp] http://localhost:8888/?token=16e888901eaa0ae4140cb0e7cb1e980f6441887e6a58b532
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [I 19:28:34.301 NotebookApp] or http://127.0.0.1:8888/?token=16e888901eaa0ae4140cb0e7cb1e980f6441887e6a58b532
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [I 19:28:34.301 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - [C 19:28:34.333 NotebookApp]
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) -
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - To access the notebook, open this file in a browser:
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - file:///home/mikolaj/.local/share/jupyter/runtime/nbserver-13993-open.html
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - Or copy and paste one of these URLs:
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - http://localhost:8888/?token=16e888901eaa0ae4140cb0e7cb1e980f6441887e6a58b532
[CRITICAL][FD] 09.12.2020 19:28:34 [] (unknown:0) - or http://127.0.0.1:8888/?token=16e888901eaa0ae4140cb0e7cb1e980f6441887e6a58b532
[CRITICAL][FD] 09.12.2020 19:29:27 [] (unknown:0) - [I 19:29:27.199 NotebookApp] Kernel started: cca1c750-cb5e-49b0-8300-9b44431ee743, name: slicer-4.11

Can you check if you can find logs of a Slicer instance that the notebook server started?

Can you try if you start Slicer and launch the kernel manually? (see instructions here)

Ad) Can you check if you can find logs of a Slicer instance that the notebook server started?
So I went to /tmp/Slicer-mikolaj (I believe this is where logs are stored) and after few minutes of running slicer with jupyter server (started from slicer standalone app with extension) I have 12 separate files. I think this is because something is spawning slicer app processes. I see a Slicer window popup for a second and then it shuts down. after about 10 seconds it happens again - Slicer window pops up and closes. This happens ad infinitum until I go to htop and kill every process that has ‘slicer’ in it (there are usually like maybe 6 or more simultanious processes).

Those are the files:
Slicer_29402_20201209_195740_529.log , Slicer_29402_20201209_200027_758.log, Slicer_29402_20201209_200050_524.log etc

And the messages in those logs look like this:
[DEBUG][Python] 09.12.2020 20:01:17 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/pydicom/datadict.py:432) - Reversing DICOM dictionary so can look up tag from a keyword…
[DEBUG][Python] 09.12.2020 20:01:26 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: Annotations
[DEBUG][Python] 09.12.2020 20:01:27 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: SegmentEditor
[DEBUG][Python] 09.12.2020 20:01:27 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Slicer-4.11/qt-scripted-modules/SubjectHierarchyPlugins/AbstractScriptedSubjectHierarchyPlugin.py:36) - Scripted subject hierarchy plugin registered: SegmentStatistics
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Session start time …: 2020-12-09 20:01:14
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Slicer version …: 4.11.20200930 (revision 29402 / 002be18) linux-amd64 - installed release
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Operating system …: Linux / 4.15.0-126-generic / #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 - 64-bit
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Memory …: 7861 MB physical, 10239 MB virtual
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - CPU …: GenuineIntel Intel® Core™ i3-3217U CPU @ 1.80GHz, 2 cores, 4 logical processors
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - VTK configuration …: OpenGL2 rendering, Sequential threading
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Qt configuration …: version 5.15.1, with SSL, requested OpenGL 3.2 (core profile)
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Developer mode enabled …: no
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Prefer executable CLI …: yes
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Application path …: /home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/bin
[DEBUG][Qt] 09.12.2020 20:01:14 [] (unknown:0) - Additional module paths …: /home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/lib/Slicer-4.11/qt-loadable-modules, /home/mikolaj/.config/NA-MIC/Extensions-29402/SlicerJupyter/lib/Slicer-4.11/qt-scripted-modules
[DEBUG][Qt] 09.12.2020 20:01:27 [] (unknown:0) - Switch to module: “Welcome”

Or like this:
[DEBUG][Python] 09.12.2020 20:02:28 [Python] (/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/lib/Python/lib/python3.6/site-packages/pydicom/datadict.py:432) - Reversing DICOM dictionary so can look up tag from a keyword…

And I have 5 log files with the shorter version and 6 log files with longer version

Ad) Can you try if you start Slicer and launch the kernel manually?

I don’t understand this:
Build the extension against the newly built Slicer with Qt5 and VTK9 enabled.

is this step necessary and if so how can I build it?

No need to build anything, just launch the kernel manually (https://github.com/Slicer/SlicerJupyter#launch-a-kernel-manually).

Launching of Slicer processes is expected, it means that the kernel is successfully registered to the jupyter server and the jupyter server attempts to start the kernel (=Slicer application).

Have you launched the server? You should only see Slicer instances popping up after you open a notebook with Slicer kernel. If you see Slicer instances started before that then it means that you have Jupyter server(s) running in the background. You can find and kill all Jupyter server instances or just reboot your computer to get rid of them.

How things should work:

  • Start Slicer, go to JupyterKernel module, click on “Start Jupyter server” => after a minute or so, your browser is opened (http://localhost:8888/tree) and you see the Jupyter server web interface
  • Create a notebook, choose Slicer 4.11 as kernel (or 4.13 if you use latest preview release) => a new notebook is created, a Slicer instance is launched

The Jupyter server launches Slicer with a command like this:

"C:\Users\andra\AppData\Local\NA-MIC\Slicer 4.13.0-2020-11-25\bin\SlicerApp-real.exe"  --no-splash --python-code "connection_file=r'C:\Users\andra\AppData\Roaming\jupyter\runtime\kernel-b1663e74-4f82-455e-9fa4-7abd9db6c811.json'; print('Jupyter connection file: ['+connection_file+']'); slicer.modules.jupyterkernel.startKernel(connection_file);slicer.util.mainWindow().showMinimized()"

Therefore, in the logs of the Slicer instance that Jupyter server started you should see something like this:

[DEBUG][Qt] 09.12.2020 14:52:12 [] (unknown:0) - Switch to module:  "Welcome"
[INFO][Stream] 09.12.2020 14:52:13 [] (unknown:0) - Loading Slicer RC file [C:/Users/andra/.slicerrc.py]
[INFO][Stream] 09.12.2020 14:52:13 [] (unknown:0) - Jupyter connection file: [C:\Users\andra\AppData\Roaming\jupyter\runtime\kernel-b1663e74-4f82-455e-9fa4-7abd9db6c811.json]
[DEBUG][Qt] 09.12.2020 14:52:14 [] (unknown:0) - Starting Jupyter kernel server

How things should work:

  • Start Slicer, go to JupyterKernel module, click on “Start Jupyter server” => after a minute or so, your browser is opened ( http://localhost:8888/tree ) and you see the Jupyter server web interface
  • Create a notebook, choose Slicer 4.11 as kernel (or 4.13 if you use latest preview release) => a new notebook is created, a Slicer instance is launched

This is exactly what I did. However, after I create a notebook and choose slicer 4.11 as kernel I get message that the kernel died and it will restart. A slicer instance window pops up and briefly closes. It then goes on and on like this (restarting kernel, problem with connection to kernel, slicer window up and down, restarting).

What I think it should do, is that jupyter notebook should stay connected to the Slicer kernel and allow me use it. Instead it looses connection and restarts.

I can’t interact with instances of Slicer that pop up, because after a second they disappear. I can only interact with instance of Slicer that I run with ./Slicer

The same behaviour is also inside docker container.

I wonder what may cause this.

OK, this is pretty good. The next step is to see why startup fails:
A. because Slicer crashes at startup, when it starts the kernel (e.g., due to conflicts with some system libraries)
B. because the server cannot connect to the kernel (e.g., due to network access issues).

If you launch Slicer and launch a kernel manually and it crashes then we know it is A, otherwise it is B.

It crashed when launched manually.

Great, this information narrows down what the issue can be. Can you get a stack trace of the crash?

I don’t know where it might be, the console where I ran ./Slicer has this:

Switch to module: “Welcome”
Python console user input: connection_file=r’/home/mikolaj/.local/share/jupyter/runtime/kernel-7a4a79ef-474e-4810-8ec7-dd86d6eaaf38.json’
Python console user input: slicer.modules.jupyterkernel.startKernel(connection_file)
error: [/home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/bin/SlicerApp-real] exit abnormally - Report the problem.

in /tmp/Slicer-mikolaj logs there is this (only pasting last two lines):

[DEBUG][Qt] 09.12.2020 21:53:34 [] (unknown:0) - Switch to module: “Welcome”
[DEBUG][Qt] 09.12.2020 22:08:21 [] (unknown:0) - Python console user input: connection_file=r’/home/mikolaj/.local/share/jupyter/runtime/kernel-7a4a79ef-474e-4810-8ec7-dd86d6eaaf38.json’

is there any other place where I can find trace of the crash? like, linux logs?

Yes, you can use standard linux tools to get stack trace of the crash. I don’t use linux much, so I don’t know the details. Maybe @RafaelPalomar can help.

Hi @RafaelPalomar could you please help me with Slicer and Jupyter bug?

I checked logs in /var/log/kern.log and I saw this error:

Dec 9 21:44:56 machine kernel: [181247.526073] traps: SlicerApp-real[24615] trap invalid opcode ip:7f2eb59e4c6c sp:7ffeb7a186f0 error:0 in libxeus.so.1[7f2eb5982000+7b000]

Invalid opcode means the CPU encountered an invalid instruction. It may be due to data corruption/incorrect loading, but there is a chance that your CPU actually does not recognize the instruction.

Do you run on a physical computer or in a virtualized environment? What is your CPU model?

according to Slicer logs it’s i3-3217U CPU @ 1.80GHz, 2 cores, 4 logical processors:

[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Session start time …: 2020-12-09 21:53:16
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Slicer version …: 4.11.20200930 (revision 29402 / 002be18) linux-amd64 - installed release
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Operating system …: Linux / 4.15.0-126-generic / #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 - 64-bit
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Memory …: 7861 MB physical, 10239 MB virtual
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - CPU …: GenuineIntel Intel® Core™ i3-3217U CPU @ 1.80GHz, 2 cores, 4 logical processors
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - VTK configuration …: OpenGL2 rendering, Sequential threading
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Qt configuration …: version 5.15.1, with SSL, requested OpenGL 3.2 (core profile)
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Developer mode enabled …: no
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Prefer executable CLI …: yes
[DEBUG][Qt] 09.12.2020 21:53:16 [] (unknown:0) - Application path …: /home/mikolaj/Downloaded_Programs/Slicer-4.11.20200930-linux-amd64/bin