I'm getting not a useful error when I try to start Slicer Jupyter kernel, how to debug?

[myousefi2016 (Mehrdad Yousefi)]

I’m using Slicer 4.10 on our cluster that has Centos8. It’s a headless server and I’m able successfully to run Slicer and even its graphical interface by using Xvfb. The problem is that I want to use Slicer in Jupyter notebook. I installed the extension and installed the kernel. This is my kernel.json file:

{
    "display_name": "Slicer 4.10",
    "language" : "python",
    "argv": [
	"xvfb-run",
	"-a",
        "/home/yousefi/Slicer-4.10.2-linux-amd64/bin/../Slicer",
        "--no-splash",
        "--python-code",
        "connection_file=r'{connection_file}'; print('Jupyter connection file: ['+connection_file+']'); slicer.modules.jupyterkernel.startKernel(connection_file);slicer.util.mainWindow().showMinimized()"
    ]
}

When I’m trying to check the kernel by using:

jupyter-kernel --kernel=slicer-4.10

I’m seeing this error which is not helpful at all:

[KernelApp] Starting kernel 'slicer-4.10'
[KernelApp] Connection file: /home/yousefi/.local/share/jupyter/runtime/kernel-562c7160-c8ee-47a3-a89c-92bba771b660.json
[KernelApp] To connect a client: --existing kernel-562c7160-c8ee-47a3-a89c-92bba771b660.json
/home/yousefi/Slicer-4.10.2-linux-amd64/bin/../Slicer: /software/spackages/linux-centos8-x86_64/gcc-8.3.1/anaconda3-2020.07-4obfocw3kpymwz7obsjukroelfwutobz/lib/libuuid.so.1: no version information available (required by /lib64/libSM.so.6)
/home/yousefi/Slicer-4.10.2-linux-amd64/bin/SlicerApp-real: /software/spackages/linux-centos8-x86_64/gcc-8.3.1/anaconda3-2020.07-4obfocw3kpymwz7obsjukroelfwutobz/lib/libuuid.so.1: no version information available (required by /lib64/libSM.so.6)
/home/yousefi/Slicer-4.10.2-linux-amd64/bin/SlicerApp-real: /software/spackages/linux-centos8-x86_64/gcc-8.3.1/anaconda3-2020.07-4obfocw3kpymwz7obsjukroelfwutobz/lib/libuuid.so.1: no version information available (required by /lib64/libblkid.so.1)
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/local_scratch/pbs.440991.pbs02/runtime-yousefi'
"Failed to create temporary directory" "/local_scratch/pbs.435439.pbs02/Slicer-yousefi"
"Failed to create temporary directory" "/local_scratch/pbs.435439.pbs02/Slicer-yousefi"
"Failed to create temporary directory" "/local_scratch/pbs.435439.pbs02/Slicer-yousefi"
"Failed to create temporary directory" "/local_scratch/pbs.435439.pbs02/Slicer-yousefi"
Switch to module:  "Welcome"
Jupyter connection file: [/home/yousefi/.local/share/jupyter/runtime/kernel-562c7160-c8ee-47a3-a89c-92bba771b660.json]
error: [/home/yousefi/Slicer-4.10.2-linux-amd64/bin/SlicerApp-real] exit abnormally - Report the problem.

The error at the end just says SlicerApp-real exit abnormally. Note that Slicer works perfectly fine and I’m even able to run my Python code. The only problem is that Slicer Jupyter kernel does not start. Any idea how to fix it? Thanks!

[lassoan (Andras Lasso)]

Could you try if it works with latest Slicer Preview Release? (it is recommended anyway, as it has huge Jupyter support improvements)

[myousefi2016 (Mehrdad Yousefi)]

I tried the latest preview release version 4.11 and get the same error. Any idea what’s wrong here?

[myousefi2016 (Mehrdad Yousefi)]

I’m not sure if it helps or not but this command also fails with the same error:

xvfb-run -a /home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/PythonSlicer -m pip install --upgrade pillow --force-reinstall

[myousefi2016 (Mehrdad Yousefi)]

Finally, I think I found something. When I run PythonSlicer like this:

xvfb-run -a /home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/PythonSlicer

and then in the console, I try these Python commands:

Python 3.6.7 (default, Sep 17 2020, 03:03:22) 
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system('pip install --upgrade pillow --force-reinstall')

I got this error:

Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/../lib/Python/lib/python3.6/site.py", line 553, in <module>
    main()
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/../lib/Python/lib/python3.6/site.py", line 539, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/../lib/Python/lib/python3.6/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/../lib/Python/lib/python3.6/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/bin/../lib/Python/lib/python3.6/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/lib/Python/lib/python3.6/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/lib/Python/lib/python3.6/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/yousefi/Slicer-4.11.0-2020-09-16-linux-amd64/lib/Python/lib/python3.6/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata__linux_x86_64-linux-gnu'
256

Any idea what’s missing here that says no module named '_sysconfigdata__linux_x86_64-linux-gnu' 256'?

[lassoan (Andras Lasso)]

This is probably due to mixup of different Python environments.

We have encountered this problem when we simply wanted to run a terminal in Slicer’s Python environment:

Crash when initializing app built and run from Ubuntu 20.04

Knowing that the crash occurred in libc is not very useful. Can you get a stack trace? This may help: https://www.slicer.org/wiki/Documentation/Nightly/Developers/Tutorials/Debug_Instructions#Analyze_a_segmentation_fault I’ve tried to start Slicer in gdb on Linux by following these instructions, I get the error below. When I try to debug using VS Code, I get the same error. osboxes@osboxes:~/D/Slicer-SuperBuild-Release/Slicer-build$ ./Slicer --launch /usr/bin/gnome-terminal Fatal Python error:…

@jcfr can you help us here? This user site packages/sysconfigdata issues has come up again. Should we add Py_NoSiteFlag=1 in Slicer as suggested here and here? (although I’m not sure how that could fix launching of a terminal or gdb)

[myousefi2016 (Mehrdad Yousefi)]

Do you think if I build Slicer from scratch in our cluster with our system installed Python this error will go away?

[lassoan (Andras Lasso)]

I’ve found a much better solution! Run jupyter notebook server from Slicer’s Python environment to resolve the Python site package conflict issue. We need to get this change integrated first and then you can start jupyter notebook server from Slicer’s Python environment using PythonSlicer.exe -m notebook.

[myousefi2016 (Mehrdad Yousefi)]

I tried the latest preview release built on 09/19/2020 revision 29380 and still have the same problem. Pretty much nothing is changed. I tried PythonSlicer -m notebook but it says No module named notebook despite the fact I already installed SlicerJupyter extension.

[lassoan (Andras Lasso)]

SlicerJupyter extension does not install the jupyter package in Slicer’s Python environment (you might not want to use it and it is a huge package). If you want to run the jupyter notebook server in Slicer’s Python environment then run the following commands after installing SlicerJupyter:

# Install jupyter
pip_install('jupyter --no-warn-script-location')
# Create Slicer kernel
slicer.modules.jupyterkernel.updateKernelSpec()
# Install kernel
import jupyter_client
jupyter_client.kernelspec.KernelSpecManager().install_kernel_spec(slicer.modules.jupyterkernel.resourceFolderPath(), user=True, replace=True)

After that, you can start the notebook server by running this command in a terminal:

PythonSlicer -m notebook

[myousefi2016 (Mehrdad Yousefi)]

I run pip_install('jupyter --no-warn-script-location') and I received this error:

[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux2
>>> pip_install('jupyter --no-warn-script-location')
/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/../bin/PythonSlicer: /software/spackages/linux-centos8-x86_64/gcc-8.3.1/anaconda3-2020.07-4obfocw3kpymwz7obsjukroelfwutobz/lib/libuuid.so.1: no version information available (required by /lib64/libSM.so.6)
error: [/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/./python-real] exit abnormally - Report the problem.
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/Python/slicer/util.py", line 2569, in pip_install
    _executePythonModule('pip', args)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/Python/slicer/util.py", line 2545, in _executePythonModule
    logProcessOutput(proc)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/Python/slicer/util.py", line 2517, in logProcessOutput
    raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command '['/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/../bin/PythonSlicer', '-m', 'pip', 'install', 'jupyter', '--no-warn-script-location']' returned non-zero exit status 1.

[lassoan (Andras Lasso)]

/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/bin/…/bin/PythonSlicer: /software/spackages/linux-centos8-x86_64/gcc-8.3.1/anaconda3-2020.07-4obfocw3kpymwz7obsjukroelfwutobz/lib/libuuid.so.1: no version information available (required by /lib64/libSM.so.6)

I don’t know how or why anaconda’s libuuid is found instead of the system libuuid. Can you check if your environment has some conda variables? Was this Slicer started from some virtual Python environment?

Does ./PythonSlicer -m jupyter --no-warn-script-location work from a terminal?

[myousefi2016 (Mehrdad Yousefi)]

Yes that happened because I had an anaconda module loaded in my environment. I just unload that anaconda module and your above script worked fine. Now, I’m running PythonSlicer -m notebook and I’m getting this error, which I think is on our cluster’s fault that has a firewall:

[yousefi@node1266 bin]$ xvfb-run -a ./PythonSlicer -m notebook
Traceback (most recent call last):
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/notebook/__main__.py", line 5, in <module>
    app.launch_new_instance()
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/traitlets/config/application.py", line 663, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/notebook/notebookapp.py", line 2037, in initialize
    self.init_webapp()
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/notebook/notebookapp.py", line 1711, in init_webapp
    success = self._bind_http_server()
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/notebook/notebookapp.py", line 1718, in _bind_http_server
    return self._bind_http_server_unix() if self.sock else self._bind_http_server_tcp()
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/notebook/notebookapp.py", line 1744, in _bind_http_server_tcp
    self.http_server.listen(port, self.ip)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/tornado/tcpserver.py", line 151, in listen
    sockets = bind_sockets(port, address=address)
  File "/home/yousefi/Slicer-4.11.0-2020-09-19-linux-amd64/lib/Python/lib/python3.6/site-packages/tornado/netutil.py", line 174, in bind_sockets
    sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

Is there any way to start notebook and workaround the firewall?

[myousefi2016 (Mehrdad Yousefi)]

Never mind. I solved that problem and it works fine! I just run this:

PythonSlicer -m notebook --ip=127.0.0.1 --port=8888

Yay!!!