OpenSSL issue when building SlicerCAT in Debug mode

keri · May 10, 2021, 2:08pm

Hi,

On Linux (Ubuntu 20.04) when SlicerCAT is built in Debug mode and I try to use wget in a way:

>>> import wget
>>> url = "https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.1-linux-x86_64.tar.gz"
>>> wget.download(url)

I get an error:

Traceback (most recent call last):
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/site-packages/wget.py", line 526, in download
    (tmpfile, headers) = ulib.urlretrieve(binurl, tmpfile, callback)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/home/kerim/Documents/Colada/d/python-install/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>

There is no problems when doing the same on SlicerCAT Release mode.

I’ve tried to google the problem but as I understood usually this problem happens on macOS

Also python notes tells that some problems may arise when building python 3.6 from source against OpenSSL 1.1.0c:

If you are building Python from source, beware that the OpenSSL 1.1.0c release, the most recent as of this update, is known to cause Python 3.6 test suite failures and its use should be avoided without additional patches. It is expected that the next release of the OpenSSL 1.1.0 series will fix these problems. See Issue 28689: OpenSSL 1.1.0c test failures - Python tracker for more information.

By the way on Windows 10 there were no such problems. At least it worked for SlicerCAT Debug (haven’t tried Release).

Could someone approve this problem or refute it?
May this happen because of OpenSSL config (release/debug) mismatch with python config?

P.S. additional information

keri · June 22, 2021, 8:05pm

I’m recompiling my slicercat and I noticed that there is actually no difference whether I use debug or release config: any of them gives the same mentionned error.

I guess the error is caused by OpenSSL missing certificate.

On almost fresh Ubuntu 20.04 the PythonSlicer command:

import ssl
ssl.get_default_verify_paths()

gives:

DefaultVerifyPaths(cafile=None, capath=’/home/kerim/.vs/T/out/build/Linux-GCC-Debug/OpenSSL/certs’, openssl_cafile_env=‘SSL_CERT_FILE’, openssl_cafile=’/home/kerim/.vs/T/out/build/Linux-GCC-Debug/OpenSSL/cert.pem’, openssl_capath_env=‘SSL_CERT_DIR’, openssl_capath=’/home/kerim/.vs/T/out/build/Linux-GCC-Debug/OpenSSL/certs’)

(by the way there is no /home/kerim/.vs/T/out/build/Linux-GCC-Debug/OpenSSL/certs folder)

and on Windows 10 the output is:
DefaultVerifyPaths(cafile=‘C:/C/d/python-install/bin/…/…/Slicer-build/share/Colada-4.13/Slicer.crt’, capath=‘C:\Program Files\Common Files\SSL/certs’, openssl_cafile_env=‘SSL_CERT_FILE’, openssl_cafile=‘C:\Program Files\Common Files\SSL/cert.pem’, openssl_capath_env=‘SSL_CERT_DIR’, openssl_capath=‘C:\Program Files\Common Files\SSL/certs’)

Output on Ubuntu is gotten while slicercat was not completely installed as it has external package (External_julia.cmake) that depends on python and uses wget inside python and thus produces the mentionned error while build step.

Output on Windows 10 is gotten after complete slicercat installation.

So what may be the reason why SSL path are so different on Ubuntu and Windows 10?
Also will Ubuntu add this file after complete Slicer build C:/C/d/python-install/bin/../../Slicer-build/share/Colada-4.13/Slicer.crt?

keri · June 25, 2021, 3:16pm

I understood the error that I used to get.

During build step Slicer somehow changes ssl.get_default_verify_paths() as after full Slicer installation I can see that these paths are fine even on my Ubuntu. I have not found where Slicer do that but it does.

The question then: if I have an external dependency External_julia.cmake which calls wget inside python script (and thus gives me the error above) how can I tell to that external julia project that it needs to be compiled after Slicer is built in my SlicerCAT app? Because usually External_julia.cmake project is called before Slicer is built and this gives me error above (I need somehow specify Slicer as julia dependency).

lassoan · June 25, 2021, 5:30pm

You can set dependencies between projects in CMake. See for example here:

github.com

Slicer/Slicer/blob/master/SuperBuild/External_ParameterSerializer.cmake#L8

    
      
          
          
set( proj ParameterSerializer )
          
          
# Set dependency list
          set(${proj}_DEPENDENCIES JsonCpp ITK)
          
          
# Include dependent projects if any
          ExternalProject_Include_Dependencies(${proj} PROJECT_VAR proj DEPENDS_VAR ${proj}_DEPENDENCIES)
          
          
# Sanity checks
          if(Slicer_USE_SYSTEM_${proj})
            message(FATAL_ERROR "Enabling Slicer_USE_SYSTEM_${proj} is not supported !")
          endif()
          if(DEFINED ${proj}_DIR AND NOT EXISTS ${${proj}_DIR})
            message(FATAL_ERROR "${proj}_DIR variable is defined but corresponds to nonexistent directory")
          endif()
          
          
if(NOT DEFINED ${proj}_DIR AND NOT Slicer_USE_SYSTEM_${proj})

keri · June 25, 2021, 11:49pm

@lassoan thank you for answer
The problem is that it seems I can’t write it:

set(${proj}_DEPENDENCIES
  Slicer    # Slicer is not an external project so I can't add it as a dependency
  )

ExternalProject_Include_Dependencies(${proj} PROJECT_VAR proj DEPENDS_VAR ${proj}_DEPENDENCIES)

and I need to build julia after all OpenSSL staff are done. And I guess it is done on the Slicer build step

lassoan · June 26, 2021, 12:13am

If you depend on openssl then you can add that as dependency. If Slicer does not depend on Julia (it should not) then you can create a superbuild type extension and build Julia in that. Extensions are built after Slicer build is completed and superbuild type extensions can build additional libraries. An additional advantage of building everything in an extension is that users can install it by a single click in the Extensions manager in Slicer.

By the way, there seems to beany options for running Python code from Julia. Have you looked into those? The simplest is probably to implement a CLI module in Julia, the same way as the Slicer MatlabBridge works.

keri · June 26, 2021, 10:56am

Probably making an extension is the right decision. Thank you

For now I rely on pyjulia abilities. I remember we had an conversation on that. I will refer to that later when I work with julia staff more closely

Topic		Replies	Views
Building From Source on Ubuntu 16.04 - OpenSSL Development build	10	3472	October 9, 2017
Which to choose: Crash or Build failure? Development	5	767	April 1, 2020
SSL error with the sample data module (Linux only) Support	4	698	October 16, 2019
Truncated gzip input on the new Stable version (on Linux) Support	6	1872	October 5, 2020
Strange compiling error Development build	7	363	June 21, 2022

OpenSSL issue when building SlicerCAT in Debug mode

Related topics