Slicer fails to download custom sample data

I have narrowed the problem down. This script works:

import sys
import traceback
import urllib.request
import requests

def processEvents():
    try:
        import slicer
        slicer.app.processEvents()
    except (AttributeError, ImportError):
        pass

    
def display(text, file=sys.stdout):
    processEvents()
    print(text, flush=True, file=file)

    
def error(text):
    display(text, file=sys.stderr)


for url, expected_success in [
    ("https://data.kitware.com", True),
    ("https://www.httpvshttps.com/", True),
    ("https://slicer.org/", True),
    ("https://expired.badssl.com/", False),
    ("https://github.com/", True),
]:
    display("-" * 8)
    display(f"Checking {url}")
    goodCert = r"C:\Program Files\Python39\Lib\site-packages\certifi\cacert.pem"
    badCert = requests.utils.DEFAULT_CA_BUNDLE_PATH  # C:\Users\Dzenan\AppData\Local\.certifi
    try:
        with urllib.request.urlopen(url, cafile=goodCert) as response:
           data = response.read()
           html = data.decode('utf8')
           assert "<head>" in html[:600]
        if expected_success:
            display(f"Checking {url} - OK")
        else:
            error(f"Checking {url} - FAILED - should have raised an exception")
    except Exception as exc:
        if expected_success:
            error(f"Checking {url} - FAILED - unexpected exception")
            traceback.print_exc()
        else:
            display(f"Checking {url} - OK  [Expected {exc}]")

I tried copying goodCert into badCertā€™s place. But when this script is run, the goodCert gets turned into badCert by extending it by a bunch more entries. The fileā€™s size is increased from 253KB to 425KB.

Does anyone have an idea what to try next?

I have now narrowed it down to a single certificate.

-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----

human readable variant:

Certificate[129]:
Owner: CN=R3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 400175048314a4c8218c84a90c16cddf
Valid from: Wed Oct 07 15:21:40 EDT 2020 until: Wed Sep 29 15:21:40 EDT 2021
Certificate fingerprints:
SHA1: 48:50:4E:97:4C:0D:AC:5B:5C:D4:76:C8:20:22:74:B2:4C:8C:71:72
SHA256: 73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: caIssuers
   accessLocation: URIName: http://apps.identrust.com/roots/dstrootcax3.p7c
]
]

#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: C4 A7 B1 A4 7B 2C 71 FA   DB E1 4B 90 75 FF C4 15  .....,q...K.u...
0010: 60 85 89 10                                        `...
]
]

#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.identrust.com/DSTROOTCAX3CRL.crl]
]]

#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.23.140.1.2.1]
[]  ]
  [CertificatePolicyId: [1.3.6.1.4.1.44947.1.1.1]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 22 68 74 74 70 3A 2F   2F 63 70 73 2E 72 6F 6F  ."http://cps.roo
0010: 74 2D 78 31 2E 6C 65 74   73 65 6E 63 72 79 70 74  t-x1.letsencrypt
0020: 2E 6F 72 67                                        .org

]]  ]
]

#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

#7: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]

#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 14 2E B3 17 B7 58 56 CB   AE 50 09 40 E6 1F AF 9D  .....XV..P.@....
0010: 8B 14 C2 C6                                        ....
]
]

If this certificate is not present in cacert.pem, the script works as it should. If this certificate is present, all websites have ā€œcertificate has expiredā€ problem within Python.

I tried deleting it from the local computer, but it reappeared.

The DST Root CA X3 certificate expired [1] in September 2021.

Consider updating the version of certifi associated with your Python install. See https://pypi.org/project/certifi/

But when this script is run, the goodCert gets turned into badCert by extending it by a bunch more entries.

Involved functions are the following:


  1. DST Root CA X3 Expiration (September 2021) - Let's Encrypt ā†©ļøŽ

JC and I had a debugging session on my computer. We didnā€™t resolve the problem. But JC found this:

Starting with Python 2.7.9 and 3.2, the function ā€œssl.create_default_context()ā€ automatically loads system certificates. This explains why the ā€œSlicer.crtā€ we provide (through the SSL_CERT_FILE env. variable expected by openssl) is being ignored

I also figured out why the certificate comes back after I delete it:
Screenshot 2023-04-07 15.49.39
And in text form for easier searching:

Deleting system root certificates might prevent some Windows components from working properly. The list of system critical root certificates can be reviewed at Microsoft Support. If Update Root Certificates is installed, any deleted third-party root certificates will be restored automatically, but the system root certificates will not. Do you want to delete the selected certificate(s)?

Downloading and installing certificates from this website solved the problem for me. Here are the links:

Root CA Certificates (PEM format):

Intermediate Certificate (PEM format):

Thank you JC, Andras and @jake.stookey for the help.

1 Like