Transition of nightly build from Python 3.6.7 to 3.9.10

Development
1

Few days ago, the version of CPython integrated in Slicer has been updated from 3.6.7 to 3.9.10 (see Slicer@34e48e8ae).
As of 2022.01.27, 10:39am ET, I expect all identified issues to be resolved :white_check_mark: .

Acknowledgments

This was made possible by the contributions and help of many :pray:

More specifically, I would like to acknowledge the specific help of @fbordignon, @jamesobutler & @lassoan for the Slicer integration :100: as well as @fbordignon, dand-oss and dbrnz for the contributions related to the update of python-cmake-buildsystem to support CPython 3.7, 3.8 and 3.9 :100:

Status of Jan 27th nightly builds

While working on addressing the PR-6138, we disabled the nightly build.

After manually updating the Slicer source checkout on each build machine, we just restarted them.

This was done by disabling the source checkout update setting the env. variable run_ctest_with_update to FALSE (see here)

Status of transition

Following the transition, we had a few issues to address.

For future reference, here is the list of associated pull requests:
PR-6131 COMP: Update python packages to latest
PR-6132 STYLE: Upgrade python syntax to 3.9 and newer
PR-6133 COMP: Update LibFFI to fix macOS link error related to HAVE_AS_X86_PCREL test
PR-6134 COMP: Fix windows build updating SimpleITK install step
PR-6137 BUG: Fix use of removed time.clock() method

and more recently this one:
PR-6138 BUG: Fix compiled python module import on windows

Notes related to os.add_dll_directory

Starting with CPython 3.8, the directories associated with dependencies of compiled python modules are expected to be added using the os.add_dll_directory function from the CPython standard library.

This snippet from the porting notes is particularly relevant:

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

Source: https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew

To address this in Slicer, we added two modules generated by SuperBuild/python_configure_python_launcher.cmake:
slicer_dll_directories.py Helper module for providing additional search paths for native dependencies when importing extension modules or loading DLLs using ctypes.
sitecustomize.py Ensures additional search paths are systmatically added when using PythonSlicer or Slicer. See https://docs.python.org/3/library/site.html

Notes related to python-cmake-buildsystem

The project now supports building the following versions of CPython:
2.7.3 to 2.7.15
3.5.1 to 3.5.10
3.6.0 to 3.6.15
3.7.0 to 3.7.12
3.8.0 to 3.8.12
3.9.0 to 3.9.10

Additionally, since starting with CPython 3.7 on Linux, and CPython 3.8 on Windows, the LibFFI dependencies is not included anymore in CPython sources anymore.

To address this, we now have a fork of the project adding CMake support.

Last, note that the plan is to contribute this changes updating the existing pull request (PR-535 - Add cmake configuration files) :facepunch: :ok_hand:

See python-cmake-buildsystem/libffi@libffi-cmake-buildsystem-v3.4.2-2021-06-28-f9ea416

Notes related to the CTKAppLauncher

While working on PR-6138, we also:
this also led me to:

  • updated the deployment target of the pre-build macOS binary package from 10.6 (Snow Leopard, released in 2009) to 10.13 (High Sierra, release in 2017). This is consistent with the Slicer macOS deployment target being 10.13.

  • transition the macOS continuous integration from TravisCI to GitHub Actions

  • fixed an issue happening when building the launcher against Qt >= 5.14. See commontk/AppLauncher@ef13633

  • published a static build of Qt 5.15.2 for macOS. See Release qt-5.15.2-macosx10.13-static-x86_64 · jcfr/qt-static-build · GitHub

Related posts

11 Likes
2

Thanks a lot to all, it works!

image

1 Like