I have problems installing lmfit package. Initially, it installs fine with no errors but then when importing it in a script this happens.
The developers said that while installing the setup.py converts the python2 code into python3, but this was not done for me. Can you try and see if there is any problem or it is just my system?
just
slicer.util.pip_install("lmfit") # this works
import lmfit # This gives an error
Using Python 3.6.7 (not the one in Slicer), then installing lmfit has no problems as associated with the dependency uncertainties. There is problems installing it within Slicer though so I would say this is a Slicer or other local problem involving the install.
The failure to build the wheel for the uncertainties package can happen in the local version too, but it is a result of something incorrect in pip cache. Once, I removed an old cache at âC:\Users\JamesButler\AppData\Local\pip\cache\wheelsâ then pip install uncertainties began to compile correctly again within python 3.6.7 outside of Slicer.
I had to manually uninstall uncertainties and reinstall it again using SlicerPython -m pip for it to work. But this breaks automatic installation of the package from within python modules. Using just slicer.util.pip_install I didnât achieve to make it work.
Doing Slicer upgrade, breaks again this package. is there a way to make it run properly? I even installed the 2to3 package and nothing worked out.
I have found that the lib2to3 is a core Python3 library and shall be accesible from within the python interpreter as
from setuptools.lib2to3_ex import Mixin2to3
However, this simple line fails in Slicer with this error
from setuptools.lib2to3_ex import Mixin2to3
File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.6/site-packages/setuptools/lib2to3_ex.py", line 12, in <module>
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
ModuleNotFoundError: No module named 'lib2to3.refactor'
It is not enough to install the 2to3 package, the library must be compiled inside python itself, when compiling python this must be added in order for this mechanism to work.
It may be worth investing time in improving these two packages so that python wheels are made available:
To be very specific, it doesnât look like any file from these libraries is âcompiledâ (there are no C/C++/fortran/⌠files)
That said, since there are no wheel for uncertainties package, the conversion from python2 to python3 code seems to happen for every install. See here
If anyone has time to invest to improve these packages, I would be happy to provide guidance and reference example of packages that are successfully packaged.
The simple line in the console
/usr/bin/2to3- -w /Applications/Slicer.app/Contents/bin/../lib/Python/lib/python3.6/site-packages/uncertainties
Makes the packages to work, however I have not succeeded to make an automatic run inside a script. So far I do
pip_install('lmfit')
# Correct uncertainties
e2to3lib = Path(site.getsitepackages()[0]).parent
sys.path.append(str(e2to3lib))
e2to3 = shutil.which("2to3")
if not e2to3:
pip_install('2to3')
if sys.platform.startswith('win'):
e2to3 = Path(shutil.which("conda")).parent / '2to3.exe' # Windows
else:
e2to3 = Path(shutil.which("python3")).parent / '2to3-' # MACOS
if not e2to3.exists:
e2to3 = Path(shutil.which("python3")).parent / '2to3' # Linux
uncertainties = Path(site.getsitepackages()[0]) / "uncertainties"
try:
print(subprocess.check_output([str(e2to3), "-w", str(uncertainties)],shell=True,stderr=subprocess.STDOUT, env=startupEnvironment()))
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
import lmfit
But this fails with this error:
subprocess.CalledProcessError: Command '['/usr/bin/2to3-', '-w', '/Applications/Slicer.app/Contents/bin/../lib/Python/lib/python3.6/site-packages/uncertainties']' returned non-zero exit status 2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.6/imp.py", line 170, in load_source
module = _exec(spec, sys.modules[name])
File "<frozen importlib._bootstrap>", line 618, in _exec
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/alexvergaragil/Documents/GIT/dosimetry4d/Dosimetry4D/Dosimetry4D.py", line 21, in <module>
from Logic import Dosimetry4DLogic, Dosimetry4DTest, dicomutils, vtkmrmlutils, vtkmrmlutilsTest, testutils, xmlutils, xmlutilsTest, nodes, nodesTest, errors, config, logging, sentry, utils, export
File "/Users/alexvergaragil/Documents/GIT/dosimetry4d/Dosimetry4D/Logic/__init__.py", line 1, in <module>
from .Dosimetry4DLogic import *
File "/Users/alexvergaragil/Documents/GIT/dosimetry4d/Dosimetry4D/Logic/Dosimetry4DLogic.py", line 32, in <module>
from Logic.fit_values import fit_values
File "/Users/alexvergaragil/Documents/GIT/dosimetry4d/Dosimetry4D/Logic/fit_values.py", line 41, in <module>
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
RuntimeError: command '['/usr/bin/2to3-', '-w', '/Applications/Slicer.app/Contents/bin/../lib/Python/lib/python3.6/site-packages/uncertainties']' return with error (code 2): b'At least one file or directory argument required.\nUse --help to show usage.\n'
If creating a wheel is not too long for someone who has never done it, I could indeed use some pointers/guidance and see if I can do this for uncertainties.
I am studying this file. But I canât make it to execute automatically the setup.py from the uncertainties project. If I just clone everything the lmfit is installed but uncertainties still has python2 looking. Basically I need that in the configuration step Slicer executes the setup.py from the project before installing it.
Perhaps a similar solution by just adding the 2to3 package is enough, what do you think?
Hello Eric!
I think you shall take a look a this
Apparently we were not the only ones who need this python wheel. A PR is ready for this.
I could indeed use some pointers/guidance and see if I can do this for uncertainties.
If you would like to setup Continuous Integration to support distribution of wheels, you may want to look at what we did for this package: GitHub - amueller/word_cloud: A little word cloud generator in Python
Most of the old python2 packages implemented an automatic conversion for python3 in the setup itself, this is made using the 2to3 tool that shall be available in the system or in the bundled python. Something like this is inside those packages setup.py file:
...
if sys.version_info >= (3, ):
import subprocess
subprocess.check_call(["2to3", "-w", "."])
this routine will fail if there is no 2to3 executable. That is why I think bundling 2to3 package inside Slicer is very good to avoid all these issues.
It could be nice to include this 2to3 tool in Slicer if somebody has time for it, as there is a chance that it could make some modules work.
However, I have quite bad experience with 2to3: it obfuscated the code (made changes in a way that are not reasonable or maintainable for a human) and caused errors at several places. I would avoid relying on packages that still use 2to3âs automatic conversion to provide a Python3-compatible version because it indicates that the maintainer does not have enough time to do essential development & maintenance tasks.
I agree with @lassoan
Ideally, the project should be updated to support python3. Indeed, in early 2020, there will be no more release of python 2.7
If both python2 and python3 really need to be supported, leveraging package like https://pypi.org/project/six/ can be helpful.
To help identify, part of the code that need to be updated. When working on transitioning Slicer code based, we used https://python-future.org/futurize.html
I agree that Slicer shall not rely on 2to3 converted packages. But it shall be userâs decision to include his own packages, and this 2to3 tool makes some packages to work, as it is in this specific case. Therefore the tool would be a great addition and it is part of the setuptools library (already bundled) as optional package.It is just to tell cmake to compile it.
Specifically remove this line.
Have you tried if removing that line fixes the problem?
@jcfr Do you know why lib2to3 is excluded?
Regardless if Slicer supports it or not, I would not recommend on relying on a package that still have not been properly ported to Python3, as it means that the maintainer could not find 1-2 days in the last 2-3 years to upgrade. This indicates that other aspects of the package might have been neglected, too, and there may not be enough resources to keep the package alive for long.
Removing that line makes cmake to actually install 2to3 executable and it became available in Slicer. The lmfit is now a pure python3 package with wheels (see here the discussion) and the uncertainties package is making a switch for python3 see here and started up by making wheels see here
Features not explicitly required have been excluded.
If we ship the 2to3 executable, we would have to make sure a launcher is also configured for macOS.
Considering that python 2 will be officially retired in few months, I would suggest to work with maintainers to migrate their code.
To address issue #5042, I just submitted this pull request: