Fundamental libraries that Slicer is built on are already available as separate Python packages (VTK, ITK, etc.). There are more libraries that could be made available as standalone Python packages (such as MRML, FreeSurfer, SegmentationCore, PythonQt) with some more work. However, Slicer’s main added value (compared to just using fundamental libraries from Python scripts) that it is an end-user application and so most of the development efforts focuses on improving this aspect.
You can consider each Slicer installation as a virtual Python environment - the same way as you create virtual environments using conda. You should be able to use Slicer’s executables in place of Python executables for most use cases: you can use Slicer application as a Jupyter kernel (https://github.com/Slicer/SlicerJupyter), if you need any more features you can import any Python packages from pip directly into Slicer’s environment, you can connect to Slicer using an external debugger, run Slicer scripts from the command-line, run CLI modules as standalone applications, etc.