Fetch unavailable libraries directly from pypi

[kaste]

If a requested library is not available in one of the repositories,
usually the channels "libraries_cache", fetch the information directly
from PyPi.

The core-feature is to automatically discover suitable versions of a
library, even when the Python host changes or gets compiled differently.

Vendors packaging. Requires the 3.8 host.

This should e.g. fetch 313 automatically if a new host arrives without changing the libraries cache.

[deathaxe]

I am aware of packaging package. That's something for a ST4-only py38+-only future version of Package Control.

sdists requires more then just unpacking, thus doesn't need to be fetched. PC couldn't handle it anyway and it won't probably want to do so as it might involve compiling etc.

It's a nice POC for how to leverage packaging package to resolve python version and platform information. Looking up pypi metadata is what current PyPi client basically also does. Parsing file name to extract info wouldn't be a big challange either. PyPi client just uses existing strategies to provide a ST like registry to ensure certain control about which files to pick. E.g. musl vs. manylinux etc.

Libraries channel actually also acts as gate keeper to only support self contained packages or those with small amount of deps, which can be handled manually.

Supporting arbitrary packages without resolving their dependencies is more or less useless in modern python's dependency hell.

Lumbering that into the messy PackageManager is good enough for a POC, but nothing I'd push the overall design to. This could actually also be done by current PyPiClient, alternatively.

[deathaxe]

FWIW, with most libraries such as aiohttp or others maybe vendored by Package Control 5 have already moved on to require more modern python (e.g. py3.9) it is possible PC5 also no longer supporting python 3.8 to not start something new with an already dead ecosystem.

As PC4 has to support py33, this PR and thus a solution to directly manage library installs from pypi may therefore not land in current PC major version.

[kaste]

A fun fact here is: I was looking into the libraries crawler, and hence the pypi json, and so on, and so these json's are just so big, I needed a script to understand that. And the PR here is basically a part of that.

[deathaxe]

The info is certainly of value and is very likely to be used as installing libraries directly from pypi without maintaining a dedicated index is certainly a primary goal for the next major release of PC, but combined with proper recursive dependency resolving.

I am just not yet at the point to say PC5.x will support all ST4 builds including those with py38 or whether we'll hit a point of requiring it py3.13+, leaving all users with current ST4 releases behind with PC4.x

Thus any upstream ecosystem may need to be prepared to handle that.

[deathaxe]

Another blocker and argument for some gatekeeping is the python-markdown package. The current version 3.2.2 is the latest version, compatible with ST's embedded python environment. Any more recent version (e.g. 3.5), which is compatible with python 3.8 does no longer work caused by the way it hacks with pythons stdlib. It would fail with import errors and break all packages depending on it.

They refused to fix the issue as it is a python bug, which was fixed in python 3.10.

Without any posibility to restrict version numbers, blindly fetching everything frop pypi may likely cause trouble.

[kaste]

Yes, totally correct. This PR did look for the libraries first, and then had the fallback to fetch directly. I hope I did that. That kind of hybrid approach is a must. For performance as well. Gatekeeping is also important, there are actually so mixed emotions for this one.

a) I just wanted to ensure or saw a way how 3.13 dependencies could just work without the need for stressful actions on our (yours) side.
b) I saw finally a way to install a lib easily (it was possible before of course) on my own PC; it is my computer, power to the people. Gate-keeping would be then up to package reviews if a user then would like to have that on the public tracker. But my personal stuff, I don't want that on the public tracker.

(b) is basically a user story. It is my computer, can PC help with installing libraries on it

Maybe maintaining allow- and deny-lists would be an option.