[ABI Dependency] Best-Effort support for pip

[DEKHTIARJonathan]

Functional and "best effort" implementation for pip.

Should work just fine.

Implementation details

• Can not be a plugin because it needs to break plugin venv isolation.
• User controllable using export VARIANT_ABI_DEPENDENCY=packageA==1.2.3,...,packageZ==7.8.9
• Should support perfectly "sequential installs"

Should support these usecases

pip install torch
# `torch` is pre-installed and automatically fed by `variantlib`
pip install vllm flash-attn

export VARIANT_ABI_DEPENDENCY="torch==2.8.0"
pip install ${VARIANT_ABI_DEPENDENCY} vllm flash-attn

export VARIANT_ABI_DEPENDENCY="torch==2.8.0"
pip install vllm flash-attn

export VARIANT_ABI_DEPENDENCY="torch==2.8.0"
# Assuming the `requirements.txt` file does not contain torch - or pin to `==2.8.0` 
pip install -r requirements.txt

Versioning

The interface supports "sort of range support" and not exclusively major.minor.micro pinning

• export VARIANT_ABI_DEPENDENCY="torch==2.8.0" => only 2.8.0 torch release
• export VARIANT_ABI_DEPENDENCY="torch==2.8" => only >=2.8,<2.9 torch releases
• export VARIANT_ABI_DEPENDENCY="torch==2" => only >=2,<3 torch releases

Which allows developers for different package to express different level of compatibility.

It should be noted that the package maintainers can also combine abi_dependency tags to express an OR match:

• abi_depedency :: torch :: 2.8.0 => exact 2.8.0 match
• abi_depedency :: torch :: 2.9.0 => exact 2.9.0 match
• abi_depedency :: torch :: 3 => >=3,<4 match

=> Any version of torch that matches one of the above will be compatible.

[mgorny]

Mixed feelings about this. We can't handle versions with more than 3 components here.

My main worry is upstream doing something like succession of 2.8, 2.8.1, 2.8.2... — but I guess we are hacking it into 2.8.0 then, so that would kinda work.

[DEKHTIARJonathan]

If you have a better idea let me know but IMHO it's reasonable and easy to read.

[mgorny]

This (quite complex) bit is repeated twice — move it to a function?

[DEKHTIARJonathan]

I'm not sure it's worth it - it's easy to read - moving the whole thing to a function would make it slightly less easy to read.

You tell me - honestly I don't mind enough to argue