LocalPackageResolver: Handle multiple package dirs inside one Python env

[jherland]

Until now, the --pyenv option has accepted any path that points inside a Python environment, e.g.:

• .venv/
• .venv/bin
• .venv/bin/python
• .venv/lib
• .venv/lib/python3.11
• .venv/lib/python3.11/site-packages

These all unambiguously point to the Python environment rooted at .venv/, and they will all result in LocalPackageResolver looking for packages inside .venv/lib/python3.11/site-packages.

But this is all assuming that .venv/lib/python3.11/site-packages is the only directory inside that Python environment that can hold Python packages.

We could have a situation where there are multiple site-packages dirs inside .venv/:

• .venv/lib/python3.9/site-packages
• .venv/lib/python3.11/site-packages

In this case¹, things start to go off the rails. Before this commit, we would first follow parents until .venv/ was found, and then we would use the first result of path.glob("lib/python?.*/site-packages"), in other words, they would end up using .venv/lib/python3.9/site-packages.

This is maybe not what the user intended if they passed in .venv/ or .venv/lib, but it's certainly not what the user wanted if they passed in a full/versioned path like .venv/lib/python3.11/site-packages.

There are two things to be fixed here, and this commit fixes both:

1. When passing the path to a Python environment, there might be multiple package dirs hiding inside. In other words:

   determine_package_dir(cls, path: Path) -> Optional[Path]
   

   is NOT a suitable signature, we rather need the more general:

   find_package_dirs(cls, path: Path) -> Iterator[Path]
   

   signature, which is able to return zero, one, or more results. And the surrounding code should handle this by adding all matching package dirs to the LocalPackageResolver.package_dirs.

2. We should also filter the package dirs found inside the given Python environment to exclude those that are not relative to the given path. That is, if you pass .venv/lib/python3.11/site-packages, you should only get .venv/lib/python3.11/site-packages back, and not .venv/lib/python3.9/site-packages, even if that also exists within the same Python environment.

As a result, a user can now specify --pyenv .venv to get all Python packages within any .venv/lib/python?.*/site-packages/ directory, OR they can specify --pyenv .venv/lib/python3.11/site-packages to be very explicit about which package directory they do want.

The only "casualty" of this change is that specifying a path within the environments that is not a parent of any package dir (for example: --pyenv .venv/bin/python) is no longer supported. This was already a corner case, and I don't think we lose much by no longer supporting it.

Footnotes

1. This whole argument is somewhat strained when using .venv/ virtualenvs as an example: virtualenvs typically only exist within the context of a single Python version (the one in .venv/bin/python), and there is always only one site-packages dir in a virtualenv. However, the same argument applies equally to system environments under e.g. /usr/ or /usr/local/, and they certainly apply to __pypackages__ environments that are not bound to a specific interpreter. ↩

[dpulls]

🎉 All dependencies have been resolved !

[mknorps]

Thank you @jherland !
It looks like good handling of multiple Python version environment handling. I think we are very good at making sure that many corner cases are covered in FawltyDeps 💯

I have some small suggestions and a question if we need the suppress mechanism. Otherwise, I think this is good to go :)

[mknorps]

Thank you @jherland !
It looks like good handling of multiple Python version environment handling. I think we are very good at making sure that many corner cases are covered in FawltyDeps 💯

I have some small suggestions and a question if we need the suppress mechanism. Otherwise, I think this is good to go :)