Consider system site packages if activated #11670
Draft
+682
−204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
konstin
added
the
enhancement
konstin
force-pushed
the
konsti/sys-path-discovery
branch
6 times, most recently
from
2ab39f4 to
658d4d4
Compare
konstin
force-pushed
the
konsti/sys-path-discovery
branch
from
658d4d4 to
f926830
Compare
konstin
force-pushed
the
konsti/sys-path-discovery
branch
from
f926830 to
0c24298
Compare
konstin
force-pushed
the
konsti/sys-path-discovery
branch
from
0c24298 to
690a068
Compare
This better reflects the intended behavior of this object, which is to serve as a database of InstalledPackages rather than a representation of the relevant environment/interpreter path.
This behavior was originally written for supporting `sys.path` instead of `site.getsitepackages()`: It's currently not load-bearing, but it is correct deduplication if we need to add more paths to the package discovery in the future. With this change the behavior of `uv pip list` mirrors the behavior of `pip list`: It shows only the first package in `sys.path`, which is the package that Python will load, and ignores the others. In the JSON output, we show all packages, but tagged with a `shadowed` bool. The exception is `--outdated`, in which case we only check if the main packages are outdated (updating shadowed packages is ineffective). The consumer is responsible for filtering on `shadowed` if applicable. We show a hint (stderr, not stdout like the main output), if shadowed packages exist. The logic includes handling of the fedora lib/lib64 venv split.
Previously, uv was discovering packages only in purelib/platlib, which excluded system site packages if activated. `site.getsitepackages()` is more stable than `sys.path`. `sys.path` can change in a number of ways such as `.pth` files and runtime modification, while `site.getsitepackages()` is generally stable for a Python environment. Using the `site`, we get the same paths that Python itself uses for `sys.path`.
Co-authored-by: Pradyun Gedam <[email protected]> When a venv is created, values such as `sys.path` and `site.getsitepackages()` are different from the base interpreter, which we query from the interpreter directly instead of trying to do the transformation ourselves.
konstin
force-pushed
the
konsti/sys-path-discovery
branch
from
690a068 to
459604e
Compare
konstin
changed the title
uv pip list
konstin
force-pushed
the
konsti/sys-path-discovery
branch
from
459604e to
d0e84cf
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
We use
site.getsitepackages()for discovering system packages for venvs created with--system-site-packages, while trying to avoid breaking exiting workflows.Details
We currently only check in ~1 directory for install packages when looking a system or venv interpreter, usually the venv site packages. But in reality, Python (and
importlib.metadata) look in allsys.pathentries for packages.This can lead to a difference between the runtime behavior and
uv pip listas well as other uv operation. Notably, we miss system site packages when used (--system-site-packageson venv installation) and PyPy's std vendored packages (such ascffi).Using the full
sys.pathfor discovery means that we're using a variable that has many sources (including.pthfiles) and that can change at runtime. It easily clashes with our caching as we need to either figure all the places that influencesys.pathor re-query the Python interpreter on each operation.Besides the usual venv or system Python operation mode,
uv pipalso hastargetandprefixoptions that install into a directory (structure). We keeptargetandprefixinstallation as they were, so that they only consider and modify packages in their directory prefix.Packages may be uninstallable or not un-uninstallable, where the latter category are system packages, e.g., those installed into std for PyPy. Currently, a PyPy venv will for example shadow the system cffi version with its own version. With the current change, we still don't consider the PyPy std installed packages (they are not reported by
site.getsitepackages()), preserving the current behavior. If we add support in the future, we need to consider what happens when let's say a lockfile requires cffi 1.71.1 but PyPy has a vendored, non-removable cffi 1.80.0.TODO
--exactwith--system-site-packages.An open question is how to handle system packages during:
uv syncanduv pip install. Foruv sync, do we error or try to upgrade (and potentially fail) when the versions mismatch? Foruv pip installshould we set them as constraint, and if so, do we need to add a new incompatibility kind for system packages (currently, system packages are preferences).Best reviewed commit-by-commit.
Closes #9849
Resolves #2500
TODO #4466