Unhandled error from already-installed dependency with non-standard dependency specifier #12950

edmorley · 2024-09-02T10:41:45Z

Description

In #12063 support for non-standard (ie: broken) dependency specifiers was first deprecated, and then in pip 24.1.0, removed entirely.

As such pip now ignores such broken packages when resolving dependencies during package installation - meaning they can no longer be installed. This is fine, and currently working as expected.

However, what it not working as expected, is that if there are any already-installed dependencies with those dependency specifiers then pip install fails with an internal error and does not explain what happened or how to resolve it.

Expected behavior

Using the steps to reproduce below, pip should either:

Succeed as a no-op, since the requested dependency is satisfied (ie: there is nothing to install/uninstall), since "celery" is satisfied by any installed version. (And in fact the log output below even says "Requirement already satisfied".)
Or, successfully uninstall the broken package and install a non-broken version (which for Celery, is 5.3.1+).
Or error, but do so with a helpful error message (explaining broken packages are installed and to either manually remove those packages or downgrade pip), rather than show an unhandled internal error/traceback.

pip version

24.2

Python version

3.12.5 (homebrew)

OS

macOS

How to Reproduce

mkdir testcase && cd $_
python -m venv .venv && source .venv/bin/activate
pip install pip==24.0
pip install celery==4.4.7
pip install pip==24.2
pip install celery

Output

(.venv) testcase $ pip install celery
Requirement already satisfied: celery in ./.venv/lib/python3.12/site-packages (4.4.7)
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 118, in _parse_requirement_details
    specifier = _parse_specifier(tokenizer)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 208, in _parse_specifier
    with tokenizer.enclosing_tokens(
  File "/opt/homebrew/Cellar/[email protected]/3.12.5/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_tokenizer.py", line 189, in enclosing_tokens
    self.raise_syntax_error(
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    pytz (>dev)
         ~^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 379, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 401, in iter_dependencies
    for r in self.dist.iter_dependencies():
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py", line 215, in iter_dependencies
    req = get_requirement(req_string.strip())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
pip._vendor.packaging.requirements.InvalidRequirement: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    pytz (>dev)
         ~^

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

edmorley · 2024-09-02T10:44:26Z

This blocks Heroku upgrading from pip 24.0 to something newer (eg heroku/heroku-buildpack-python#1619), since we will no doubt have a long tail of users with existing cached site-packages with packages using non-standard dependency specifiers installed. (Whilst users should be upgrading to newer versions of packages with those specifiers fixed, if there isn't a clear error message, they won't know what the issue is or how to resolve it.)

Note: This issue seems similar to #12945, though that issue seems more about initial installation, rather than "existing installation", plus doesn't have explicit steps to reproduce - so it seemed best to file a separate issue.

notatallshaw · 2024-09-04T00:20:34Z

I've created #12953 that gives a much clearer error message:

$ python -m pip install --dry-run celery
Requirement already satisfied: celery in ./.venv/lib/python3.12/site-packages (4.4.7)
error: invalid-installed-package

× Cannot process installed package celery 4.4.7 in '/home/damian/opensource/contributions/pip/.venv/lib/python3.12/site-packages' because it has an invalid requirement:
│ Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
│     pytz (>dev)
│          ~^
╰─> Starting with pip 24.1, packages with invalid requirements can not be processed.

hint: To proceed this package must be uninstalled using 'pip<24.1', some other Python package tool, or manually deleted.

Edit: Updated message to reflect latest version of PR

Refactors cache handling to dedicated functions under `lib/cache.sh` (rather than being scattered around the buildpack), and makes the following improvements: - Ensures the cache is now also discards the cache when the package manager (or its version) changes. - Improves the build log output shown when restoring or discarding the cache. For example, if the cache was invalidated all reasons are now shown. - Stops performing unnecessary cache file copies when the cache is due to be invalidated. This required moving the cache restoration step to after the `bin/pre_compile` hook runs. - Fixes cache restoration in the case where an app's `requirements.txt` was formerly a symlink. - Adds buildpack metrics for the status of the cache and duration of cache restoration/saving. Fixes #1673. Fixes #1674. Fixes #1675. Fixes #1676. Fixes #1677. Fixes #1678. Prep for #796. Unblocks upgrading pip (since #1674 prevents pypa/pip#12950). GUS-W-16811131.

edmorley added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Sep 2, 2024

This was referenced Sep 2, 2024

Error when installing celery < 5.0.0 #12945

Closed

Bump pip from 24.0 to 24.2 heroku/heroku-buildpack-python#1619

Closed

edmorley changed the title ~~Unhandled error from already satisified dependency with non-standard dependency specifier~~ Unhandled error from already installed dependency with non-standard dependency specifier Sep 2, 2024

edmorley changed the title ~~Unhandled error from already installed dependency with non-standard dependency specifier~~ Unhandled error from already-installed dependency with non-standard dependency specifier Sep 2, 2024

notatallshaw mentioned this issue Sep 4, 2024

Diagnostic error when processing package already installed with invalid requirement #12953

Merged

notatallshaw mentioned this issue Sep 27, 2024

pip doesn't work (pyodbc version related error) #12979

Closed

1 task

sbidoul closed this as completed in #12953 Oct 21, 2024

edmorley mentioned this issue Oct 30, 2024

Improve cache handling heroku/heroku-buildpack-python#1679

Merged

github-actions bot locked as resolved and limited conversation to collaborators Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unhandled error from already-installed dependency with non-standard dependency specifier #12950

Unhandled error from already-installed dependency with non-standard dependency specifier #12950

edmorley commented Sep 2, 2024 •

edited

Loading

edmorley commented Sep 2, 2024 •

edited

Loading

notatallshaw commented Sep 4, 2024 •

edited

Loading

Unhandled error from already-installed dependency with non-standard dependency specifier #12950

Unhandled error from already-installed dependency with non-standard dependency specifier #12950

Comments

edmorley commented Sep 2, 2024 • edited Loading

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

edmorley commented Sep 2, 2024 • edited Loading

notatallshaw commented Sep 4, 2024 • edited Loading

edmorley commented Sep 2, 2024 •

edited

Loading

edmorley commented Sep 2, 2024 •

edited

Loading

notatallshaw commented Sep 4, 2024 •

edited

Loading