Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependencies of Git Dependencies are not always installed #9066

Closed
meln5674 opened this issue Feb 29, 2024 · 9 comments · Fixed by #9099
Closed

Dependencies of Git Dependencies are not always installed #9066

meln5674 opened this issue Feb 29, 2024 · 9 comments · Fixed by #9099
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@meln5674
Copy link

meln5674 commented Feb 29, 2024

Description

Adding https://github.com/kubernetes-client/python as a dependency via git does not install its dependencies. Using pip directly installs dependencies, as does installing the package from pypi.

Workarounds

Manually copying dependencies to pyproject.toml

Poetry Installation Method

pipx

Operating System

Debian GNU/Linux 12 (bookworm)

Poetry Version

Poetry (version 1.8.1)

Poetry Configuration

cache-dir = "/home/andrew/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/andrew/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

[tool.poetry]
name = "git-deps-issue"
version = "0.1.0"
description = ""
authors = ["Someone <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
kubernetes = {git = "https://github.com/kubernetes-client/python"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

# poetry -vvv install
Loading configuration file /home/andrew/.config/pypoetry/config.toml
Creating virtualenv git-deps-issue-5ERwWHeK-py3.11 in /home/andrew/.cache/pypoetry/virtualenvs
[virtualenv] find interpreter for spec PythonSpec(path=/home/andrew/.local/pipx/venvs/poetry/bin/python)
[filelock:filelock] Attempting to acquire lock 139883611697616 on /home/andrew/.local/share/virtualenv/py_info/1/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.lock
[filelock:filelock] Lock 139883611697616 acquired on /home/andrew/.local/share/virtualenv/py_info/1/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.lock
[virtualenv] got python info of %s from (PosixPath('/usr/bin/python3.11'), PosixPath('/home/andrew/.local/share/virtualenv/py_info/1/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.json'))
[filelock:filelock] Attempting to release lock 139883611697616 on /home/andrew/.local/share/virtualenv/py_info/1/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.lock
[filelock:filelock] Lock 139883611697616 released on /home/andrew/.local/share/virtualenv/py_info/1/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.lock
[virtualenv] proposed PythonInfo(spec=CPython3.11.2.final.0-64, system=/usr/bin/python3.11, exe=/home/andrew/.local/pipx/venvs/poetry/bin/python, platform=linux, version='3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] accepted PythonInfo(spec=CPython3.11.2.final.0-64, system=/usr/bin/python3.11, exe=/home/andrew/.local/pipx/venvs/poetry/bin/python, platform=linux, version='3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] filesystem is case-sensitive
[virtualenv] create virtual environment via CPython3Posix(dest=/home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11, clear=False, no_vcs_ignore=False, global=False)
[virtualenv] create folder /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin
[virtualenv] create folder /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages
[virtualenv] write /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/pyvenv.cfg
[virtualenv] 	home = /usr/bin
[virtualenv] 	implementation = CPython
[virtualenv] 	version_info = 3.11.2.final.0
[virtualenv] 	virtualenv = 20.25.1
[virtualenv] 	include-system-site-packages = false
[virtualenv] 	base-prefix = /usr
[virtualenv] 	base-exec-prefix = /usr
[virtualenv] 	base-executable = /usr/bin/python3.11
[virtualenv] symlink /usr/bin/python3.11 to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/python
[virtualenv] create virtualenv import hook file /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/_virtualenv.pth
[virtualenv] create /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/_virtualenv.py
[virtualenv] ============================== target debug ==============================
[virtualenv] debug via /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/python /home/andrew/.local/pipx/venvs/poetry/lib/python3.11/site-packages/virtualenv/create/debug.py
[virtualenv] {
[virtualenv]   "sys": {
[virtualenv]     "executable": "/home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/python",
[virtualenv]     "_base_executable": "/usr/bin/python3.11",
[virtualenv]     "prefix": "/home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11",
[virtualenv]     "base_prefix": "/usr",
[virtualenv]     "real_prefix": null,
[virtualenv]     "exec_prefix": "/home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11",
[virtualenv]     "base_exec_prefix": "/usr",
[virtualenv]     "path": [
[virtualenv]       "/usr/lib/python311.zip",
[virtualenv]       "/usr/lib/python3.11",
[virtualenv]       "/usr/lib/python3.11/lib-dynload",
[virtualenv]       "/home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages"
[virtualenv]     ],
[virtualenv]     "meta_path": [
[virtualenv]       "<class '_virtualenv._Finder'>",
[virtualenv]       "<class '_frozen_importlib.BuiltinImporter'>",
[virtualenv]       "<class '_frozen_importlib.FrozenImporter'>",
[virtualenv]       "<class '_frozen_importlib_external.PathFinder'>"
[virtualenv]     ],
[virtualenv]     "fs_encoding": "utf-8",
[virtualenv]     "io_encoding": "utf-8"
[virtualenv]   },
[virtualenv]   "version": "3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]",
[virtualenv]   "makefile_filename": "/usr/lib/python3.11/config-3.11-x86_64-linux-gnu/Makefile",
[virtualenv]   "os": "<module 'os' (frozen)>",
[virtualenv]   "site": "<module 'site' (frozen)>",
[virtualenv]   "datetime": "<module 'datetime' from '/usr/lib/python3.11/datetime.py'>",
[virtualenv]   "math": "<module 'math' (built-in)>",
[virtualenv]   "json": "<module 'json' from '/usr/lib/python3.11/json/__init__.py'>"
[virtualenv] }
[virtualenv] add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, via=copy, app_data_dir=/home/andrew/.local/share/virtualenv)
[virtualenv] install setuptools from wheel /home/andrew/.local/pipx/venvs/poetry/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/setuptools-69.1.0-py3-none-any.whl via CopyPipInstall
[virtualenv] install pip from wheel /home/andrew/.local/pipx/venvs/poetry/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-24.0-py3-none-any.whl via CopyPipInstall
[filelock:filelock] Attempting to acquire lock 139883609168016 on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Attempting to acquire lock 139883609175184 on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Lock 139883609168016 acquired on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Lock 139883609175184 acquired on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Attempting to release lock 139883609168016 on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Lock 139883609168016 released on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Attempting to release lock 139883609175184 on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Lock 139883609175184 released on /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/pkg_resources to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/pkg_resources
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip-24.0.dist-info to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/pip-24.0.dist-info
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/pip
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/_distutils_hack to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/_distutils_hack
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools-69.1.0.dist-info to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/setuptools-69.1.0.dist-info
[virtualenv] copy /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/distutils-precedence.pth to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/distutils-precedence.pth
[virtualenv] copy /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools-69.1.0.virtualenv to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/setuptools-69.1.0.virtualenv
[virtualenv] copy directory /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/setuptools
[virtualenv] generated console scripts 
[virtualenv] copy /home/andrew/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip-24.0.virtualenv to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages/pip-24.0.virtualenv
[distlib:distlib.util] changing mode of /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/pip-3.11 to 755
[distlib:distlib.util] changing mode of /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/pip3.11 to 755
[distlib:distlib.util] changing mode of /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/pip3 to 755
[distlib:distlib.util] changing mode of /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/bin/pip to 755
[virtualenv] generated console scripts pip3 pip3.11 pip-3.11 pip
[virtualenv] add activators for Bash, CShell, Fish, Nushell, PowerShell, Python
[virtualenv] write /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/pyvenv.cfg
[virtualenv] 	home = /usr/bin
[virtualenv] 	implementation = CPython
[virtualenv] 	version_info = 3.11.2.final.0
[virtualenv] 	virtualenv = 20.25.1
[virtualenv] 	include-system-site-packages = false
[virtualenv] 	base-prefix = /usr
[virtualenv] 	base-exec-prefix = /usr
[virtualenv] 	base-executable = /usr/bin/python3.11
[virtualenv] 	prompt = git-deps-issue-py3.11
Using virtualenv: /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals

  - Installing kubernetes (29.0.0+snapshot d36ec43): Pending...
  - Installing kubernetes (29.0.0+snapshot d36ec43): Installing...
  - Installing kubernetes (29.0.0+snapshot d36ec43)

Installing the current project: git-deps-issue (0.1.0)
  - Building package git-deps-issue in editable mode
  - Adding git_deps_issue.pth to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages for /home/andrew/git/src/github.com/meln5674/poetry-test
  - Adding the git_deps_issue-0.1.0.dist-info directory to /home/andrew/.cache/pypoetry/virtualenvs/git-deps-issue-5ERwWHeK-py3.11/lib/python3.11/site-packages
@meln5674 meln5674 added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Feb 29, 2024
@dimbleby
Copy link
Contributor

https://github.com/kubernetes-client/python/blob/851dc2a0b10d1bc07826c6ade489d1bd2ef431c3/setup.py#L33-L45

I guess the setup reader spots REQUIRES = [], reckons it understands that, and decides that it is done.

In most cases where setup.py builds up requirements via a series of conditional tests, this mistake doesn't really matter: in the sense that poetry only supports projects where the metadata is the same for all distributions, so a setup that goes "if windows this, if linux that" is fundamentally not compatible with poetry.

In this case the code is providing a static list, but doing it in a rather elaborate way.

if what you care about is getting this working with kubernetes-client then probably your best bet is to go and offer them a pull request to rearrange things so that poetry doesn't get fooled.

eg

with open('requirements.txt') as f:
    REQUIRES = [line.split(" #")[0] for line in f.readlines()]

seems like a simplification they might want anyway, and should be something that poetry recognises it cannot understand

@meln5674
Copy link
Author

While that might be the case (and I'm already working off a fork, so I can just do it there), I don't think its too unreasonable to expect poetry to behave "close enough" to "pip install", and not do something completely different, silently. I'll refer you to my comment on the previous issue regarding this same bug: Heuristics are great, right up until they are wrong. I stand by my position that poetry ought to have some mechanism to say "Your guess is wrong, please build it.". I'd be happy to contribute something like this if it would be accepted.

@dimbleby
Copy link
Contributor

I'm not a fan of "make it work" flags: if this is going to be fixed, let it be fixed without a config option.

I guess it's possible to go in either direction:

  • make the setup reader more sophisticated, so that it recognises that this is in fact a case that it doesn't know how to handle
  • make the setup reader less sophisticated, handling a narrower range of cases about which it really can be sure
    • eg it would be easy to make it not find REQUIRES at all: once it realises that the install_requires argument is a variable it could just give up there
  • or maybe go all the way and abandon the setup reader altogether.

I would give serious consideration to removing the setup reader, though I don't know how maintainers would feel about that

The python ecosystem is quite different than it was when this code was written, most popular projects now have wheels declaring static metadata. So the trade between "time saved by not building a wheel" and "sometimes being wrong" is in a different place than it was.

@abn
Copy link
Member

abn commented Mar 1, 2024

Setup reader was always considered a fallback. I thought we have had pep 517 build before falling back to setup reader. Or I might have incorrectly believed that since an empty list was previously being treated as a bad read and we fell back to a 517 metadata build.

I'm curious if we are just way too optimistic about setup reader's parsed list.

@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2024

setup reader happens before build and has done for a long time, probably since forever

@meln5674
Copy link
Author

meln5674 commented Mar 1, 2024

If you don't like "make it work" options, and while this might be considered a breaking change, what about needing to opt into the setupreader for path and git dependencies? It seems like it'd be much safer to do the slow, but reliable build by default unless the user explicitly gives the go-ahead for the faster, but riskier option.

@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2024

That is still a "make it work" flag. IMO any config option here is a mistake

@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2024

To be concrete: the intermediate "keep the setup reader but make it less ambitious" middle option probably looks something like this:

diff --git a/src/poetry/utils/setup_reader.py b/src/poetry/utils/setup_reader.py
index 3c233cd9..a348e3b3 100644
--- a/src/poetry/utils/setup_reader.py
+++ b/src/poetry/utils/setup_reader.py
@@ -222,9 +222,6 @@ class SetupReader:
         if value is None:
             return []

-        if isinstance(value, ast.Name):
-            value = self._find_variable_in_body(body, value.id)
-
         if isinstance(value, ast.Constant) and value.value is None:
             return []

@@ -261,9 +258,6 @@ class SetupReader:
         if value is None:
             return {}

-        if isinstance(value, ast.Name):
-            value = self._find_variable_in_body(body, value.id)
-
         if isinstance(value, ast.Constant) and value.value is None:
             return {}

(plus corresponding fixes to testcases)

If I cared to make this more reliable, and did not want to try and persuade maintainers that the setup reader should be abandoned altogether, that is the approach I would pursue

Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants