dependency resolution discrepancy between install and update

[dericcrago]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian GNU/Linux 10 (buster)

• Poetry version: 1.1.4

• Link of a Gist with the contents of your pyproject.toml file: See below

Issue

Immediately running poetry update after poetry install results in an update.

Reproducer with wildcards:

aiohttp 2.3.10 is installed, but is then updated to aiohttp 3.7.3

docker container run -it --rm python:3.9.1 /bin/bash

root@11ded5370ef7:/# python3 -m pip install poetry

root@11ded5370ef7:/# poetry --version
Poetry version 1.1.4

root@11ded5370ef7:/# poetry init --no-interaction --name foo --dependency 'aiohttp==*' --dependency 'sphinx==*'

root@11ded5370ef7:/# cat pyproject.toml
[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
aiohttp = "*"
Sphinx = "*"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

root@11ded5370ef7:/# poetry install
Creating virtualenv foo-il7asoJj-py3.9 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (34.4s)

Writing lock file

Package operations: 28 installs, 0 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0)
  • Installing idna (2.10)
  • Installing markupsafe (1.1.1)
  • Installing multidict (5.1.0)
  • Installing pyparsing (2.4.7)
  • Installing pytz (2020.5)
  • Installing urllib3 (1.26.2)
  • Installing alabaster (0.7.12)
  • Installing async-timeout (3.0.1)
  • Installing babel (2.9.0)
  • Installing docutils (0.16)
  • Installing idna-ssl (1.1.0)
  • Installing imagesize (1.2.0)
  • Installing jinja2 (2.11.2)
  • Installing packaging (20.8)
  • Installing pygments (2.7.3)
  • Installing requests (2.25.1)
  • Installing snowballstemmer (2.0.0)
  • Installing sphinxcontrib-applehelp (1.0.2)
  • Installing sphinxcontrib-devhelp (1.0.2)
  • Installing sphinxcontrib-htmlhelp (1.0.3)
  • Installing sphinxcontrib-jsmath (1.0.1)
  • Installing sphinxcontrib-qthelp (1.0.3)
  • Installing sphinxcontrib-serializinghtml (1.1.4)
  • Installing yarl (1.6.3)
  • Installing aiohttp (2.3.10)
  • Installing sphinx (3.4.3)

root@11ded5370ef7:/# poetry show --outdated
aiohttp 2.3.10 3.7.3 Async http client/server framework (asyncio)
idna    2.10   3.1   Internationalized Domain Names in Applications (IDNA)

root@11ded5370ef7:/# poetry show --tree
aiohttp 2.3.10 Async http client/server framework (asyncio)
├── async-timeout >=1.2.0
├── chardet *
├── idna-ssl >=1.0.0
│   └── idna >=2.0
├── multidict >=4.0.0
└── yarl >=1.0.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
│   ├── certifi >=2017.4.17
│   ├── chardet >=3.0.2,<5
│   ├── idna >=2.5,<3
│   └── urllib3 >=1.21.1,<1.27
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

root@11ded5370ef7:/# poetry update
Updating dependencies
Resolving dependencies... (1.2s)

Writing lock file

Package operations: 2 installs, 2 updates, 1 removal

  • Removing idna-ssl (1.1.0)
  • Updating chardet (4.0.0 -> 3.0.4)
  • Installing attrs (20.3.0)
  • Installing typing-extensions (3.7.4.3)
  • Updating aiohttp (2.3.10 -> 3.7.3)

root@11ded5370ef7:/# poetry show --outdated
chardet 3.0.4 4.0.0 Universal encoding detector for Python 2 and 3
idna    2.10  3.1   Internationalized Domain Names in Applications (IDNA)

root@11ded5370ef7:/# poetry show --tree
aiohttp 3.7.3 Async http client/server framework (asyncio)
├── async-timeout >=3.0,<4.0
├── attrs >=17.3.0
├── chardet >=2.0,<4.0
├── multidict >=4.5,<7.0
├── typing-extensions >=3.6.5
└── yarl >=1.0,<2.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
│   ├── certifi >=2017.4.17
│   ├── chardet >=3.0.2,<5
│   ├── idna >=2.5,<3
│   └── urllib3 >=1.21.1,<1.27
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

root@11ded5370ef7:/# poetry update
Updating dependencies
Resolving dependencies... (0.3s)

No dependencies to install or update

root@11ded5370ef7:/# poetry show --outdated
chardet 3.0.4 4.0.0 Universal encoding detector for Python 2 and 3
idna    2.10  3.1   Internationalized Domain Names in Applications (IDNA)

root@11ded5370ef7:/# poetry show --tree
aiohttp 3.7.3 Async http client/server framework (asyncio)
├── async-timeout >=3.0,<4.0
├── attrs >=17.3.0
├── chardet >=2.0,<4.0
├── multidict >=4.5,<7.0
├── typing-extensions >=3.6.5
└── yarl >=1.0,<2.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
│   ├── certifi >=2017.4.17
│   ├── chardet >=3.0.2,<5
│   ├── idna >=2.5,<3
│   └── urllib3 >=1.21.1,<1.27
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

Reproducer with specific versions:

requests 2.15.1 is installed, but is then updated to requests 2.25.1

docker container run -it --rm python:3.9.1 /bin/bash

root@7a29451bf5eb:/# python3 -m pip install poetry

root@7a29451bf5eb:/# poetry --version
Poetry version 1.1.4

root@7a29451bf5eb:/# poetry init --no-interaction --name foo --dependency 'aiohttp>=3.7.3' --dependency 'sphinx>=3.4.3'

root@7a29451bf5eb:/# cat pyproject.toml
[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
aiohttp = ">=3.7.3"
Sphinx = ">=3.4.3"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

root@7a29451bf5eb:/# poetry install
Creating virtualenv foo-il7asoJj-py3.9 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (26.7s)

Writing lock file

Package operations: 27 installs, 0 updates, 0 removals

  • Installing idna (3.1)
  • Installing markupsafe (1.1.1)
  • Installing multidict (5.1.0)
  • Installing pyparsing (2.4.7)
  • Installing pytz (2020.5)
  • Installing alabaster (0.7.12)
  • Installing async-timeout (3.0.1)
  • Installing attrs (20.3.0)
  • Installing babel (2.9.0)
  • Installing chardet (3.0.4)
  • Installing docutils (0.16)
  • Installing imagesize (1.2.0)
  • Installing jinja2 (2.11.2)
  • Installing packaging (20.8)
  • Installing pygments (2.7.3)
  • Installing requests (2.15.1)
  • Installing snowballstemmer (2.0.0)
  • Installing sphinxcontrib-applehelp (1.0.2)
  • Installing sphinxcontrib-devhelp (1.0.2)
  • Installing sphinxcontrib-htmlhelp (1.0.3)
  • Installing sphinxcontrib-jsmath (1.0.1)
  • Installing sphinxcontrib-qthelp (1.0.3)
  • Installing sphinxcontrib-serializinghtml (1.1.4)
  • Installing typing-extensions (3.7.4.3)
  • Installing yarl (1.6.3)
  • Installing aiohttp (3.7.3)
  • Installing sphinx (3.4.3)

root@7a29451bf5eb:/# poetry show --outdated
chardet  3.0.4  4.0.0  Universal encoding detector for Python 2 and 3
requests 2.15.1 2.25.1 Python HTTP for Humans.

root@7a29451bf5eb:/# poetry show --tree
aiohttp 3.7.3 Async http client/server framework (asyncio)
├── async-timeout >=3.0,<4.0
├── attrs >=17.3.0
├── chardet >=2.0,<4.0
├── multidict >=4.5,<7.0
├── typing-extensions >=3.6.5
└── yarl >=1.0,<2.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

root@7a29451bf5eb:/# poetry update
Updating dependencies
Resolving dependencies... (1.4s)

Writing lock file

Package operations: 2 installs, 2 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Updating idna (3.1 -> 2.10)
  • Installing urllib3 (1.26.2)
  • Updating requests (2.15.1 -> 2.25.1)

root@7a29451bf5eb:/# poetry show --outdated
chardet 3.0.4 4.0.0 Universal encoding detector for Python 2 and 3
idna    2.10  3.1   Internationalized Domain Names in Applications (IDNA)

root@7a29451bf5eb:/# poetry show --tree
aiohttp 3.7.3 Async http client/server framework (asyncio)
├── async-timeout >=3.0,<4.0
├── attrs >=17.3.0
├── chardet >=2.0,<4.0
├── multidict >=4.5,<7.0
├── typing-extensions >=3.6.5
└── yarl >=1.0,<2.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
│   ├── certifi >=2017.4.17
│   ├── chardet >=3.0.2,<5
│   ├── idna >=2.5,<3
│   └── urllib3 >=1.21.1,<1.27
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

root@7a29451bf5eb:/# poetry update
Updating dependencies
Resolving dependencies... (0.3s)

No dependencies to install or update

root@7a29451bf5eb:/# poetry show --outdated
chardet 3.0.4 4.0.0 Universal encoding detector for Python 2 and 3
idna    2.10  3.1   Internationalized Domain Names in Applications (IDNA)

root@7a29451bf5eb:/# poetry show --tree
aiohttp 3.7.3 Async http client/server framework (asyncio)
├── async-timeout >=3.0,<4.0
├── attrs >=17.3.0
├── chardet >=2.0,<4.0
├── multidict >=4.5,<7.0
├── typing-extensions >=3.6.5
└── yarl >=1.0,<2.0
    ├── idna >=2.0
    └── multidict >=4.0
sphinx 3.4.3 Python documentation generator
├── alabaster >=0.7,<0.8
├── babel >=1.3
│   └── pytz >=2015.7
├── colorama >=0.3.5
├── docutils >=0.12
├── imagesize *
├── jinja2 >=2.3
│   └── markupsafe >=0.23
├── packaging *
│   └── pyparsing >=2.0.2
├── pygments >=2.0
├── requests >=2.5.0
│   ├── certifi >=2017.4.17
│   ├── chardet >=3.0.2,<5
│   ├── idna >=2.5,<3
│   └── urllib3 >=1.21.1,<1.27
├── snowballstemmer >=1.1
├── sphinxcontrib-applehelp *
├── sphinxcontrib-devhelp *
├── sphinxcontrib-htmlhelp *
├── sphinxcontrib-jsmath *
├── sphinxcontrib-qthelp *
└── sphinxcontrib-serializinghtml *

[TBBle]

I think I just hit this bug, and have another repro/symptom, after observing it in a larger project.

Given pyproject.toml created by poetry init and testing with Poetry 1.1.5:

[tool.poetry]
name = "poetrybug"
version = "0.1.0"
description = ""
authors = ["Paul \"Hampy\" Hampson <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
docker = "^4.4.4"
aiohttp = "^3.7.4"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Starting without a poetry.lock, both poetry lock and poetry update --lock give the same resolution:

>poetry show --latest
aiohttp           3.7.4.post0 3.7.4.post0 Async http client/server framework (asyncio)
async-timeout     3.0.1       3.0.1       Timeout context manager for asyncio programs
attrs             20.3.0      20.3.0      Classes Without Boilerplate
chardet           4.0.0       4.0.0       Universal encoding detector for Python 2 and 3
docker            4.4.4       4.4.4       A Python library for the Docker Engine API.
idna              3.1         3.1         Internationalized Domain Names in Applications (IDNA)
multidict         5.1.0       5.1.0       multidict implementation
pywin32           227         300         Python for Window Extensions
requests          2.15.1      2.25.1      Python HTTP for Humans.
six               1.15.0      1.15.0      Python 2 and 3 compatibility utilities
typing-extensions 3.7.4.3     3.7.4.3     Backported and Experimental Type Hints for Python 3.5+
websocket-client  0.58.0      0.58.0      WebSocket client for Python with low level API options
yarl              1.6.3       1.6.3       Yet another URL library

Where requests has been chose as 2.15.1, pywin32 is 227, and everything else is latest.

poetry lock repeatedly won't change this state.

However, poetry update --lock now does change the status:

> poetry show --latest
aiohttp           3.7.4.post0 3.7.4.post0 Async http client/server framework (asyncio)
async-timeout     3.0.1       3.0.1       Timeout context manager for asyncio programs
attrs             20.3.0      20.3.0      Classes Without Boilerplate
certifi           2020.12.5   2020.12.5   Python package for providing Mozilla's CA Bundle.
chardet           4.0.0       4.0.0       Universal encoding detector for Python 2 and 3
docker            4.4.4       4.4.4       A Python library for the Docker Engine API.
idna              2.10        3.1         Internationalized Domain Names in Applications (IDNA)
multidict         5.1.0       5.1.0       multidict implementation
pywin32           227         300         Python for Window Extensions
requests          2.25.1      2.25.1      Python HTTP for Humans.
six               1.15.0      1.15.0      Python 2 and 3 compatibility utilities
typing-extensions 3.7.4.3     3.7.4.3     Backported and Experimental Type Hints for Python 3.5+
urllib3           1.26.4      1.26.4      HTTP library with thread-safe connection pooling, file post, and more.
websocket-client  0.58.0      0.58.0      WebSocket client for Python with low level API options
yarl              1.6.3       1.6.3       Yet another URL library

Rerunning poetry update --lock doesn't change further. Running poetry lock will change it back to where it started.

It seems that poetry update has the same effect, I was just using --lock to speed up the tests by not actually modifying the venv.

---

The full flip-flop, including venv updates

> poetry lock
Updating dependencies
Resolving dependencies... (0.9s)

Writing lock file
> poetry install --remove-untracked
Installing dependencies from lock file

Package operations: 0 installs, 2 updates, 2 removals

  • Removing certifi (2020.12.5)
  • Removing urllib3 (1.26.4)
  • Updating idna (2.10 -> 3.1)
  • Updating requests (2.25.1 -> 2.15.1)
> poetry update
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 2 installs, 2 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Updating idna (3.1 -> 2.10)
  • Installing urllib3 (1.26.4)
  • Updating requests (2.15.1 -> 2.25.1)
> poetry install --remove-untracked
Installing dependencies from lock file

No dependencies to install or update

---

Specifically, this is the same 'requests' flip-flop seen in the original bug report.

My suspicion originally fell on docker, because it has requests >=2.14.2,<2.18.0 || >2.18.0 in poetry show --tree, but that turns out to be requests >= 2.14.2, != 2.18.0. And it didn't reproduce until I added aiohttp to the mix.

Since the original ticket here is showing it with aiohttp + Sphinx, I assume it's related to something in aiohttp, or when aiohttp is used in combination with something that pulls in requests.

---

My observation is that poetry lock and poetry install (and run-without lock poetry update) all agree from-scratch on the resolution of latest idna but older requests, but poetry update really wants to upgrade requests at the cost of downgrading idna where poetry lock still prefers latest idna with older requests.

[TBBle]

An even closer-to-the-bone repro:

[tool.poetry]
name = "poetrybug"
version = "0.1.0"
description = ""
authors = ["Paul \"Hampy\" Hampson <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
idna = "*"
requests = "*"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

> poetry update --lock
Creating virtualenv poetrybug2-w-Z-e8D0-py3.9 in C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs
Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file
  • Installing idna (3.1): Pending...
  • Installing idna (3.1)
  • Installing requests (2.15.1): Pending...
  • Installing requests (2.15.1)
> poetry update --lock
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file
  • Installing certifi (2020.12.5): Pending...
  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0): Pending...
  • Installing chardet (4.0.0)
  • Installing idna (2.10): Pending...
  • Installing idna (2.10)
  • Installing urllib3 (1.26.4): Pending...
  • Installing urllib3 (1.26.4)
  • Installing requests (2.25.1): Pending...
  • Installing requests (2.25.1)

So it reproduces simply with idna and requests.

The first release after requests 2.15.1 was 2.16.0, with the changelog

Unvendor ALL the things!

requests 2.15.1 had vendored idna and a few other things, but also had a security extra: 'pyOpenSSL>=0.14', 'cryptography>=1.3.4', 'idna>=2.0.0'

requests 2.25.1 (latest) now requires idna>=2.5,<3, i.e. it conflicts with latest idna 3.10. The other packages it no longer vendors are acceptable at the latest version, so they won't trigger this conflict, just get dragged in or out depending on the version of requests chosen.

So there are two valid resolutions here, the problem is just the weighting used by poetry upgrade seems to be different for the same pyproject.toml if poetry.lock is present.

[dimbleby]

duplicate #2883

[Secrus]

Closing as duplicate

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.