poetry 1.1.x causes SSLCertVerificationError when installing packages from private repo

[zyxue]

I have a poetry project, configured with a private pypi repo, that used to work well, but after upgrading poetry to 1.1.x (tried 1.1.0 to 1.1.2), it causes error like

#16 26.50 Retrying HTTP request in 0.5 seconds.
#16 27.13 Retrying HTTP request in 1.0 seconds.
#16 28.25 Retrying HTTP request in 1.5 seconds.
#16 29.87 Retrying HTTP request in 2.0 seconds.
#16 32.00 Retrying HTTP request in 2.5 seconds.
#16 34.71 
#16 34.71   SSLError
#16 34.71 
#16 34.71   HTTPSConnectionPool(host='<PRIVATE-REPO-HOST>)', port=443): Max retries exceeded with url: <URL-to-myapp-0.1.3-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)')))
#16 34.71 
#16 34.71   at /usr/local/lib/python3.8/site-packages/requests/adapters.py:514 in send
#16 34.76       510│                 raise ProxyError(e, request=request)
#16 34.76       511│ 
#16 34.76       512│             if isinstance(e.reason, _SSLError):
#16 34.76       513│                 # This branch is for urllib3 v1.22 and later.
#16 34.76     → 514│                 raise SSLError(e, request=request)
#16 34.76       515│ 
#16 34.76       516│             raise ConnectionError(e, request=request)
#16 34.76       517│ 
#16 34.76       518│         except ClosedPoolError as e:
#16 34.76 
#16 ERROR: executor failed running [/bin/sh -c poetry install --no-dev]: runc did not terminate sucessfully

when downloading packages that are only available in the private repo.

When I downgrade poetry to 1.0.10, it then works fine.

[abn]

@zyxue can you try setting REQUESTS_CA_BUNDLE?

[neilpanchal]

Is it possible to ignore SSL certs in poetry? We have an internal private pypi repository hosted without SSL certs.

[zyxue]

@zyxue can you try setting REQUESTS_CA_BUNDLE?

tried REQUESTS_CA_BUNDLE=/app/certs.pem (it's used when building Docker image)

but it doesn't seem to be working, still the same error.

[zyxue]

@abn could you please point me to the place in the code how certs file is used differently between poetry 1.0.x and 1.1.x?

[absassi]

I also have the same problem. The funny thing is that when doing poetry add my-private-package it is able to find the latest version of that package and resolve its dependencies (so certificates.my-private-repo.cert config is used there), but then fails when trying to actually install it.

Setting REQUESTS_CA_BUNDLE (or CURL_CA_BUNDLE, as requests check for them both) fixes it for me.

[zyxue]

The funny thing is that when doing poetry add my-private-package it is able to find the latest version of that package and resolve its dependencies (so certificates.my-private-repo.cert config is used there), but then fails when trying to actually install it.
@absassi , I'm seeing the same behavior. the version is resolved, but it can't install.

Do I understand correctly that you set certificates.my-private-repo.cert and REQUESTS_CA_BUNDLE to the same value (i.e. path to a pem file) and then it works?

[irremotus]

I am seeing the same issue, and can confirm that certificates.my-private-repo.cert and REQUESTS_CA_BUNDLE are set to the same value.

[zyxue]

@irremotus , do you mean it's still not working for you even with "certificates.my-private-repo.cert and REQUESTS_CA_BUNDLE are set to the same value"?

It's not working for me.

[absassi]

To make it clear: I've set both certificates.my-private-repo.cert (where my-private-repo is the name of the repository as defined in pyproject.toml and in Poetry config) and REQUESTS_CA_BUNDLE and it works for me (it should probably work too by setting only REQUESTS_CA_BUNDLE, but I haven't checked).

[irremotus]

@zyxue When I set REQUESTS_CA_BUNDLE, both dependency resolution and package installation work. When I set certificates.my-private-repo.cert (and not REQUESTS_CA_BUNDLE), dependency resolution works, but package installation fails, as @absassi mentioned. This worked in previous versions (e.g. 1.0.0), but is failing in recent versions (e.g. 1.1.[1-4]).

[zyxue]

I haven't had a problem with before poetry 1.1.0. Currently, I explicitly set the poetry version to avoid break with python -m pip install poetry==1.0.10, but still curious why this is happening.

I tried with poetry 1.1.4 again, here is what I experienced:

poetry config certificates. .cert	set REQUESTS_CA_BUNDLE	note
yes	yes	get [SSL: CERTIFICATE_VERIFY_FAILED] before dependency resolution
yes	no	dependency resolution succeeds, but get [SSL: CERTIFICATE_VERIFY_FAILED] during installation
no	yes	get [SSL: CERTIFICATE_VERIFY_FAILED] before dependency resolution
no	no	get [SSL: CERTIFICATE_VERIFY_FAILED] before dependency resolution

[absassi]

@zyxue, since dependency resolution only works when you have the file in Poetry config and the variable is unset, it means that whatever you are setting in the variable is causing the validation to fail (the variable takes precedence over the configuration). Please double check the path for any typos or maybe wrong use of quotes.

[zyxue]

what I have done is basically

poetry config certificates.mypypi.cert /app/cacerts.pem
export REQUESTS_CA_BUNDLE=/app/cacerts.pem

am I missing something?

[absassi]

@zyxue that looks correct to me. Are you using a private repo in addition to PyPI (i.e. no default = true in the source in pyproject.toml)? If so, do you have public CA certificates in this bundle or only the private CAs? The only thing I could think of is that Poetry is trying to get packages from both sources and your bundle have only the certificate for the private one, so when you set the variable, it works for the private repo, but starts to fail when accessing pypi.org.

[zyxue]

Are you using a private repo in addition to PyPI (i.e. no default = true in the source in pyproject.toml)?

No, I was not using default=true.

If so, do you have public CA certificates in this bundle or only the private CAs?

From your reasoning, I feel I'm only having private CAs. (I'm not too familiar with how CA works, and the difference between public and private)

After adding default=true, I'm seeing error like

#17 14.97   • Installing pygments (2.6.1)
#17 14.97   • Installing rsa (4.6)
#17 14.97   • Updating urllib3 (1.25.11 -> 1.25.10)
#17 16.19 
#17 16.19   RuntimeError
#17 16.19 
#17 16.19   Unable to find installation candidates for nest-asyncio (1.4.0)
#17 16.19 
#17 16.19   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:72 in choose_for
#17 16.22        68│ 
#17 16.22        69│             links.append(link)
#17 16.22        70│ 
#17 16.22        71│         if not links:
#17 16.22     →  72│             raise RuntimeError(
#17 16.22        73│                 "Unable to find installation candidates for {}".format(package)
#17 16.22        74│             )
#17 16.22        75│ 
#17 16.22        76│         # Get the best link
#17 16.22 
#17 16.22 
#17 16.22   RuntimeError
#17 16.22 
#17 16.22   Unable to find installation candidates for pygments (2.6.1)
#17 16.22 
#17 16.22   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:72 in choose_for
#17 16.24        68│ 
#17 16.24        69│             links.append(link)
#17 16.24        70│ 
#17 16.24        71│         if not links:
#17 16.24     →  72│             raise RuntimeError(
#17 16.24        73│                 "Unable to find installation candidates for {}".format(package)
#17 16.24        74│             )
#17 16.24        75│ 
#17 16.24        76│         # Get the best link
#17 16.24 
#17 16.24 
#17 16.24   RuntimeError

It's kind of confusing why the behavior changed from poetry 1.0.x?

[absassi]

@zyxue, I think we found the problem then (because I can't think of anything else). so what happens is that you need to use the private source only for private packages, and you still need PyPI for public packages. So, if you use default=true, it doesn't work, because it stops looking for packages in PyPI and if you don't use it, then it looks into both repositories, but you don't have a CA bundle that can validate both servers.

I think there's three ways you can fix this:

• Download the open source wheels you need from PyPI and upload them to your private repository. Or configure your private repository to also mirror PyPI, if it has this feature (I know Artifactory can do that).
• Include the CAs required to validate PyPI servers in your bundle. If you are on Linux, you can just cat the system certificates and your bundle together into a single file. If you are on Windows, then you can use the Mozilla CA bundle (you can download a copy from curl website), as Windows doesn't have any bundle in that format that you can use directly.
• Downgrade to Poetry 1.0.10 or 1.0.9, if possible.

As for why this changed, this is simply a bug, as far as I can tell, and hopefully will be fixed soon.

[zyxue]

Thank you for the detailed explanation, @absassi !

Regarding the first solution, based on the log in my last comment, do you have any insight on why it does not have trouble till updating urllib3. In other words, it seems fine downloading pygments and rsa, which I thought are also from the public PyPI?

[absassi]

@zyxue, I can't say for sure. Maybe they are locally cached and urllib3 is not, or maybe you have pygments and rsa in your private repo?

[zyxue]

@absassi, thank you for your reply! I see, I'll stick to Poetry 1.0.10 till this issue is fixed.

[wjhrdy]

Workaround on macOS

Get selfsigned cert

From within your project directory that contains pyproject.toml

Download your selfsigned cert or just make sure it is in the project directory

curl -O https://examplepathtocacert.com/selfsigned.pem

Disable the new installer

WARNING:

you can no longer revert to the old installer in 1.4.0 as the old installer is depricated
#7358

If you don't want to build your own REQUESTS_CA_BUNDLE it looks like disabling the new installer from 1.1.x will remove the need for all the REQUESTS_CA_BUNDLE building. You still have to do the tool.poetry.source and config certificates steps though.

poetry config experimental.new-installer false

Or build your own REQUESTS_CA_BUNDLE

since macs don't come with a requests compatible cacert bundle we download one. This part is potentially dangerous because it could lead to your requests package trusting a certificate from a malicious site, however I do trust this particular site. There could be better ways to reference a trusted cacert bundle that is already on your machine I just don't know where that exists setup-agnostic.

cd $HOME && { curl -O https://curl.se/ca/cacert.pem ; cd -; }

The REQUESTS_CA_BUNDLE environment variable wants the full set of certs not just the one you are adding for this repo so we add our selfsigned.pem to the bottom of the cacert.pem bundle we just downloaded.

cat selfsigned.pem >> $HOME/cacert.pem

This temporarily sets REQUESTS_CA_BUNDLE so if you want it to be set permanently add this following line to ~/.zshrc

export REQUESTS_CA_BUNDLE=$HOME/cacert.pem

Setup pypi repo

This adds your private pypi repo to your project config

echo '\n[[tool.poetry.source]]\nname = "self-signed-pypi"\nurl = "https://somepypi.com/simple/"' >> pyproject.toml

This makes sure the selfsigned cert is configured for the repo.

poetry config certificates.self-signed-pypi.cert ./selfsigned.pem

Then you can add your package.

poetry add your-package-from-self-signed-pypi

[gazpachoking]

This is still an issue on 1.2.0b3

[neersighted]

The last set of steps listed by @wjhrdy are correct for a private repo with a self-signed certificate, and should work properly on 1.2.x. If you're still running into issues using them, please open a new issue with a detailed reproduction.

If you're trying to replace the certificate for PyPI/files.pythonhosted.org, you'll want to see #1012 -- in short, REQUESTS_CA_BUNDLE works but is not the most ergonomic experience. If you're trying to replace PyPI with another URL/index globally (and are unwilling to set a custom source with default = true), that is tracked at #1632.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.