poetry install fails when installing dependencies, from git repositories, in gitlab CI.

[pawelrubin]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: python3.8 docker image
• Poetry version: 1.0.5 (installed via pip3 install poetry)
• Link of a Gist with the contents of your pyproject.toml file: link

Issue

poetry install fails when installing dependencies, from git repositories, in gitlab CI.

It does work on my local machine. Dependency has been added via poetry add git+https://github.com/devopshq/artifactory.git@support-python-38-glob
Traceback from poetry install -vvv:

Installing dependencies from lock file

[CalledProcessError]
Command '['git', '--git-dir', '/tmp/pypoetry-git-artifactory60aj4enu/.git', '--work-tree', '/tmp/pypoetry-git-artifactory60aj4enu', 'checkout', 'support-python-38-glob']' returned non-zero exit status 1.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/usr/local/lib/python3.8/site-packages/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/usr/local/lib/python3.8/site-packages/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 225, in _do_install
    ops = solver.solve(use_latest=whitelist)
  File "/usr/local/lib/python3.8/site-packages/poetry/puzzle/solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "/usr/local/lib/python3.8/site-packages/poetry/puzzle/solver.py", line 180, in _solve
    result = resolve_version(
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/version_solver.py", line 80, in solve
    next = self._choose_package_version()
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/version_solver.py", line 355, in _choose_package_version
    packages = self._provider.search_for(dependency)
  File "/usr/local/lib/python3.8/site-packages/poetry/puzzle/provider.py", line 130, in search_for
    packages = self.search_for_vcs(dependency)
  File "/usr/local/lib/python3.8/site-packages/poetry/puzzle/provider.py", line 167, in search_for_vcs
    package = self.get_package_from_vcs(
  File "/usr/local/lib/python3.8/site-packages/poetry/puzzle/provider.py", line 198, in get_package_from_vcs
    git.checkout(reference, tmp_dir)
  File "/usr/local/lib/python3.8/site-packages/poetry/vcs/git.py", line 217, in checkout
    return self.run(*args)
  File "/usr/local/lib/python3.8/site-packages/poetry/vcs/git.py", line 284, in run
    subprocess.check_output(["git"] + list(args), stderr=subprocess.STDOUT)
  File "/usr/local/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.8/site-packages/poetry/utils/_compat.py", line 205, in run
    raise CalledProcessError(

[finswimmer]

Hello @pawelrubin,

it looks a bit like git cannot find the branch. Can you try to clone the repository and checkout the branch in the target machine only with git? At best, copy&paste the branch name to avoid oversee any minor typo.

fin swimmer

[balan-jayavictor]

The target here is a docker container (based on the image we provide) provisioned by Gitlab during the build time.
It works fine when running the docker container locally.
@pawelrubin did you manage to fix it / work around it?

[guhcampos]

I believe this problem is related to something I've experienced.

It looks like poetry will clone the repository with it's default name during poetry add but will try to infer the local repository name from the package name when doing anything else, so in my case I had something like:

poetry add git+ssh://[email protected]:/mycompany-sdk.git

which cloned such repository under $HOME/.cache/pypoetry/virtualenvs/myproject-pbdKQ0bZ-py3.6/src/mycompany-sdk/

However, during a subsequent poetry update it failed with:

[CalledProcessError]
Command '['git', '--git-dir', '$HOME.cache/pypoetry/virtualenvs/myproject-pbdKQ0bZ-py3.6/src/mycompany/.git', '--work-tree', '$HOME/.cache/pypoetry/virtualenvs/myproject-pbdKQ0bZ-py3.6/src/mycompany', 'rev-parse', 'HEAD^{commit}']' returned non-zero exit status 128.

Which makes total sense because there's no mycompany directory there, the repository is called mycompany-sdk, yet the package it contains is, indeed, named mycompany and that's why poetry thinks it should look for that package there.

[johnmee]

There does not appear to be any rush to fix this, so I guess there is a workaround. I wonder what that might be?

Perhaps delete the virtualenv and start over?

rm -rf ~/.cache/pypoetry/virtualenvs/my-project-name-37cdnjD8-py3.8/
poetry install

[floi]

I had a similar issue with a dependency from a private Gitlab repository that used SSH for cloning (same CalledProcessError). When testing in a local Docker container, I noticed that I had to actively confirm the server fingerprint. As git has no option for automatically doing that, I solved poetry's installation of dependencies in the Gitlab CI container by adding the fingerprint to ~/.ssh/known_hosts before running poetry install in my .gitlab-ci.yml. Here's how I did that:

1. Manually clone my dependency's repository in a clean docker container
2. In that docker container, get the two lines (not sure why there's more than one) from ~/.ssh/known_hosts
3. In my .gitlab-ci.yml, add the following lines before calling poetry install:

- mkdir ~/.ssh
- echo "FIRST LINE COPIED IN STEP 2" >> ~/.ssh/known_hosts
- echo "SECOND LINE COPIED IN STEP 2" >> ~/.ssh/known_hosts 

I'm aware that the problem in the original comment might be something else (given that you, @pawelrubin, were trying to clone via HTTPS from a public repository), but maybe this will help other people ending up here when researching that error message.

[bolbken]

3. In my .gitlab-ci.yml, add the following lines before calling poetry install:

- mkdir ~/.ssh
- echo "FIRST LINE COPIED IN STEP 2" >> ~/.ssh/known_hosts
- echo "SECOND LINE COPIED IN STEP 2" >> ~/.ssh/known_hosts 

I know this is old... but a cleaner alternative that I found for github, guessing gitlab is similar

ssh-keyscan github.com >> ~/.ssh/known_hosts

[joooeey]

I had a similar issue and it turns out the problem was that the library I was referring to (sentinelsat) had no master branch because they renamed their main branch to main. Since a lot of repositories have done that in the last few years, that might be the problem.

Adding the branch requirement in pyproject.toml worked for me:

sentinelsat = { git = "https://github.com/sentinelsat/sentinelsat.git", branch = "main" }

This problem already has an issue at #3366.

[mecampbellsoup]

Experiencing this issue w/ a public GitHub repo.

Here is my Poetry config:

[package.source]
type = "git"
url = "ssh://[email protected]/agronholm/anyio.git"
reference = "4419414e25d17c456bdd3e5216f152dba27810de"
resolved_reference = "4419414e25d17c456bdd3e5216f152dba27810de"

And here at the GitLab CI Logs:

  CalledProcessError
  Command '['git', 'clone', '--recurse-submodules', '--', 'ssh://[email protected]/agronholm/anyio.git', '/usr/local/src/anyio']' returned non-zero exit status 128.
  at /usr/local/lib/python3.9/site-packages/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.
The command '/bin/sh -c poetry install --no-dev' returned a non-zero code: 1
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Anyone aware of a workaround?

[mrcljx]

In case it helps anyone: I had the same "returned non-zero exit status 128" issue. I investigated it by just running this:

git clone --recurse-submodules -- SOME_REPO /usr/local/src/PACKAGE_NAME

It turned out that /usr/local/src did not exist / wasn't writable, so totally a problem on my end. It would really be nice if Poetry could log the stderr of Git to make debugging situations like this easier.

For those wondering: The path is coming from the pip_installer which uses sys.prefix / "src"

[luni3359]

It turned out that /usr/local/src did not exist / wasn't writable, so totally a problem on my end.

@mrcljx Do you think you could explain what you changed? I tried the following:

RUN useradd --create-home 1000
ENV PATH="/home/1000/.local/bin:${PATH}"
WORKDIR /usr/local/src/myproject
RUN chown -R 1000:1000 /usr/local/src/myproject
RUN chmod 755 /usr/local/src/myproject
RUN chmod 755 /usr/local/src/
USER 1000

but it's still giving me the same error:

CalledProcessError

  Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/Rapptz/discord.py', '/usr/local/src/discord.py']' returned non-zero exit status 128.

  at ~/.local/lib/python3.8/site-packages/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

This dependency is written in pyproject.toml as follows:

[tool.poetry.dependencies]
... (other dependencies)
"discord.py" = {git = "https://github.com/Rapptz/discord.py", extras = ["voice"]}

The container is created without an issue if I run everything as root. Are you running Poetry as root?

[phenry2]

Experiencing exactly the same issue with 2 out of 4 github repositories. What the two that are failing have in common is that they both have submodules. Turns out I'd made a change to the way the submodules were being referenced.

I was able to identify the issue by manually adding git clone --recurse-submodules -- https://[email protected]/xxx/xxx.git /usr/local/src/xxx/xxx into the pipeline to see what the real error was. Turns out the submodules were using the url https://github.com, whereas their parents were accessed through https://[email protected]. I'd applied the following to ensure the PAT was being set for [email protected];

git config --global url.https://[email protected]/.insteadOf https://[email protected]/

This was not applying to the submodule and I got an authentication error when it tried to clone the submodule. I solved this by just updating the url to the submodules in .gitmodules.

This may not be the same issue as exit status code 1 is rather uninformative, but adding the command manually to your pipeline should reveal the true cause.

[ryo-matsuzaka]

@phenry2
Thank you for timely comments.
I had just the similar issue. I could solved it.

Experiencing exactly the same issue with 2 out of 4 github repositories. What the two that are failing have in common is that they both have submodules. Turns out I'd made a change to the way the submodules were being referenced.

I was able to identify the issue by manually adding git clone --recurse-submodules -- https://[email protected]/xxx/xxx.git /usr/local/src/xxx/xxx into the pipeline to see what the real error was. Turns out the submodules were using the url https://github.com, whereas their parents were accessed through https://[email protected]. I'd applied the following to ensure the PAT was being set for [email protected];

git config --global url.https://[email protected]/.insteadOf https://[email protected]/

This was not applying to the submodule and I got an authentication error when it tried to clone the submodule. I solved this by just updating the url to the submodules in .gitmodules.

This may not be the same issue as exit status code 1 is rather uninformative, but adding the command manually to your pipeline should reveal the true cause.

[TinDang97]

I just add #main at the end. It's worked. :D

[abn]

Sounds like ^ was a default branch issue. #5428 should handle this cleaner. Would be great to get that changes tested by folks who have been having issues here.

[abn]

Resolved-by: #5428

[simon-liebehenschel]

This fails with the latest Poetry:

pydantic = {git = "https://github.com/samuelcolvin/pydantic.git"}

Commands:

curl -sSL https://install.python-poetry.org | python3 -
poetry install --no-interaction --no-ansi --no-root --no-dev

Traceback:

#9 103.8   CalledProcessError
#9 103.8 
#9 103.8   Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/samuelcolvin/pydantic.git', '/usr/local/src/pydantic']' returned non-zero exit status 128.
#9 103.8 
#9 103.8   at /opt/poetry/venv/lib/python3.10/site-packages/poetry/utils/_compat.py:217 in run
#9 103.9       213│                 process.wait()
#9 103.9       214│                 raise
#9 103.9       215│             retcode = process.poll()
#9 103.9       216│             if check and retcode:
#9 103.9     → 217│                 raise CalledProcessError(
#9 103.9       218│                     retcode, process.args, output=stdout, stderr=stderr
#9 103.9       219│                 )
#9 103.9       220│         finally:
#9 103.9       221│             # None because our context manager __exit__ does not use them.
#9 103.9 
#9 ERROR: executor failed running [/bin/bash -o pipefail -c poetry install --no-interaction --no-ansi --no-root --no-dev]: exit code: 1

[mkniewallner]

#5428 was merged against master branch, which corresponds to version 1.2 of Poetry, which is still in beta.

Could you try on latest beta?

curl -sSL https://install.python-poetry.org | python3 - --preview

Otherwise, for 1.1, #2475 (comment) may also fix the issue, as mentioned.

[janbaykara]

Tried upgrading to the preview version 1.2.0b3 and now getting another error. Neither the 1.1 comment nor upgrading has worked for poetry add git+https://github.com/commonknowledge/wagtail-localize.git#main

Command ['/usr/local/bin/python3.9', '/usr/local/venv/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-22.2.2-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--prefix', '/usr/local', '--no-deps', '/home/vscode/.cache/pypoetry/artifacts/08/9f/47/c16ae03035fc69eaf100ea39657a49baaeef714e25a52575710c34cd48/six-1.16.0-py2.py3-none-any.whl'] errored with the following
      1459│ 
  • Installing six (1.16.0): Failed

  CalledProcessError

  Command '['/usr/local/bin/python3.9', '/usr/local/venv/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-22.2.2-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--prefix', '/usr/local', '--no-deps', '/home/vscode/.cache/pypoetry/artifacts/08/9f/47/c16ae03035fc69eaf100ea39657a49baaeef714e25a52575710c34cd48/six-1.16.0-py2.py3-none-any.whl']' returned non-zero exit status 1.

  at /usr/local/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/usr/local/bin/python3.9'

[isosphere]

I had a similar issue and it turns out the problem was that the library I was referring to (sentinelsat) had no master branch because they renamed their main branch to main. Since a lot of repositories have done that in the last few years, that might be the problem.

Adding the branch requirement in pyproject.toml worked for me:

sentinelsat = { git = "https://github.com/sentinelsat/sentinelsat.git", branch = "main" }

This problem already has an issue at #3366.

This set me on the right track. It wasn't at all clear to me that there was a problem fetching a particular dependency via git from the error. If I had been told more explicitly "failed to get bla from git @ address" by poetry I might have been able to handle this on my own.

[tkaessmann]

In case it helps anyone: I had the same "returned non-zero exit status 128" issue. I investigated it by just running this:

git clone --recurse-submodules -- SOME_REPO /usr/local/src/PACKAGE_NAME

It turned out that /usr/local/src did not exist / wasn't writable, so totally a problem on my end. It would really be nice if Poetry could log the stderr of Git to make debugging situations like this easier.

For those wondering: The path is coming from the pip_installer which uses sys.prefix / "src"

Thx so much for your help. Same problem here. But we've fixed it by setting:
poetry config virtualenvs.create true && poetry config virtualenvs.in-project true
In my understanding the clone then will put the repo content inside the virtualenv instead of trying to place it into /usr/...

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.