Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using pypi mirror sites to speed up the dependency resolving. #3261

Closed
2 tasks done
qiuwei opened this issue Oct 21, 2020 · 3 comments
Closed
2 tasks done

Using pypi mirror sites to speed up the dependency resolving. #3261

qiuwei opened this issue Oct 21, 2020 · 3 comments
Labels
kind/feature Feature requests/implementations

Comments

@qiuwei
Copy link

qiuwei commented Oct 21, 2020

  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Feature Request

Poetry needs to inevitably download some packages(either in sdist or in wheel format) to inspect the dependencies during the dependency resolving phase.
However, due to some reasons, downloading from the official PyPI website is quite slow in my local environment.

Is it possible to utilize a mirror site for the downloading?

I have experimented with a very simple solution.
I add the following function to poetry/repositories/pypi_repository.py, and call it at the very beginning of the function of _get_info_from_wheel and _get_info_from_sdist.

    def _get_mirror_url(self, url):
        """
        url is normally in the format of https://files.pythonhosted.org/packages/<blahblah>
        url of the same package on a mirroring site is
        :param url:
        :return:
        """
        import re
        ORIG_URL = "https://files.pythonhosted.org/"
        MIRROR_URL = "https://mirrors.cloud.tencent.com/pypi/"
        return re.sub(ORIG_URL, MIRROR_URL, url)

Basically, it replaces the default host(pythonhosted.org) with the PyPI mirror host in the URLs.
This dramatically speeds up the dependency resolving(5000+s -> 137s in my local environment).

Do you think it is worth a pull request to add this feature?
BTW, I am wondering where to put the option to let the users specify the mirror site URL for dependency resolving?

@qiuwei qiuwei added kind/feature Feature requests/implementations status/triage This issue needs to be triaged labels Oct 21, 2020
@peldszus
Copy link
Contributor

Have you checked the documentation on custom repositories?
https://python-poetry.org/docs/repositories/#repositories

@qiuwei
Copy link
Author

qiuwei commented Oct 27, 2020

Have you checked the documentation on custom repositories?
https://python-poetry.org/docs/repositories/#repositories

Yes, I was using custom repositories.
According to my observation, for packages that exist in both PyPI and custom repositories, poetry still downloads them from PyPI.
image

EDIT:
The above problem is fixed in the latest 1.1.4 release.

@qiuwei qiuwei closed this as completed Oct 27, 2020
@abn abn removed the status/triage This issue needs to be triaged label Mar 3, 2022
Copy link

github-actions bot commented Mar 2, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/feature Feature requests/implementations
Projects
None yet
Development

No branches or pull requests

3 participants