PIP install few orders of magnitude slower in PIP 20.3.2@master than 20.2.4

[potiuk]

I tried installing airflow 2.0.0rc3 with the new master version of PIP (to be 20.3.2).

When I run the same installation using 20.2.4 the whole install takes about 20 seconds when I already pre-installed some of the prerequisites. With PIP 20.3.2@master it takes about 30 minutes (and fails but this is a separate issue).

The whole installation log is attached.
pip-20-3-2-master.txt

Installation environment:

Latest official Python 3.6 image with airflow dependencies (based on debian buster):

docker pull apache/airflow:master-python3.6-ci

Installation method:

pip install apache-airflow[devel_ci]==2.0.0rc3 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt

The easiest way to reproduce:

git clone https://github.com/apache/airflow.git
./breeze --install-airflow-version 2.0.0rc3 --skip-mounting-local-sources
# While in the container

# uninstall airflow and all 60 providers
pip uninstall apache-airflow -y
pip freeze  |grep airflow-providers |xargs pip uninstall -y

# Install Pip from master
# NOTE! Skip this step if you want to compare 2.20.4 - this is the default version we have in the image)
pip install git+https://github.com/pypa/pip@master

# install airflow in recommended way
pip install apache-airflow[devel_ci]==2.0.0rc3 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt

Related issues #9299 #9297

[pradyunsg]

Seems to be hitting #9284. Closing as a duplicate of that.

[potiuk]

Can you please re-open @pradyunsg ?

This is not duplicate of #9284. Downloading multiple versions of setuptols/wheels is separate issue #9297. Even if exclude the time needed to download multiple versions of setuptools and wheels the whole installation takes 25 minutes or so vs. 20 seconds with PIP 20.2.4. This is totally different issue.

[uranusjr]

This is not #9284, but a better worded and more specific variant of #9187.

[potiuk]

This is not #9284, but a better worded and more specific variant of #9187.

And just for the record - seems that adding the Pyarrow==2.0.0 (see #9299) changed completely the behaviour of PIP (or maybe it was something merged in master in the meantime??). It looks like not only the setuptools/wheel #9297 stopped happening but also It started to run pretty much as fast as the old PIP 20.2.4. I am waiting to see the 20.3.3 to see if this really is not a mistake somewhere ....