no-cache-dir is not respected for deps that require building

[orgads]

Description

I'm running on alpine:

apk add --no-cache --virtual .build-deps gcc musl-dev jpeg-dev zlib-dev libffi-dev cairo-dev pango-dev gdk-pixbuf-dev
pip install --no-cache-dir psutil

Expected behavior

There should be no cache at all

pip version

23.1.2

Python version

3.11.3

OS

alpine

How to Reproduce

See description.

Output

~ # pip install --no-cache-dir psutil
Collecting psutil
  Downloading psutil-5.9.5.tar.gz (493 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 493.5/493.5 kB 3.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.5-cp311-abi3-linux_x86_64.whl size=274372 sha256=982672ee0b70da28e06d9cb4470fb0daec32df0a394addd569cb1aed88d256bd
  Stored in directory: /tmp/pip-ephem-wheel-cache-xivhj57g/wheels/d0/3f/6a/c3f82d406cd630dc70b35f9b124c75bc2f4db2c6a32e3b2eb8
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.9.5

~ # du
8       ./.cache/pip/selfcheck
20      ./.cache/pip/http/4/d/2/7/2
...

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Try adding --disable-pip-version-check - the version check uses the same directory as the cache, as you can see.

Maybe --no-cache-dir could imply --disable-pip-version-check - PRs welcome.

[orgads]

@pfmoore No change.

[orgads]

I simplified the example. Install psutil alone reproduces the issue.

[pfmoore]

Did you delete the files in ./.cache/pip/? The selfcheck file should only be created or referenced by the self-check logic, so I'm confused how it would exist if you're adding --disable-pip-version-check.

If you add rm -r ./.cache/pip; du just before the pip invocation, what does it show? I don't know what apk does, so I'm not sure if that could be creating these files.

Otherwise, I'm out of ideas.

[orgads]

I'm running in docker, and did not run pip before. I also verified that there is no .cache directory before running pip.

[orgads]

apk is alpine's package manager. It installs the build dependencies for psutil.

[pfmoore]

I tried reproducing this.

> docker run --rm -it alpine sh
/ # apk add --no-cache --virtual .build-deps gcc musl-dev jpeg-dev zlib-dev libffi-dev cairo-dev pango-dev gdk-pixbuf-dev
# lots of stuff...
/ # ls -la
total 80
drwxr-xr-x    1 root     root          4096 May 15 16:05 .
drwxr-xr-x    1 root     root          4096 May 15 16:05 ..
-rwxr-xr-x    1 root     root             0 May 15 16:05 .dockerenv
drwxr-xr-x    2 root     root          4096 May  9 18:39 bin
drwxr-xr-x    5 root     root           360 May 15 16:05 dev
drwxr-xr-x    1 root     root          4096 May 15 16:06 etc
drwxr-xr-x    2 root     root          4096 May  9 18:39 home
drwxr-xr-x    1 root     root          4096 May 15 16:06 lib
drwxr-xr-x    5 root     root          4096 May  9 18:39 media
drwxr-xr-x    2 root     root          4096 May  9 18:39 mnt
drwxr-xr-x    2 root     root          4096 May  9 18:39 opt
dr-xr-xr-x  337 root     root             0 May 15 16:05 proc
drwx------    1 root     root          4096 May 15 16:06 root
drwxr-xr-x    2 root     root          4096 May  9 18:39 run
drwxr-xr-x    2 root     root          4096 May  9 18:39 sbin
drwxr-xr-x    2 root     root          4096 May  9 18:39 srv
dr-xr-xr-x   11 root     root             0 May 15 16:05 sys
drwxrwxrwt    2 root     root          4096 May  9 18:39 tmp
drwxr-xr-x    1 root     root          4096 May 15 16:06 usr
drwxr-xr-x    1 root     root          4096 May  9 18:39 var
/ # pip3 install --disable-pip-version-check --no-cache-dir psutil
Collecting psutil
  Downloading psutil-5.9.5.tar.gz (493 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 493.5/493.5 kB 4.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [47 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pswindows.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psposix.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psosx.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_compat.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/__init__.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pslinux.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_pssunos.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psbsd.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_common.py -> build/lib.linux-x86_64-cpython-311/psutil
      copying psutil/_psaix.py -> build/lib.linux-x86_64-cpython-311/psutil
      creating build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_process.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/__main__.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_system.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/runner.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/__init__.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-cpython-311/psutil/tests
      running build_ext
      building 'psutil._psutil_linux' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/psutil
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=595 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -I/usr/include/python3.11 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-cpython-311/psutil/_psutil_common.o
      psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory
          9 | #include <Python.h>
            |          ^~~~~~~~~~
      compilation terminated.
      psutil could not be installed from sources. Perhaps Python header files are not installed. Try running:
        sudo apk add gcc python3-dev
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for psutil
Failed to build psutil
ERROR: Could not build wheels for psutil, which is required to install pyproject.toml-based projects

OK, that's nothing like what you show. Please provide a reproducible test case starting from a vanilla docker image that I can use, and I'll try to reproduce.

Just in case, I tried running apk add gcc python3-dev as the error suggested. That allowed the install to work but I see no ./.cache directory.

[orgads]

The base image is python:3.11-alpine3.17

[pfmoore]

Quick correction - pip3 didn't work at first, I needed to do python3 -m ensurepip as well.

On doing some more digging, I did find a .cache directory in ~ (which is not where you said, but I should probably have checked there). I suspect it's coming from --disable-pip-version-check not being propagated to the pip subprocess that's used to populate the build environment.

And if you set the pip options via environment variables:

export PIP_DISABLE_PIP_VERSION_CHECK=1
export PIP_NO_CACHE_DIR=1

you will indeed not get a .cache directory.

Just as a gentle reminder, please don't assume any particular knowledge on the part of people trying to help with your issue. For example, I know almost nothing about Linux, I'm a Windows developer, and it took me a while to realise I needed to use the pip3 command rather than pip. Maybe if you'd included details on what container to use, I would have avoided that problem, but in general the more you can provide "assume nothing" reproduction instructions, the better. It's not a big issue in this case, and there was no harm done, but at one point I was close to just dropping the issue as I couldn't reproduce it, which would have been a shame.

Anyway, it looks like your solution is to set the options via the environment and that will do what you need. Hope that helps.

[orgads]

@pfmoore Thank you, I'll try to provide complete reproduction steps next time.

Regarding you solution, looks like PIP_DISABLE_PIP_VERSION_CHECK is not needed, setting PIP_NO_CACHE_DIR alone solves the issue. But do you know why the command-line flag has no effect on this case? If I install a package that doesn't need building, it works as expected.

/ # cd
~ # pip install --no-cache-dir pluggy
Collecting pluggy
  Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Installing collected packages: pluggy
Successfully installed pluggy-1.0.0
~ # du
12      .

[pfmoore]

No problem - your instructions were sufficient in practice, so it’s fine.

The reason is that when building, pip creates a separate “build environment”, and does the build in that. To do the build it needs to install the tools needed for your build, such as setuptools and wheel. Pip runs itself in a subprocess to do that, and it’s that second copy of pip that doesn’t get the --no-cache flag applied. It’s arguably a bug that we don’t copy that flag over, but setting it via an environment variable is a reasonable workaround.

[orgads]

How hard will it be to pass it on to the subprocess?

Regarding the workaround, I applied it, and also suggested it upstream: docker-library/python#831.

[pfmoore]

There's code that passes other flags on, so it's probably not too hard to add this flag (but the details may be fiddly - presumably we would also need to handle --cache-dir for consistency, for example).

As with most things, PRs are welcome (and we can deal with the tricky edge cases, if any, in the PR) 🙂