Documentation incorrect for Linux build from source

[dnygren]

I asked and got answered this question on how to build Pillow on Linux from source on StackOverflow:
https://stackoverflow.com/questions/74537064/unable-to-build-pillow-python-imaging-library-from-source-on-linux

Based on the response I received, I am filing an issue because there may be incorrect documentation for building Pillow from source on Linux.

What did you do?

I tried to build Pillow from source by following the directions at https://pillow.readthedocs.io/en/stable/installation.html#building-from-source

What did you expect to happen?

Have Pillow built from source, including my changes.

What actually happened?

Pillow appeared to build from a different downloaded copy of Pillow.

What are your OS, Python and Pillow versions?

• OS:

$ uname -a
Linux mms-labpc1 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

• Python:

$ python3 --version
Python 3.9.2

• Pillow:
  9.4.0 (unreleased)

The following is how to reproduce the problem. Note I am following the Linux build from source Pillow documentation.

$ git clone https://github.com/python-pillow/Pillow.git
Cloning into 'Pillow'...
remote: Enumerating objects: 67869, done.
remote: Counting objects: 100% (163/163), done.
remote: Compressing objects: 100% (95/95), done.
remote: Total 67869 (delta 90), reused 128 (delta 68), pack-reused 67706
Receiving objects: 100% (67869/67869), 83.17 MiB | 9.39 MiB/s, done.
Resolving deltas: 100% (50202/50202), done.

$ sudo apt-get install python3-dev python3-setuptools
[sudo] password for dan: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-dev is already the newest version (3.9.2-3).
python3-setuptools is already the newest version (52.0.0-4).
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.

Remove all the source files from libImaging directory to prove its not building from source:

$ cd Pillow
$ rm ./src/libImaging/*

Continue to follow the Building From Source instructions at https://pillow.readthedocs.io/en/stable/installation.html#building-from-source for Debian Linux:

$ python3 -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /home/dan/.local/lib/python3.9/site-packages (22.3.1)

$ python3 -m pip install --upgrade Pillow --no-binary :all:
Defaulting to user installation because normal site-packages is not writeable
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting Pillow
  Using cached Pillow-9.3.0.tar.gz (50.4 MB)
  Preparing metadata (setup.py) ... done
Installing collected packages: Pillow
  DEPRECATION: Pillow is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currently disables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/11451
  Running setup.py install for Pillow ... done
Successfully installed Pillow-9.3.0

Pillow works on my system, but as I removed a large number of source files, I'm obviously not building from the cloned source files. The output says "Using cached Pillow-9.3.0.tar.gz" which implies it isn't using the 9.4.0 (unreleased) source files I cloned.

The following instructions do work. I proved if I removed the files from ./src/libImaging/ I now correctly get an error. Furthermore, after restoring the deleted files, I was later able to make changes to the Pillow source code and see that my changes affected the behavior of Pillow.

$ git clone https://github.com/python-pillow/Pillow.git
$ cd Pillow
$ python3 -m pip install .

[hugovk]

The instructions say:

1. download and extract the last released compressed archive (aka sdist) from PyPI
2. install necessary dependencies
3. install Pillow directly from PyPI via python3 -m pip install --upgrade Pillow --no-binary :all:, which builds from source using the sdist.

It may be that the first download and extract in (1) is a redundant step because we're essentially doing that in (3). But they should be correct otherwise, assuming you want to build from the source distribution.

And we haven't documented how to build from source from the repo after a git clone, which is where the confusion comes.

Building from main is valid use case, and maybe even the more common one?

---

The following instructions do work. I proved if I removed the files from ./src/libImaging/ I now correctly get an error. Furthermore, after restoring the deleted files, I was later able to make changes to the Pillow source code and see that my changes affected the behavior of Pillow.

$ git clone https://github.com/python-pillow/Pillow.git
$ cd Pillow
$ python3 -m pip install .

Yes, although you've already done step (2) which is also required.

---

Maybe we should update the instructions to also show how to install from source from GitHub?

Perhaps we could use https://sphinx-tabs.readthedocs.io to help organise the page?

For example, tabs under Basic installation:

macOS | Linux | FreeBSD

And so on.

[radarhere]

I've created #6756 as a suggestion for this. You can preview this at https://pillow--6756.org.readthedocs.build/en/6756/installation.html#building-from-source

[dnygren]

I previewed #6756 and found it to be helpful. Thank you Andrew. I also appreciate Hugo's comments. In my mind Building From Source means you are actually building from the local source code obtained from a git clone or by unpacking Pillow-9.3.0.tar.gz and running the commands indicated. It doesn't mean getting the Pillow source code and then building from different source code on PyPl. Perhaps the titles should read Building From PyPl and then Building From Local Source Code? I don't know why building from the PyPl source is a more common activity than building from the local source code. I had presumed if you want build from source, the ability to change the source would be included in that activity. Anyways, I'm pleased you paid attention to this issue and have a fixed proposed that would have saved me a day of head scratching.