pex is inexplicably slow for a small project

[zmanji]

Reproduction

1. Clone https://github.com/zmanji/git-squash
2. Checkout 505cf06c598409db9d119a521480e43de3fa8908
3. Run pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex

With latest pex on macOS.

Expectation

Because of --no-build, --wheel, --no-transitive and a fully resolved requirements.txt from pip-compile building a pex should be very fast. The wheels can be fetched from pypi if they are not cached locally, the package it self a single file, so building the wheel should be very fast and then building the pex should be straight forward

For reference building the wheel for git-squash via python ./setup.py bdist_wheel takes about 0.3s on my computer.

Reality

I see this output

pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macpex: Hashing pex                                                                                                                                                                                pex: Hashing pex: 67.4ms                                                                                                                            
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Building distributions for:
  BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', pex: Building /Users/zmanji/code/git-squash to /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39                                                                                                                                                                                                                                                                                                                                
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Rpex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-pex: Using cached installation of GitPython-3.1.17-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/59d63e79b53503f5403220d3159d8e409c46e107/GitPython-3.1.17-py3-none-any.whl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
pex: Using cached installation of smmap-4.0.0-py2.py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f/smmap-4.0.0-py2.py3-none-any.whl
pex: Using cached installation of gitdb-4.0.7-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/9dacbc989a247dabdad014d6397d8482fd55f26c/gitdb-4.0.7-py3-none-any.whl
pex: Installing /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl in /Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl
pex: Set entrypoint to console_script 'git_squash.git_squash:main' in git-squash 0.0.1 (/Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl)
pex: Building pex: 13722.4ms
pex:   Resolving distributions (['.', 'requirements.txt']): 13720.7ms
pex:     Resolving requirements.: 13709.1ms
pex:       Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))): 6592.5ms
pex:       Building distributions for:
  BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), source_path='/Users/zmanji/code/git-squash', fingerprint='fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e'): 6475.2ms
pex:       Calculating project names for direct requirements:
  LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Requirement.parse('smmap==4.0.0'), editable=False): 4.3ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl', fingerprint='c355002f497d1f36a0bae44b05e55b4338da8a0f'): 430.6ms
Saving PEX file to dist/out.pex
pex: Zipping PEX file.: 148.4ms
pex . -vv -r requirements.txt --no-build --wheel --no-transitive    --script   9.65s user 4.27s system 96% cpu 14.348 total

Notice:

pex:     Resolving requirements.: 13709.1ms

This is very unexpected and there is no information in the output as to why it takes so long. With all of the caching PEX uses, I would expect this to be very fast.

[jsirois]

Can you run the equivalent pip wheel command and report the timing? Pex fundamentally can't beat Pip without picking back up aspects of resolution it dropped when transitioning to Pip.

[jsirois]

Related:

• Avoid Resolving Dependencies When Building in Intransitive Mode  #1093
• Support dependency lock files to save the results of package resolution #1086

[zmanji]

There are the results of comparing pip wheel with pex. I am using hyperfine to produce benchmarks and I believe this is the equivalent pip command.

hyperfine 'pip wheel -w dist/ --only-binary ":all:" -r requirements.txt .'
Benchmark #1: pip wheel -w dist/ --only-binary ":all:" -r requirements.txt .
  Time (mean ± σ):      7.270 s ±  0.259 s    [User: 4.640 s, System: 2.305 s]
  Range (min … max):    6.853 s …  7.648 s    10 runs

vs

hyperfine 'pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex'
Benchmark #1: pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex
  Time (mean ± σ):      9.114 s ±  3.147 s    [User: 5.976 s, System: 2.798 s]
  Range (min … max):    7.327 s … 15.257 s    10 runs

I would expect pex to have minimal to no overhead over pip in cases like this.

[jsirois]

Hrm, I find different. My only change was to eliminate some default Pex flags for brevity (--wheel and --inherit-path) and add a missing equivalent flag for pip wheel to be fair to pip (--no-deps):

/tmp $ git clone https://github.com/zmanji/git-squash
Cloning into 'git-squash'...
remote: Enumerating objects: 93, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 93 (delta 10), reused 27 (delta 10), pack-reused 66
Receiving objects: 100% (93/93), 11.62 KiB | 11.62 MiB/s, done.
Resolving deltas: 100% (37/37), done.
/tmp $ cd git-squash/
/tmp/git-squash (master) $ git reset --hard 505cf06c598409db9d119a521480e43de3fa8908
HEAD is now at 505cf06 fix
/tmp/git-squash (master) $ python -mvenv tools.venv
/tmp/git-squash (master) $ source tools.venv/bin/activate
(tools.venv) /tmp/git-squash (master) $ pip -q install -U pip
(tools.venv) /tmp/git-squash (master) $ pip -q install -U pex
(tools.venv) /tmp/git-squash (master) $ pip --version
pip 21.1.2 from /tmp/git-squash/tools.venv/lib/python3.9/site-packages/pip (python 3.9)
(tools.venv) /tmp/git-squash (master) $ pex --version
2.1.42
(tools.venv) /tmp/git-squash (master) $ which pip
/tmp/git-squash/tools.venv/bin/pip
(tools.venv) /tmp/git-squash (master) $ which pex
/tmp/git-squash/tools.venv/bin/pex
(tools.venv) /tmp/git-squash (master) $ hyperfine -w2 'tools.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' 'tools.venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'
Benchmark #1: tools.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .
  Time (mean ± σ):      2.826 s ±  0.021 s    [User: 2.450 s, System: 0.259 s]
  Range (min … max):    2.792 s …  2.854 s    10 runs
 
Benchmark #2: tools.venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash
  Time (mean ± σ):      2.664 s ±  0.029 s    [User: 2.285 s, System: 0.265 s]
  Range (min … max):    2.627 s …  2.710 s    10 runs
 
Summary
  'tools.venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash' ran
    1.06 ± 0.01 times faster than 'tools.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .'

[jsirois]

That result is consistent for me. After 5 runs of that same hyperfine setup I get pex as 3%-5% faster than pip wheel each time. Informally I've found this to be generally true and it makes sense since pip wheel uses no parallelism but pex uses pip download and then runs pip wheel && pip install on each resulting downloaded distribution in parallel. My machine has 8 cores, so by default that backend stage gets that much parallelization.

I'm on a quiet Linux laptop with ssd and that's the only other set of strongly pertinent variables I can think of.

[jsirois]

@zmanji if you can't repro my results on a quiet machine I'd like to dig more to understand the performance considerations I'm not taking into account. If you can repro my result I'd like to close and let the issues I linked above stand for the case of optimizing pinned resolves / lockfiles.

[zmanji]

On a second mac I can see pip and pex are pretty much equal:

 hyperfine -w2 'venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'
Benchmark #1: venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .
  Time (mean ± σ):      5.930 s ±  0.765 s    [User: 3.346 s, System: 1.565 s]
  Range (min … max):    5.253 s …  7.352 s    10 runs
 
Benchmark #2: venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash
  Time (mean ± σ):      5.652 s ±  0.145 s    [User: 3.502 s, System: 1.697 s]
  Range (min … max):    5.328 s …  5.825 s    10 runs
 
Summary
  'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash' ran
    1.05 ± 0.14 times faster than 'venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .'

I also thought it was related to this pip issue but deleting the .git directory makes no difference in the above results.

Adding --use-feature=in-tree-build to pip shows:

hyperfine -w2 'venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'
Benchmark #1: venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .
  Time (mean ± σ):      4.342 s ±  0.740 s    [User: 2.800 s, System: 0.600 s]
  Range (min … max):    3.608 s …  5.673 s    10 runs
 
Benchmark #2: venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash
  Time (mean ± σ):      5.656 s ±  0.166 s    [User: 3.481 s, System: 1.700 s]
  Range (min … max):    5.445 s …  6.034 s    10 runs
 
Summary
  'venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' ran
    1.30 ± 0.23 times faster than 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'

Feel free to close this. I would suggest pex would be aggressive in adopting --use-feature=in-tree-build when it becomes the default.

[zmanji]

Some additional data for the record.

It seems that adding a pyproject.toml to the root slows down pip wheel immensely

With the pyproject.toml in the root:

Benchmark #1: venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .
  Time (mean ± σ):      4.726 s ±  0.210 s    [User: 3.801 s, System: 0.681 s]
  Range (min … max):    4.431 s …  5.111 s    10 runs
 
Benchmark #2: venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash
  Time (mean ± σ):      7.445 s ±  0.266 s    [User: 4.872 s, System: 2.265 s]
  Range (min … max):    7.035 s …  7.790 s    10 runs
 
Summary
  'venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' ran
    1.58 ± 0.09 times faster than 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'

After deleting pyproject.toml

hyperfine -w2 'venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'
Benchmark #1: venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .
  Time (mean ± σ):      1.218 s ±  0.033 s    [User: 915.5 ms, System: 200.2 ms]
  Range (min … max):    1.189 s …  1.306 s    10 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark #2: venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash
  Time (mean ± σ):      3.396 s ±  0.047 s    [User: 1.483 s, System: 1.782 s]
  Range (min … max):    3.348 s …  3.481 s    10 runs
 
Summary
  'venv/bin/pip wheel --use-feature=in-tree-build -w pip.dist/ --only-binary ":all:" --no-deps -r requirements.txt .' ran
    2.79 ± 0.09 times faster than 'venv/bin/pex . -r requirements.txt --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash'

[zmanji]

Reported the above to pip disabling pep517 builds makes this a bit more tolerable for me.

[jsirois]

@zmanji that bug report is probably wanting in two ways:

1. It makes no mention of the important detail of which pep-517 build system you're using. Is it poetry, flit, setuptools ... ?
2. No significant performance change between --use-pep517 and --no-use-pep517 is a super debatable expectation.

For example, 2 - by design of the PEP - will download the build system (say setuptools) - and then run it. It will not expect the build system to be present in your current interpreters environment. So that hermetic bit of resolving the build system is new and should be expected to take time.

[jsirois]

Hrm, I guess it does - but not in too friendly a way: zmanji/git-squash@e714352

Looks like setuptools / wheel with no bounds. So latest.

[jsirois]

It looks like --use-feature=in-tree-build is available in latest Pip. Pex is stuck back on the last Pip to support Python 2.7 / 3.5, but Pex 3 will drop support for those and move back to latest Pip. At that time we can enable the in-tree-build feature or add a flag to do so.

[jsirois]

This issue is rambling a bit, but I'll close as agreed upon.

[NiklasRosenstein]

Hey @jsirois, what's the current status on using in-tree-build?

[jsirois]

I've made good progress on the Pip upgrade to latest over in #1805 this week. It looks like Pex will be able to support both the current vendored Pip 20.3.4-patched and be able to download and use Pip 22.2.2 un-patched on the fly behind a flag (currently: --pip-version 22.2.2). I would have likely forgotten about exposing a knob to enable (or default and disable) in-tree-build so thanks for the reminder.

[jsirois]

Actually, in-tree-build became the default in 21.3: https://pip.pypa.io/en/stable/news/#v21-3 and the ability to fall back was removed in 22.1b4 https://pip.pypa.io/en/stable/news/#id45, so just by using --pip-version 22.2.2 you'll get this when #1805 ships.

Using Pip 22.2.2 and my branch implementing #1805 on @zmanji's test cases I find:

1. git-squash HEAD (no pyproject.toml):

   $ hyperfine -w2 'test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' 'test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' 'test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
   Benchmark 1: test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):     731.7 ms ±  10.8 ms    [User: 512.5 ms, System: 59.7 ms]
     Range (min … max):   723.2 ms … 757.7 ms    10 runs
    
   Benchmark 2: test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):     766.3 ms ±   4.2 ms    [User: 550.7 ms, System: 56.9 ms]
     Range (min … max):   761.7 ms … 776.3 ms    10 runs
    
   Benchmark 3: test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):     850.1 ms ±   5.6 ms    [User: 628.2 ms, System: 63.3 ms]
     Range (min … max):   842.9 ms … 857.8 ms    10 runs
    
   Summary
     'test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' ran
       1.05 ± 0.02 times faster than 'test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
       1.16 ± 0.02 times faster than 'test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
   

2. git-squash @ 63e53de - just before pyproject.toml deletion:

   $ hyperfine -w2 'test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' 'test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' 'test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
   Benchmark 1: test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):      1.715 s ±  0.010 s    [User: 1.296 s, System: 0.135 s]
     Range (min … max):    1.701 s …  1.733 s    10 runs
    
   Benchmark 2: test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):      2.179 s ±  0.019 s    [User: 1.752 s, System: 0.126 s]
     Range (min … max):    2.142 s …  2.205 s    10 runs
    
   Benchmark 3: test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/
     Time (mean ± σ):      1.892 s ±  0.019 s    [User: 1.432 s, System: 0.140 s]
     Range (min … max):    1.871 s …  1.934 s    10 runs
    
   Summary
     'test.venv/bin/pip wheel -w pip.dist/ --only-binary ":all:" --no-deps -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/' ran
       1.10 ± 0.01 times faster than 'test.venv/bin/pex --pip-version 22.2.2 --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
       1.27 ± 0.01 times faster than 'test.venv/bin/pex --no-build --no-transitive -o git-squash-0.0.1.pex -c git-squash -r ~/dev/zmanji/git-squash/requirements.txt ~/dev/zmanji/git-squash/'
   

So it's a mixed bag, but Pex is always a bit slower. Pex can come out ahead mixing in --no-compress, but that's not a fair fight since there is no such option that I'm aware of to get pip wheel to build an uncompressed wheel.

[zmanji]

For the pep517 case could pex ship with a basic wrapper around the pep517 library that creates a venv pex to build the wheel?

It's my understanding the slowest part is building a wheel is creating a clean venv with the dependencies specified in pyproject.toml. pex could just re-use an existing venv in PEX_ROOT

[jsirois]

It could take on building wheels, but currently it offloads to Pip all it can (Pex currently uses download for resolve, wheel for building wheels and install for installing wheels - all to be maximally robust and "works like you'd expect" on the premise Pip should always be better at both.). Pex would need to handle both legacy and PEP-517 and the building would potentially be non-uniform, since implicit partial legacy builds (egg-info) get run by Pip to fulfill its dowload command.

[jsirois]

The issues to contend with if Pex did tackle wheel building speed optimizations would include what to do about re-use of venvs. There is no way to know just by looking at the top-level [build-system] requires whether anything has changed. Even if all the top-level requirements are pinned, a year later the venv may be stale because some transitive deps are not pinned. In short, a lock file is needed to both be performant and correct but this is not PEP'd at all let alone for PEP-518 build systems requirements. See https://peps.python.org/pep-0665/ which actually died due to this very issue!

[jsirois]

FWIW the infra does partially exist with an entry point here:
https://github.com/pantsbuild/pex/blob/9cc674b5b45d80338d77a20fe8a3ccaa9c74fa20/pex/build_system/pep_517.py#L60-L65

That is used when locking local projects to get a consistent hashable thing (build an sdist tarball, then extract it to a dir and hash that dir tree) that represents the relevant project contents and not extraneous bits (.pyc laying about, tool caches like .tox/, etc.)

That just supports build_sdist right now but could support building a wheel. It takes a Resolver which support reading lockfiles, but currently doesn't have the plumbing to create a lock file.