Python3.13 performance Issue with python.org macOS installers on ARM Macs

[hg0428]

Bug description:

Python3.13rc1 seems to be about 20-25% slower than Python3.12.4.
Platform: M1 Mac running MacOS Sonoma 14.5

I first noticed the issue when running this repository where Python3.13rc1 ran on average 20-25% slower than Python3.12.4

With help from the Python discord server, we were able to create this minimal reproducible example:

from time import process_time

class Lexer:
    def __init__(self, data):
        self.data = data
        self.index = 0

    def detect(self, text):
        i = 0
        for char in text:
            if self.peek(i) != char:
                return False
            i += 1
        return True

    def peek(self, amt=1):
        if self.index + amt < len(self.data):
            return self.data[self.index + amt]
        else:
            return None

text = "return text + indent * ' ' + '}'\n" * 200
needle = "return text + indent * ' ' + '}'\n" * 199 + "Nope\n" 
lexer = Lexer(text)
start = process_time() 
for x in range(10_000):
    lexer.detect(needle)
print(process_time() - start)

These were the results across many tests:
3.12.4: 7.742122s
3.13rc1: 8.309326s

The performance issue was not reproducible on other Operating Systems.

CPython versions tested on:

3.11.1, 3.12.4, 3.13

CPython Options:

For my testing, I used the official Python3.13rc1 installer from Python.org, which has the following configuration:

OPT:

3.13rc1: -DNDEBUG -g -O3 -Wall
3.12.4: -DNDEBUG -g -O3 -Wall

CONFIG_ARGS:

3.13rc1: '--enable-framework' '--with-framework-name=Python' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--enable-optimizations' '--with-lto' '--without-ensurepip' '--with-system-libmpdec' '--with-openssl=/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local' 'LIBLZMA_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBLZMA_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -llzma' 'LIBMPDEC_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBMPDEC_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lmpdec -lm' 'LIBSQLITE3_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'LIBSQLITE3_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -lsqlite3' 'TCLTK_CFLAGS=-I/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/include' 'TCLTK_LIBS=-L/Users/nad/release-tools/macos-installer/installer/variant/binaries/build/libraries/usr/local/lib -ltcl -ltk' 'CC=clang'
3.12.4: '-C' '--enable-framework' '--enable-universalsdk=/' '--with-universal-archs=universal2' '--with-computed-gotos' '--without-ensurepip' '--with-openssl=/tmp/_py/libraries/usr/local' '--enable-optimizations' '--with-lto' 'TCLTK_CFLAGS=-I/tmp/_py/libraries/usr/local/include' 'TCLTK_LIBS=-ltcl8.6 -ltk8.6' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=clang'

Operating systems tested on:

Linux, macOS, Windows

[ZeroIntensity]

Me and @devdanzin have figured out that this is only on ARM, Windows and Linux x86 seemed to be faster based on the benchmarks.

[mdboom]

Thanks for this report. Across a broad range of benchmarks measuring on the Faster CPython benchmarking machine, we see between a 3% and 8% speedup for 3.13.0b4 vs. 3.12.0 on a MacOS ARM machine.

That doesn't mean that this particular example hasn't found a narrow regression case. In our own benchmarking, some specific benchmarks are in fact slower, but only telco is that much slower (which is a known regression in the decimal module). Maybe some line profiling of 3.12 vs. 3.13 would be helpful here, or some pystats results.

I also noticed that you have --with-computed-gotos for 3.12, but not 3.13 here. That may not be relevant since it's on by default anyway (and I'm not sure it even applies for clang), but we should make sure all other variables are equal.

[mdboom]

I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:

3.12.0: 7.471261
3.13.0rc1: 5.986742

I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce.

[hg0428]

I just tried to reproduce the above result on our benchmarking M1 Mac, also running macOS 14.5, and got different results:

3.12.0: 7.471261
3.13.0rc1: 5.986742

I suspect this comes down to different configurations between the builds. It looks like 3.12.0 on your machine was built on a different machine? Try building each in exactly the same environment and see if you can still reproduce.

I am using 3.12.4, not 3.12.0.
I have tested 3.11, 3.12.1, 3.12.3, all from Python.org at different points in time.
I have also tested many different betas of 3.13, and the results have been consistent.

All the versions installed on my machine are the default ones from Python.org.

[brandtbucher]

I also can't reproduce with local optimized builds on an M2 Pro running Sonoma 14.5 (3 runs each):

v3.12.4:          5.718666, 6.035776, 5.577658
v3.13.0rc1:       4.933225, 5.214851, 4.837962
v3.13.0rc1 (JIT): 4.746776, 4.747096, 4.764923

However, I can reproduce with the python.org downloads:

3.12.4:           5.684782, 5.808816, 5.734691
3.13.0rc1:        6.673098, 6.930707, 7.369320

So the issue seems to be with the releases themselves. @ned-deily, anything different with macOS releases this time around?

[brandtbucher]

So, this is kinda strange. Consider this reproducer:

import time

s = "X" * 100_000_000

start = time.perf_counter()
for _ in s:
    pass
duration = time.perf_counter() - start
print(f"global: {duration:.3f}s")

def foo():
    for _ in s:
        pass

start = time.perf_counter()
foo()
duration = time.perf_counter() - start
print(f"function: {duration:.3f}s")

With the 3.12.4 python.org download, this prints:

global: 1.237s
function: 0.364s

On 3.13.0rc1:

global: 0.930s
function: 0.680s

So the loop over the string is faster at module level on 3.13, but faster at function level in 3.12. And 3.13 is faster on both if I replace "X" * 100_000_000 with range(100_000_000).

[brandtbucher]

The issue could be some sort of messed-up (or skipped) PGO with the release build. The global version has lots of FOR_ITER -> STORE_NAME pairs while the function version has lots of FOR_ITER -> STORE_FAST pairs. That's really the only difference that I can see. I'd expect PGO to make the latter very fast, at the expense of the former.

But at this point I'm just guessing.

[mdboom]

However, I can reproduce with the python.org downloads:

Just dittoing this. I agree, there may be something that changed in the build environment between releases.

[ned-deily]

I can reproduce a similar slower performance between the python.org 3.12.4 and 3.13.0rc1 builds on an M3 MacBook Pro with macOS 14.6. However, using the same python.org installers on an Intel i9 iMac also with macOS 14.6 shows a performance improvement between 3.12.4 and 3.13.0rc1. These are both universal2 builds meaning they should be running native arm or x86_64 code on each. And, interestingly, if I force running the universal2 builds in x86_64 mode using Rosetta2 emulation on the same M3 (by using python3.1x-intel64), I see performance differences similar to running natively on the Intel Mac, that is 3.13.0rc1 is somewhat faster.

There are a number of differences in the build environments used for 3.12.4 and for 3.13.0rc1, including the compiler versions and deployment targets. I agree that something with optimizations seems a likely culprit. I will dig into it and report back.

Thanks for the report and the simple reproducer, @hg0428!

[ned-deily]

A quick update: I've run the test case on the full range of python.org macOS installers from 3.13.0a5 through 3.13.0rc1 on both Apple Silicon (M3, arm64) and Intel (i9) Macs and it appears that rc1 on Apple Silicon is the outlier. I also ran the test case on all of the macOS installers where we also provided a free-threading build, that is, 3.13.0b2 through 3.13.0rc1, again, on both CPU types; there, the test-case performance differences roughly mirror those of the the normal (GIL) builds except that the rc1 free-threading version does not exhibit the performance degradation that the normal version does on Apple Silicon. So, the good news at this point is that this particular performance issue appears to be isolated but there is more work to be done to pin it down further. More tomorrow.