Crash when free-threaded Python is built with `--with-trace-refs`

[XuehaiPan]

Crash report

What happened?

I'm using ThreadPoolExecutor to write tests for free-threading enabled C extensions.

• deps(python): enable Python 3.13t support metaopt/optree#137

When running concurrent threads larger than num_workers, CPython crashes. CPython crashes when num_futures >= 2 if the test function is complex enough.

The Python is installed via the following configuration with PyDebug enabled:

./configure --prefix="${PWD}/pydev" \
    --enable-ipv6 --with-openssl="$(brew --prefix openssl)" \
    --with-system-expat --with-system-libmpdec \
    --with-assertions --with-pydebug --with-trace-refs \
    --disable-gil

from concurrent.futures import ThreadPoolExecutor

NUM_WORKERS = 32
NUM_FUTURES = 1024

def concurrent_run(func):
    with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
        for _ in range(NUM_FUTURES):
            executor.submit(func)


concurrent_run(lambda : None)

Python 3.13.0rc2+ experimental free-threading build (heads/3.13:112b1704fa6, Sep 13 2024, 15:34:48) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from concurrent.futures import ThreadPoolExecutor
>>> NUM_WORKERS = 32
... NUM_FUTURES = 1024
... 
... def concurrent_run(func):
...     with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
...         for _ in range(NUM_FUTURES):
...             executor.submit(func)
...             
>>> concurrent_run(lambda : None)
Assertion failed: (value == REFCHAIN_VALUE), function _PyRefchain_Remove, file object.c, line 195.
Fatal Python error: Aborted

Thread 0x0000000173e2b000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 304 in __enter__
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 528 in release
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 87 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000000172e1f000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 89 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000000171e13000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 89 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Current thread 0x0000000170e07000 (most recent call first):
Assertion failed: (PyCode_Check(f->f_executable)), function _PyFrame_GetCode, file pycore_frame.h, line 81.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Output from running 'python -VV' on the command line:

Python 3.13.0rc2+ experimental free-threading build (heads/3.13:112b1704fa6, Sep 13 2024, 15:34:48) [Clang 15.0.0 (clang-1500.3.9.4)]

Linked PRs

• gh-124043: Disallow mixing --with-trace-refs and --disable-gil #124078
• [3.13] gh-124043: Disallow mixing --with-trace-refs and --disable-gil (GH-124078) #124138

[ZeroIntensity]

At a glance, looks like this could be a data race. I'll take a look.

[ZeroIntensity]

Ah, --with-trace-refs is unsupported under free-threading, so this is a non-issue. As a double check, I confirmed that this doesn't occur with tracerefs disabled. I wonder if configure should disallow mixing --with-trace-refs and --disable-gil, as this isn't the first case of this happening. See #122921, for example.

@encukou, this needs pending.

[colesbury]

Let's disable --with-trace-refs until it works so that more people don't run into this

[ZeroIntensity]

Let's disable --with-trace-refs until it works so that more people don't run into this

I've created #124078 for this.