[mypyc] Pickle test failure with Python 3.14.0a1

[cdce8p]

After being a bit late with Python 3.13, I figured to start testing 3.14 early this time.

With Python 3.14.0a1 the following fails:

# native.py
from typing import Any

def dec(x: Any) -> Any:
    return x

@dec
class D:
    x: int

class E(D):
    y: int

# driver.py
from native import D, E

import pickle

assert not hasattr(D, '__mypyc_attrs__')
assert E.__mypyc_attrs__ == ('y', '__dict__')

e = E()
e.x = 10
e.y = 20

assert e.__getstate__() == {'y': 20, '__dict__': {'x': 10}}
e2 = pickle.loads(pickle.dumps(e))
assert e is not e2 and e.x == e2.x and e.y == e2.y

rm -rf build
rm *.so
python3.14 -m mypyc native.py
python3.14 -c 'import driver.py'

The error message

$ python -c "import driver.py"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import driver.py
  File ".../driver.py", line 14, in <module>
    assert e is not e2 and e.x == e2.x and e.y == e2.y
                                  ^^^^
AttributeError: 'E' object has no attribute 'x'

I bisected this to python/cpython#123192 upstream.

--
This is also part of the mypyc test suite

pytest -n0 mypyc/test/test_run.py::TestRun::run-classes.test::testPickling

[hauntsaninja]

I suspect it's something funky when calling PyObject_SetAttr(obj, "__dict__", {"x": 10}), which is what mypyc will do here when unpickling

With a debug Python build you also get a segfault following the AttributeError:

Assertion failed: (0), function _PyObject_InlineValuesConsistencyCheck, file dictobject.c, line 7470.
zsh: abort      python driver.py

In lldb:

  * frame #4: 0x000000010031b524 python`_PyObject_InlineValuesConsistencyCheck.cold.3 at dictobject.c:7470:5 [opt]
    frame #5: 0x00000001000d3e54 python`_PyObject_InlineValuesConsistencyCheck(obj=<unavailable>) at dictobject.c:7470:5 [opt]
    frame #6: 0x00000001000d3b94 python`_PyObject_SetManagedDict(obj=0x0000000100bc7420, new_dict=0x0000000000000000) at dictobject.c:7114:5 [opt]
    frame #7: 0x00000001000d3fb8 python`PyObject_ClearManagedDict(obj=<unavailable>) at dictobject.c:7176:9 [opt]
    frame #8: 0x0000000100c67088 native.cpython-314d-darwin.so`E_dealloc [inlined] E_clear at __native.c:50:5 [opt]
    frame #9: 0x0000000100c67024 native.cpython-314d-darwin.so`E_dealloc at __native.c:59:5 [opt]
    frame #10: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100bc7420) at object.c:2933:5 [opt]
    frame #11: 0x00000001000d41c0 python`dictkeys_decref [inlined] Py_DECREF(filename=<unavailable>, lineno=476, op=0x0000000100bc7420) at refcount.h:367:9 [opt]
    frame #12: 0x00000001000d4174 python`dictkeys_decref [inlined] Py_XDECREF(op=0x0000000100bc7420) at refcount.h:476:9 [opt]
    frame #13: 0x00000001000d4170 python`dictkeys_decref(interp=<unavailable>, dk=0x0000000100eda3b0, use_qsbr=false) at dictobject.c:460:17 [opt]
    frame #14: 0x00000001000cfc2c python`dict_dealloc(self=0x0000000100a6c7d0) at dictobject.c:0 [opt]
    frame #15: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100a6c7d0) at object.c:2933:5 [opt]
    ...

[cdce8p]

Still an issue with 3.14.0a2

[cdce8p]

Still an issue with 3.14.0a3 as well.
Edit: And with 3.14.0a4 too.

[cdce8p]

A possible workaround could be to disable the Py_TPFLAGS_INLINE_VALUES here:

mypy/mypyc/lib-rt/misc_ops.c

Lines 299 to 304 in f44a60d

	#if PY_MINOR_VERSION == 11
	// This is a hack. Python 3.11 doesn't include good public APIs to work with managed
	// dicts, which are the default for heap types. So we try to opt-out until Python 3.12.
	t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT;
	#endif
	return (PyObject *)t;

 #if PY_MINOR_VERSION == 11
     // This is a hack. Python 3.11 doesn't include good public APIs to work with managed
     // dicts, which are the default for heap types. So we try to opt-out until Python 3.12.
     t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT;
+#elif PY_MINOR_VERSION == 14
+    t->ht_type.tp_flags &= ~Py_TPFLAGS_INLINE_VALUES;

With that the test case passes again (and no other test is failing because of it). However, I'm unsure that's the correct solution. With python/cpython#123192 Py_TPFLAGS_INLINE_VALUES is automatically set for type->tp_itemsize == 0 which is the case here.

[cdce8p]

From what I can tell the testPickling case also had issues when adding support for 3.12.

• [mypyc] Fix classes with __dict__ on 3.12 #15471
• [mypyc] Fix 3.12 issue with pickling of instances with __dict__ #15574

/CC @JukkaL

[cdce8p]

This was fixed in python/cpython#134859 which will be included in Python 3.14.0b3.