Pytest 8.2.0 breaks Jenkins unittesting [Python 3.10]

[matthuisman]

[some text/paths removed below]

Note some warnings and then the "script returned exit code -1073741819"
I can't reproduce locally - i always get exit code 0

If I go back to 8.1.2 - jenkins works fine without any other changes.
I get the same warnings etc, but the exit code is OK

Any help / ideas to try would be greatly appreciated :)

Command that runs the below:
pytest --splits 1 --group 1 --splitting-algorithm least_duration --store-durations --clean-durations --durations-path test_durations.json --junitxml=junit.xml

[2024-04-28T21:10:57.409Z] ============================= test session starts =============================
[2024-04-28T21:10:57.409Z] platform win32 -- Python 3.10.9, pytest-8.2.0, pluggy-1.5.0
[2024-04-28T21:10:57.409Z] rootdir: C:\jenkins\workspace\on_master@2
[2024-04-28T21:10:57.409Z] configfile: pytest.ini
[2024-04-28T21:10:57.409Z] testpaths: tests
[2024-04-28T21:10:57.409Z] plugins: cov-5.0.0, split-0.8.2, subtests-0.12.1
[2024-04-28T21:10:59.917Z] 
[2024-04-28T21:10:59.917Z] 
[2024-04-28T21:10:59.917Z] [pytest-split] Splitting tests with algorithm: least_duration
[2024-04-28T21:10:59.917Z] [pytest-split] Running group 1/1 (estimated duration: 24.05s)
[2024-04-28T21:10:59.917Z] 
[2024-04-28T21:10:59.917Z] collected 430 items
[2024-04-28T21:10:59.917Z] 
[2024-04-28T21:11:16.574Z] 
[2024-04-28T21:11:16.574Z] [pytest-split] Stored test durations in test_durations.json
[2024-04-28T21:11:16.574Z] 
[2024-04-28T21:11:16.574Z] 
[2024-04-28T21:11:16.574Z] ============================== warnings summary ===============================
[2024-04-28T21:11:16.574Z] .envs\py310\venv***tools\helper.py:13
[2024-04-28T21:11:16.574Z]   C***tools\helper.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
[2024-04-28T21:11:16.574Z]     import pkg_resources
[2024-04-28T21:11:16.574Z] 
[2024-04-28T21:11:16.574Z] .envs\py310\venv***testenvironment.py:180
[2024-04-28T21:11:16.574Z]   C***testenvironment.py:180: PytestCollectionWarning: cannot collect test class 'TestEnvironment' because it has a __init__ constructor (from: tests/test_framework.py)
[2024-04-28T21:11:16.574Z]     class TestEnvironment(object):
[2024-04-28T21:11:16.574Z] tests/test_options.py::TestPcuTypeOption::test_save_and_recall_after_calling_plot
[2024-04-28T21:11:16.575Z]   ***.py:188: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
[2024-04-28T21:11:16.575Z]     fig, ax = plt.subplots()
[2024-04-28T21:11:16.575Z] 
[2024-04-28T21:11:16.575Z] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[2024-04-28T21:11:16.575Z] - generated xml file: C:\jenkins\workspace\@2\junit.xml -
[2024-04-28T21:11:16.575Z] 
[2024-04-28T21:11:16.575Z] ---------- coverage: platform win32, python 3.10.9-final-0 -----------
[2024-04-28T21:11:16.575Z] Name                                                       Stmts   Miss Branch BrPart  Cover
[2024-04-28T21:11:16.575Z] --------------------------------------------------------------------------------------------
.......
[2024-04-28T21:11:16.575Z] --------------------------------------------------------------------------------------------
[2024-04-28T21:11:16.575Z] TOTAL                                                       5379   1498   1518    125    71%
[2024-04-28T21:11:16.575Z] 
[2024-04-28T21:11:16.575Z] ====================== 430 passed, 16 warnings in 19.37s ======================
script returned exit code -1073741819

[matthuisman]

we also have other repos that use pytest and are ok with 8.2.0 that contain DeprecationWarning

But, I cant see any PytestCollectionWarning in them.

Was there a change in 8.2.0 that makes PytestCollectionWarning make the command exit code != 0?

[nicoddemus]

A warning would only cause an exit code != 0 if warnings are turned into errors (with filterwarnings=error).

However the return code -1073741819 is not issued by pytest, seems like something else is causing the interpreter to crash... 🤔

[bluetech]

-1073741819 in Windows is "access violation" AKA "segmentation fault". pytest itself, written in pure Python, cannot trigger such a thing. You need to run the program under a (C) debugger to find out why it happens. You can also try changing pytest ... to python -X dev -m pytest ... to enables faulthandler for the entire duration and might catch the issue if it's coming from Python.

[matthuisman]

Im going to run some experiments. Basically keep disabling tests until it stops crashing. Then see what is different with that particular test. We have many other repos using the same jenkins nodes with same python that were running fine with latest pytest

[matthuisman]

Script and instruction to reproduce below.

"""
Python 3.10.9 - Windows X64

pip install wxpython==4.2.1 pytest==8.2.0 --prefer-binary
pytest test_abc.py

Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000020576a43c00)

running unittest directly OK
running with pytest 8.1.0 OK
running with pytest 8.2.0 + Python 3.12.3 OK
"""
import wx
import unittest

class Test_ABC(unittest.TestCase):
    def setUp(self):
        self.app = wx.App()
        wx.CallAfter(lambda: print("hello"))

    def test_a(self):
        pass

    # need two tests as the issue is triggered when setUp called the 2nd time
    def test_b(self):
        pass

if __name__ == "__main__":
    unittest.main()

we have fixed this by creating the app in class setup instead

@classmethod
def setUpClass(cls):
   cls.app = wx.App()

However, thought you may still be interested to investigate why 8.2.0 started making it crash :)

[matthuisman]

@bluetech did you see the above?

[kavanase]

Not sure if it's related, but just to mention in case it is and is useful to know. Our GitHub Actions tests for doped started crashing in certain cases (ubuntu-latest, python=3.10) upon updating to pytest==8.2.0, whereas fixing it to the previous 8.1.2 had all tests running fine, before and after.
The output from the GitHub CI wasn't particularly informative, but seemed to be due to some memory/resource overload issue with our plotting tests, which seems like it could be related to @matthuisman's issues (moving the app initialisation to setUpClass to reduce memory/resource demand?).

For the section of our tests that runs plotting tests with:

pytest --mpl -m "mpl_image_compare" tests  # all plotting tests

the log output was:

2024-05-11T08:23:16.2962832Z ============================= test session starts ==============================
2024-05-11T08:23:16.2969685Z platform linux -- Python 3.10.14, pytest-8.2.0, pluggy-1.5.0
2024-05-11T08:23:16.2979464Z Matplotlib: 3.8.4
2024-05-11T08:23:16.2982774Z Freetype: 2.6.1
2024-05-11T08:23:16.2985341Z rootdir: /home/runner/work/doped/doped
2024-05-11T08:23:16.2988990Z configfile: pyproject.toml
2024-05-11T08:23:16.2989605Z plugins: mpl-0.17.0
2024-05-11T08:23:16.2992918Z collected 197 items / 133 deselected / 64 selected
2024-05-11T08:23:16.2993313Z 
2024-05-11T08:42:54.3094183Z tests/test_analysis.py ......................                            [ 34%]
2024-05-11T08:46:03.0040480Z tests/test_corrections.py .......                                        [ 45%]
2024-05-11T08:46:07.4215628Z tests/test_displacements.py ...                                          [ 50%]
2024-05-11T09:02:25.4387407Z tests/test_plotting.py .............................                     [ 95%]
2024-05-11T09:04:29.0892314Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
2024-05-11T09:04:30.7922723Z tests/test_thermodynamics.py 
2024-05-11T09:04:30.8393348Z ##[error]The operation was canceled.

Full logs of failed and passed runs attached if useful

passed_GH_Actions_log_pytest_8.1.1.txt
failed_GH_Actions_log_pytest_8.2.0.txt

[bluetech]

@matthuisman Thanks for providing a reproduction. I bisected the segfault to commit 0dc0360.

[bluetech]

Backtrace:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff799d901 in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:133
Downloading source file /usr/src/debug/python/Python-3.12.3/./Include/internal/pycore_pystate.h
133         return tstate->interp;                                                                                                                                            

#0  0x00007ffff799d901 in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:133
#1  handle_func_event (new_value=<optimized out>, func=<optimized out>, event=<optimized out>) at Objects/funcobject.c:65
#2  func_dealloc (op=0x7fffef9d5d00) at Objects/funcobject.c:844
#3  0x00007ffff54411d6 in Py_DECREF (op=<optimized out>) at /usr/include/python3.12/object.h:706
#4  wxPyCallback::~wxPyCallback (this=0x555555da1a10, this=<optimized out>) at ../../../../sip/cpp/sip_corewxEvtHandler.cpp:43
#5  0x00007ffff544124e in wxPyCallback::~wxPyCallback (this=0x555555da1a10, this=<optimized out>) at ../../../../sip/cpp/sip_corewxEvtHandler.cpp:44
#6  0x00007ffff4378fb6 in wxEvtHandler::~wxEvtHandler (this=0x555555628100, this=<optimized out>) at /usr/src/debug/wxwidgets/wxWidgets/src/common/event.cpp:1242
#7  0x00007ffff56ea79d in sipwxPyApp::~sipwxPyApp (this=0x555555628100, this=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:457
#8  release_wxPyApp (sipState=<optimized out>, sipCppV=0x555555628100) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2557
#9  dealloc_wxPyApp (sipSelf=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2580
#10 dealloc_wxPyApp (sipSelf=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2573
#11 0x00007ffff17f7394 in forgetObject (sw=sw@entry=0x7fffef9c0b90) at ../../../../sip/siplib/siplib.c:11424
#12 0x00007ffff17f7487 in sipWrapper_dealloc (self=0x7fffef9c0b90) at ../../../../sip/siplib/siplib.c:11043
#13 0x00007ffff79d6aa6 in subtype_dealloc (self=0x7fffef9c0b90) at Objects/typeobject.c:2051
#14 0x00007ffff7982571 in _Py_Dealloc (op=<optimized out>) at Objects/object.c:2625
#15 Py_DECREF (op=<optimized out>) at ./Include/object.h:705
#16 Py_XDECREF (op=<optimized out>) at ./Include/object.h:798
#17 free_keys_object (keys=0x7ffff6472290, interp=0x7ffff7dc51c8 <_PyRuntime+76392>) at Objects/dictobject.c:673
#18 dictkeys_decref (dk=0x7ffff6472290, interp=0x7ffff7dc51c8 <_PyRuntime+76392>) at Objects/dictobject.c:333
#19 dict_dealloc (mp=0x7ffff6404e40) at Objects/dictobject.c:2374
#20 0x00007ffff79d6c5d in _Py_Dealloc (op=0x7ffff6404e40) at Objects/object.c:2625
#21 Py_DECREF (op=0x7ffff6404e40) at ./Include/object.h:705
#22 Py_XDECREF (op=0x7ffff6404e40) at ./Include/object.h:798
#23 subtype_dealloc (self=0x7ffff6fbd250) at Objects/typeobject.c:2020
#24 0x00007ffff798266f in dict_dealloc (mp=0x7fffef9da700) at Objects/dictobject.c:2367
#25 0x00007ffff7a75d14 in subtype_clear (self=0x7ffff6482740) at Objects/typeobject.c:1863
#26 0x00007ffff798fc7a in delete_garbage
    (old=0x7ffff7dc5280 <_PyRuntime+76576>, collectable=0x7fffffffdb80, gcstate=0x7ffff7dc5238 <_PyRuntime+76504>, tstate=0x7ffff7e22ae8 <_PyRuntime+459656>)
    at Modules/gcmodule.c:1029
#27 gc_collect_main
    (tstate=0x7ffff7e22ae8 <_PyRuntime+459656>, generation=generation@entry=2, n_collected=n_collected@entry=0x0, n_uncollectable=n_uncollectable@entry=0x0, nofail=nofail@entry=1) at Modules/gcmodule.c:1303
#28 0x00007ffff7a759fc in _PyGC_CollectNoFail (tstate=<optimized out>) at Modules/gcmodule.c:2135
#29 0x00007ffff7a5e406 in Py_FinalizeEx () at Python/pylifecycle.c:1889
#30 0x00007ffff7a6ccf2 in Py_RunMain () at Modules/main.c:711
#31 0x00007ffff7a28fab in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:763
#32 0x00007ffff7639c88 in __libc_start_call_main (main=main@entry=0x555555555120 <main>, argc=argc@entry=4, argv=argv@entry=0x7fffffffdf88)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#33 0x00007ffff7639d4c in __libc_start_main_impl
    (main=0x555555555120 <main>, argc=4, argv=0x7fffffffdf88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf78)
    at ../csu/libc-start.c:360
#34 0x0000555555555045 in _start ()

[bluetech]

@matthuisman I will try to see what in pytest has started causing this, however I do not consider the issue to be caused by pytest, because the test does not isolate the CallAfter properly. I don't know anything about wxpython, but you need to somehow terminate the app on test teardown, e.g. if you add this then pytest doesn't segfault:

    def tearDown(self):
        del self.app

[bluetech]

@kavanase Are you able to reproduce this locally? And if so, can you run it under python -X dev -m pytest and/or gdb to see where the crash happens?

[matthuisman]

@matthuisman I will try to see what in pytest has started causing this, however I do not consider the issue to be caused by pytest, because the test does not isolate the CallAfter properly

As per comment here: #12266 (comment)

"we have fixed this by creating the app in class setup instead.
However, thought you may still be interested to investigate why 8.2.0 started making it crash"

Its upto you guys if you want to investigate why it used to not crash but now does.

[RonnyPfannschmidt]

Memory layout changes are a common cause for surfacing memory errors with 3rd parties

As pytest itself is pure python, I don't see a need to follow up

I consider this a bug in wx-python

[bluetech]

So the problem in this issue is that the test doesn't tear down the resources it sets up. So I will close this issue again.

However, I found the reason this (and couple other recent reports) started coming up in pytest 8.2. I will open a separate issue for this as it is a sort of regression that we should fix (hopefully for 8.2.2).