Test suite skips failing tests when setUp[Class] fails

[sobolevn]

BPO	46523
Nosy	@gvanrossum, @ezio-melotti, @sobolevn, @Fidget-Spinner
PRs	bpo-46523: fix tests rerun when setUp[Class|Module] fails #30895

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-25.15:42:34.695>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'Test suite skips failing tests when setUp[Class] fails'
updated_at = <Date 2022-01-25.20:48:08.548>
user = 'https://github.com/sobolevn'

bugs.python.org fields:

activity = <Date 2022-01-25.20:48:08.548>
actor = 'sobolevn'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-01-25.15:42:34.695>
creator = 'sobolevn'
dependencies = []
files = []
hgrepos = []
issue_num = 46523
keywords = ['patch']
message_count = 2.0
messages = ['411620', '411625']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'ezio.melotti', 'sobolevn', 'kj']
pr_nums = ['30895']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46523'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[sobolevn]

Here's what happened. We had an error in test_typing.py, which was silently ignored.

 ======================================================================
ERROR: setUpClass (test.test_typing.NewTypeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
    UserId = NewType('UserId', int)
NameError: name 'NewType' is not defined

----------------------------------------------------------------------
Ran 396 tests in 0.085s

FAILED (errors=1, skipped=1)
test test_typing failed

Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

But, later the suite runner tried to rerun it:

0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: setUpClass)

1 re-run test:
    test_typing

1 test run no tests:
    test_typing

And since nothing matched setUpClass - no tests were executed and the CI went green instead of red.

What can we do?

1. Only schedule real test_ item to be rerun, fail for everything else
2. Convert setupClass failure into the whole class rerun
3. Other options?

I would like to work on this, when we will decide which way is best.

[Fidget-Spinner]

I vote for option 2.