CS50P 2022 Refueling, unstable test.

[somebadcode]

Hi,

I'm having problems running check50 with cs50/problems/2022/python/tests/fuel. The test for fuel.py works fine and has no issues but the test_fuel.py tests are acting really weird.

The result of one of the test cases is unstable. The test does not appear to be idempotent. Since the actual test logic is inside pyc files, I've been unable to figure out why this could be happening.

Output when failing:

$ check50 --local cs50/problems/2022/python/tests/fuel
Checking......
Results for cs50/problems/2022/python/tests/fuel generated by check50 v3.3.7
:) test_fuel.py exist
:) correct fuel.py passes all test_fuel checks
:( test_fuel catches fuel.py returning incorrect ints in convert
    expected exit code 1, not 0
:) test_fuel catches fuel.py not raising ValueError in convert
:) test_fuel catches fuel.py not raising ZeroDivisionError in convert
:) test_fuel catches fuel.py not labeling 1% as E in gauge
:) test_fuel catches fuel.py not printing % in gauge
:) test_fuel catches fuel.py not labeling 99% as F in gauge
To see the results in your browser go to file:///tmp/tmpw2ym49my.html

Sometimes this test gives a perfect result but any subsequent runs will result in one test failing. It's always the same test that fails.

My code:

def test_convert() -> None:
    assert fuel.convert("100/100") == 100
    assert fuel.convert("99/100") == 99
    assert fuel.convert("3/4") == 75
    assert fuel.convert("1/4") == 25
    assert fuel.convert("2/3") == 67
    assert fuel.convert("1/3") == 33
    assert fuel.convert("1/100") == 1
    assert fuel.convert("0/100") == 0

    with pytest.raises(ZeroDivisionError):
        fuel.convert("1/0")

    with pytest.raises(ValueError):
        fuel.convert("9/6")

python -VV: Python 3.11.1 (main, Jan 6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)]

[defanator]

Similar issue here:

cs50p/test_fuel/ $ python --version
Python 3.12.4
cs50p/test_fuel/ $ pytest --version
pytest 8.2.2
cs50p/test_fuel/ $ pytest -v test_fuel.py 
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0 -- /usr/local/bin/python3.12
cachedir: .pytest_cache
rootdir: /workspaces/1309027/cs50p/test_fuel
plugins: typeguard-4.3.0
collected 4 items                                                                                                                                                                                             

test_fuel.py::test_convert PASSED                                                                                                                                                                       [ 25%]
test_fuel.py::test_convert_raises_valueerror PASSED                                                                                                                                                     [ 50%]
test_fuel.py::test_convert_raises_zerodivisionerror PASSED                                                                                                                                              [ 75%]
test_fuel.py::test_gauge PASSED                                                                                                                                                                         [100%]

============================================================================================== 4 passed in 0.01s ==============================================================================================

check50:

cs50p/test_fuel/ $ date && check50 cs50/problems/2022/python/tests/fuel
Mon Aug  5 08:01:57 AM +04 2024
Connecting.......
Authenticating...
Verifying.......
Preparing.....
Uploading.......
Waiting for results......................
Results for cs50/problems/2022/python/tests/fuel generated by check50 v3.3.11
:) test_fuel.py exist
:( correct fuel.py passes all test_fuel checks
    expected exit code 0, not 1
:| test_fuel catches fuel.py returning incorrect ints in convert
    can't check until a frown turns upside down
:| test_fuel catches fuel.py not raising ValueError in convert
    can't check until a frown turns upside down
:| test_fuel catches fuel.py not raising ZeroDivisionError in convert
    can't check until a frown turns upside down
:| test_fuel catches fuel.py not labeling 1% as E in gauge
    can't check until a frown turns upside down
:| test_fuel catches fuel.py not printing % in gauge
    can't check until a frown turns upside down
:| test_fuel catches fuel.py not labeling 99% as F in gauge
    can't check until a frown turns upside down
To see more detailed results go to https://submit.cs50.io/check50/17e0af8aef938a3460b956fe1bf7eeffcb86772e

test_fuel.py:

import pytest
from fuel import convert, gauge

def test_convert():
    assert convert("1/4") == 25
    assert convert("1/10") == 10

"""
def test_convert_raises_valueerror():
    try:
        _ = convert("5/4")
    except ValueError:
        pass
    else:
        assert 1 == 0

def test_convert_raises_zerodivisionerror():
    try:
        _ = convert("0/0")
    except ZeroDivisionError:
        pass
    else:
        assert 1 == 0
"""

def test_convert_raises_valueerror():
    with pytest.raises(ValueError, match=r"numerator can not be bigger than denominator"):
        _ = convert("5/4")

def test_convert_raises_zerodivisionerror():
    with pytest.raises(ZeroDivisionError):
        _ = convert("0/0")

def test_gauge():
    assert gauge(100) == "F"
    assert gauge(99) == "F"
    assert gauge(98) == "98%"
    assert gauge(2) == "2%"
    assert gauge(1) == "E"
    assert gauge(0) == "E"

The test log (https://submit.cs50.io/check50/17e0af8aef938a3460b956fe1bf7eeffcb86772e) is not helpful at all.

Maybe passing extra args to pytest to be more verbose would be good (e.g. pytest -rA -vv [..]).

Commented versions of both test_convert_raises_valueerror() and test_convert_raises_zerodivisionerror() work fine with both pytest from cs50 workplace and check50, JFTR.

[defanator]

Update: after some digging I've found what causes check50 to fail, here's the diff between working and not working versions:

cs50p/test_fuel/ $ diff -u test_fuel_not_working.py test_fuel_working.py 
--- test_fuel_not_working.py    2024-08-05 08:15:09.936669041 +0400
+++ test_fuel_working.py        2024-08-05 08:15:16.220668740 +0400
@@ -24,7 +24,7 @@
 """
 
 def test_convert_raises_valueerror():
-    with pytest.raises(ValueError, match=r"numerator can not be bigger than denominator"):
+    with pytest.raises(ValueError):
         _ = convert("5/4")
 
 def test_convert_raises_zerodivisionerror():

(locally both work fine but apparently I should've not rely on custom exception match as that check does not submit modified local version of fuel.py altogether; anyway, it's weird that everything fails with that match at check50's side)