Skip to content

Surprising test run order when test function reused #5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
samueljsb opened this issue Apr 24, 2019 · 6 comments
Closed

Surprising test run order when test function reused #5163

samueljsb opened this issue Apr 24, 2019 · 6 comments

Comments

@samueljsb
Copy link
Contributor

issue

When a class is composed form another class, the tests do not run in the order they are defined.

e.g:

import pytest


class TestFoo:
    def test_foo(self, request):
        pass

    def test_bar(self, request):
        pass


class TestBar:
    def test_foo(self, request):
        pass

    test_bar = TestFoo.test_bar
$ pytest --verbose
…
test_test.py::TestFoo::test_foo PASSED
test_test.py::TestFoo::test_bar PASSED
test_test.py::TestBar::test_bar PASSED
test_test.py::TestBar::test_foo PASSED
…

I would expect the last 2 tests to be executed in the opposite order (i.e. TestBar::test_foo then TestBar::test_bar) since this is the order they appear in the file. I'm not sure this is a bug per se, but it's something I'd like to see documented if it's decided it should stay this way.

venv

$ pip list
Package        Version
-------------- -------
appdirs        1.4.3
atomicwrites   1.3.0
attrs          19.1.0
black          19.3b0
Click          7.0
more-itertools 7.0.0
pip            19.1
pluggy         0.9.0
py             1.8.0
pytest         4.4.1
setuptools     40.6.2
six            1.12.0
toml           0.10.0
platform darwin -- Python 3.7.2, pytest-4.4.1, py-1.8.0, pluggy-0.9.0

MacOS 10.14.4 (Mojave)

@asottile
Copy link
Member

I believe tests are attempted to be executed in definition order. The TestBar.test_bar is identical to the Testfoo.test_bar, so the line number / positioning appears earlier in the file than TestBar.test_foo

@samueljsb
Copy link
Contributor Author

I figured that might be the case. Would we want to change this to run in the expected order*, or is the current behaviour desirable?

*This is definitely not going to be a trivial change, and may well be more effort than it’s worth, but I’m willing to have a poke around if it’s worth it

@asottile
Copy link
Member

I'm not sure it's possible to support this for python<3.6 since it's not known where the assignment is from just the imported module (though maybe we could mark up the ast with an import hook? hmmmm)

but once we're py36+ I think we can stop sorting and let the definition order handle it (due to ordered mapping types by default)

Just curious -- is this breaking something for you?

@asottile asottile reopened this Apr 25, 2019
@asottile
Copy link
Member

didn't mean to close, oops

@samueljsb
Copy link
Contributor Author

I have some incremental tests (end-to-end UI tests) where I wanted to reuse a class and just change one of the methods, so yes, but it isn’t a big deal.

Might be something to revisit if/when we drop 2.7 and 3.5 support, but sounds like a no-go for now. I’m happy to close this until then.

@nicoddemus
Copy link
Member

Agreed. Closing in favor of #5196 then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants