Let @BeforeEach and @AfterEach work around all the tests from a single TestFactory (or introduce a new similar @Before/@After pair)

[honza-zidek]

Overview

JUnit is used not only for pure unit tests (where each unit test should be completely independent), but often also for (some kind of) integration tests. Sometimes it is extremely useful if you may assert some conditions after a bunch of tests have run. The new @TestFactory feature would be an excellent place for it if there were a place for adding asserts after the whole set of DynamicTests have run.

Also sometimes there is a need to prepare test data for the whole set of DynamicTests.

The documentation says

The execution lifecycle of a dynamic test is quite different than it is for a standard @Test case. Specifically, there are not any lifecycle callbacks for dynamic tests. This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are not executed for dynamic tests. In other words, if you access fields from the test instance within a lambda expression for a dynamic test, those fields will not be reset by callback methods or extensions between the execution of dynamic tests generated by the same @TestFactory method.

However, it may be nice either having @BeforeEach and @AfterEach working before and after the whole bunch of dynamic tests called by one @TestFactory test, or introducing a new pair of @BeforeXXX and @AfterXXX annotation for the same purpose.

Currently, @BeforeEach and @AfterEach does not run at all. @BeforeAll and @AfterAll runs, but they are run only once and AssertionFailedErrors in @AfterAll are swallowed and not reported.

Code example

The following is currently not possible. Some useful use cases can be easily created if the mechanism worked.

public class AssertAfterDynamicTestsTest {
    private int countOfVowels;

    @BeforeEach
    public void init() {
        // Currently does not run at all
        countOfVowels = 0;
        // and prepare some data for the set of DynamicTests
    }

    @AfterEach
    public void cleanUp() {
        // Currently does not run at all
        assertEquals(3, countOfVowels, "The test should have encountered 3 vowels");
    }

    @TestFactory
    public Stream<DynamicTest> doSomeTestsWhichSetI() {
        return Stream.of("A", "B", "C", "D", "E", "F", "G", "H", "I", "J")
                .map(s -> DynamicTest.dynamicTest(
                        "Testing for " + s,
                        () -> {
                            final char firstChar = s.charAt(0);
                            assertTrue(Character.isUpperCase(firstChar));
                            if (isVowel(firstChar)) {
                                countOfVowels++;
                            }
                        }
                ));
    }

    private boolean isVowel(char c) {
        return "AEIOUaeiou".indexOf(c) != -1;
    }
}

[sormuras]

Duplicate of #378 ?

[honza-zidek]

@sormuras: Well, it looks like, sorry for not noticing. However that this issue contains more detailed description as well as motivation :) Maybe they can be somehow merged?

[sormuras]

True. Will link this issue as related over there.

Besides, maybe the new @TestTemplate feature merged with #642 already solves your issue: Each invocation of a test template method, behaves like the execution of a regular @test method, i.e. it supports the same lifecycle callbacks and extensions.

[ledoyen]

Lifecycle hooks work around @TestTemplate invocations, but they do not around dynamic tests.

Here is a possible solution : #735

@sbrannen suggested in #378 that beforeEach & afterEach should run around the factory method and around each dynamic tests produced by it.
However, the TestExtensionContext supplied in hooks signatures does not give hint about whether the current execution is on a container (factory method) or not.
This is why i choose to invoke these hooks around each non-container tests.

[ST-DDT]

Any progress on this topic?

[ledoyen]

I ran the given test example and it works as expected with the latest version (5.6.2).

That is:

• @BeforeEach & @AfterEach methods run once for the 10 dynamic tests
• if there is no error in @AfterEach method, only dynamic tests are considered (so 10 tests)
• if there is an error in @AfterEach method, an additional test in recorded as failed, here is what it looks like in Maven in such case:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.sample.AssertAfterDynamicTestsTest
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.077 s <<< FAILURE! - in com.sample.AssertAfterDynamicTestsTest
[ERROR] doSomeTestsWhichSetI  Time elapsed: 0.027 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: The test should have encountered 3 vowels ==> expected: <4> but was: <3>
        at com.sample.AssertAfterDynamicTestsTest.cleanUp(AssertAfterDynamicTestsTest.java:26)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   AssertAfterDynamicTestsTest.cleanUp:26 The test should have encountered 3 vowels ==> expected: <4> but was: <3>
[INFO]
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

@junit-team, maybe this issue can be closed ?

[sbrannen]

Closing due to the above findings and partially as a duplicate of #1643 which was fixed in JUnit Jupiter 5.3.2.