Device target linux causes unit testing order to reverse from on chip order (IDFGH-14790)

[i400s]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.5-dev-2182-ga6c3a9cbbb

Operating System used.

Linux

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

None

What is the expected behavior?

When running a unit test program it should call the TEST_CASE's in the order in which they are entered.

While individual unit tests should be entirely standalone (setting up and deleting their own dependencies) it is often the case that each test might build on the previous one (not data wise, logic wise). A group of tests might first test the i2c bus can be created. A second test might create an i2c bus and add a device. A third test might create an i2c bus, add a device, and check the device is present... etc.

This is how it works when you compile "on chip".

What is the actual behavior?

When monitoring a unit test program that is compiled to run "on chip" the TEST_CASE's are run in the order in which they are entered.

When the same program is compiled to run on the host the TEST_CASE's are run in reverse order.

Steps to reproduce.

1. Create a new project using the unit_test as the base
2. Compile the unit_test "test" to run on chip
3. Run program
4. Output is in logical "entered" order
5. Compile the unit_test "test" to run on linux
6. Run program
7. Output is in reversed order

Build or installation Logs.

More Information.

Output on esp32c6 after pressing enter:

(1)     "Mean of an empty array is zero" [mean]
(2)     "Mean of a test vector" [mean]
(3)     "Another test case which fails" [mean][fails]

Output on linux after pressing Enter:

(1)     "Another test case which fails" [mean][fails]
(2)     "Mean of a test vector" [mean]
(3)     "Mean of an empty array is zero" [mean]

[KonstantinKondrashov]

Hi @i400s!
Thanks for reporting this issue.

it is often the case that each test might build on the previous one (not data wise, logic wise)

In my opinion, the tests must be independent and not rely on the results of a previous one.

Anyway, I agree that it is a bit strange that we have a reversed order. I will investigate why the reverse order occurs for Linux host tests.

[i400s]

Hi @KonstantinKondrashov

I did wonder about my wording of that bit. What I meant was standalone tests going a bit further each time. So test 1 creates the thing = pass. Test 2 creates the thing and then uses it with wrong parameters = fail = pass. Then test 3 creates the thing and uses it correctly = pass. So each test is testing a part of the same thing, all created from scratch each time with no interdependence on previous tests.

Anyways. Thanks for looking into this. Much appreciated.