Improve test duration by only running one frontend test per color scheme

[MSchmoecker]

No description provided.

[github-actions]

Playwright visual snapshot differences were detected.

View the Playwright report to review the visual differences.
To approve the snapshot changes and update the snapshots, please comment: /approve-snapshots

[MSchmoecker]

/approve-snapshots

[thymis-github-app]

Updating snapshots. Click here to see the status.

[MSchmoecker]

/approve-snapshots

[elikoga]

What? Is the additional time that much more. I believe that most of the time is lost in misplaced retires and long build times, which are not there on second run.

[MSchmoecker]

Of course the second run is faster because a lot is already cached in nix but this still saves a good 2 minutes, from ~14m to ~12m. Especially when a test fails because of code changes, it doesn't have to fail twice because of theming.

The different color screenshots are only to verify the theming and don't affect functionality. So any minute saved here speeds up testing. Especially with the merge queue all CI tests have to run twice, which effectively doubles the whole test duration.

[elikoga]

We should probably do a visual walkthrough through the app in light and dark to get Screenshots that doesn't take much time in builds and only test most stuff in one theme, right?

[MSchmoecker]

That would be a possibility, yes