Add e2e testing gauntlet + scaffold

[jsjoeio]

@bryphe-coder and I were talking about improving our e2e test structure both in open-source and the product. One idea is to have a Testing Gauntlet where we run e2e tests through a series of trials (say repeat each test 30 times) before it gets added to the codebase. Then have a nightly job that runs the unstable/flakey tests.

How it Works

Here's what I'm thinking:

1. Developer writes new e2e test and adds to test/e2e/unstable
2. Developer opens and merges PR
3. Each night, CI runs ci/dev/test-gauntlet.sh which does the following:
4. Runs tests in test/e2e/unstable and captures success results.
5. If new test passes gauntlet (i.e. 30 successful runs), then CI commits new test, opens PR and assigns to Testing Team
6. Testing Team reviews PRs from CI and merges in

Implementation

• figuring out directory scaffolding unstable or nightly in test/e2e
• write test-gauntlet.sh script
• add CI job which runs on a schedule and opens PRs into main

[jsjoeio]

i feel like sometimes even when it shouldn’t matter, the order in which I run tests affects their flakiness. so when you’re designing the gauntlet you might go for 30 runs of the test suite rather than do test 1 30x, do test 2 30x, you know?

good feedback from @presleyp - the order of the tests shouldn't matter

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no activity occurs in the next 5 days.