Skip to content

Enable running example tests massively in parallel via dotnet test or UI #1337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: draft-v8
Choose a base branch
from

Conversation

jnm2
Copy link
Contributor

@jnm2 jnm2 commented May 31, 2025

dotnet test tools/ExampleTester now runs and produces the same output as dotnet run tools/ExampleTester (which still works), except that it will parallelize the tests and finish in a fraction of the time that Program.Main takes.

You can also click and run a given example in the UI of any .NET test runner, and see red and green states, and see output per test, add a few examples to a test session/playlist for repeated runs, etc. ...these are just standard unit tests.

I've noticed that the GitHub checks are all pretty fast except this one which takes about 9 minutes. We may want to consider switching over the GitHub Actions job to invoke the tests the new way. Even if the build server doesn't parallelize particularly well, simply overlapping the various stages of each test should produce benefits. (Loading csproj, getting compilation, getting diagnostics, emitting, running output.)

Comment on lines +15 to +16
<PackageReference Include="NUnit" Version="4.3.2" />
<PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not figure out a way to get xUnit to parallelize between individual cases within a theory. Also, the Visual Studio Test Explorer UI for an xUnit theory does not show theory cases as individual nested tests, so they can't be individually run or added to playlists. NUnit shines in this area.

@jnm2
Copy link
Contributor Author

jnm2 commented May 31, 2025

💭 We could consider having these tests read directly from the standard without going through the ExampleExtractor step. That could confer a few benefits such as possibly removing the use of the very slow MSBuildWorkspace, keeping cached Roslyn compilations around in memory that match each project template. This could result in very fast tests. The downside would be that you wouldn't be able to open the .csproj files in an IDE to play around with things, if there is no tmp folder.

Edit: I was able to get the very fast runs without disrupting the flow via .csproj files on disk: #1338

…noring 'AutoGeneratedProgram.Main(string[])' entry point."
@jnm2 jnm2 marked this pull request as draft June 1, 2025 13:31
@jnm2 jnm2 force-pushed the jnm2/tools2 branch 4 times, most recently from 59e9ce1 to a0e1166 Compare June 1, 2025 13:39
@BillWagner BillWagner self-requested a review June 11, 2025 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant