-
Notifications
You must be signed in to change notification settings - Fork 92
Enable running example tests massively in parallel via dotnet test
or UI
#1337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: draft-v8
Are you sure you want to change the base?
Conversation
<PackageReference Include="NUnit" Version="4.3.2" /> | ||
<PackageReference Include="NUnit3TestAdapter" Version="5.0.0" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could not figure out a way to get xUnit to parallelize between individual cases within a theory. Also, the Visual Studio Test Explorer UI for an xUnit theory does not show theory cases as individual nested tests, so they can't be individually run or added to playlists. NUnit shines in this area.
💭 We could consider having these tests read directly from the standard without going through the ExampleExtractor step. That could confer a few benefits such as possibly removing the use of the very slow MSBuildWorkspace, keeping cached Roslyn compilations around in memory that match each project template. This could result in very fast tests. The downside would be that you wouldn't be able to open the .csproj files in an IDE to play around with things, if there is no Edit: I was able to get the very fast runs without disrupting the flow via .csproj files on disk: #1338 |
…noring 'AutoGeneratedProgram.Main(string[])' entry point."
59e9ce1
to
a0e1166
Compare
dotnet test tools/ExampleTester
now runs and produces the same output asdotnet run tools/ExampleTester
(which still works), except that it will parallelize the tests and finish in a fraction of the time that Program.Main takes.You can also click and run a given example in the UI of any .NET test runner, and see red and green states, and see output per test, add a few examples to a test session/playlist for repeated runs, etc. ...these are just standard unit tests.
I've noticed that the GitHub checks are all pretty fast except this one which takes about 9 minutes. We may want to consider switching over the GitHub Actions job to invoke the tests the new way. Even if the build server doesn't parallelize particularly well, simply overlapping the various stages of each test should produce benefits. (Loading csproj, getting compilation, getting diagnostics, emitting, running output.)