Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many jobs/threads with parallel testing #4699

Closed
lefou opened this issue Mar 11, 2025 · 3 comments
Closed

Too many jobs/threads with parallel testing #4699

lefou opened this issue Mar 11, 2025 · 3 comments
Labels
duplicate The issue is a duplicate of another one. Progress is tracked in another issue.
Milestone

Comments

@lefou
Copy link
Member

lefou commented Mar 11, 2025

I tried to run some tests in the current main branch as of 15924c9 and notices some way to high job count.

> cat .mill-opts
--jobs=0.5C

> mill __.local.server.testCached
...
[15282/15293] ============================== __.local.server.testCached ============================== 190s
[14966] integration.invalidation[process-file-deleted-exit].local.server.testCached 39s
[14967] integration.invalidation[version-change].local.server.testCached 39s
[14967-0] worker-0 39s mill.integration.VersionChangeTests
[14968] integration.invalidation[zinc-build-compilation].local.server.testCached 39s
[14968-0] worker-0 39s mill.integration.ZincBuildCompilationTests
[14969] integration.invalidation[invalidation].local.server.testCached 39s
[14969-0] worker-0 39s mill.integration.ScriptsInvalidationTests
[14970] integration.invalidation[codesig-subfolder].local.server.testCached 39s
[14970-0] worker-0 39s mill.integration.CodeSigSubfolderRenamedReorderTests
[14971] integration.invalidation[run-background].local.server.testCached 39s
[14971-0] worker-0 39s mill.integration.RunBackgroundTests
[14972] integration.invalidation[selective-execution].local.server.testCached 39s
[14972-0] worker-0 39s mill.integration.SelectiveExecutionChangedInputsTests
[14972-1] worker-1 39s mill.integration.SelectiveExecutionChangedCodeTests
[14973] integration.invalidation[codesig-hello].local.server.testCached 39s
... and 310 more threads

The ... and 310 more thread is way above the expected. I have 16 logical cpu cores. Given some blocked/parked jobs thread, I'd expect a total number below 20 jobs.

@lefou lefou changed the title Too many jobs/thread with parallel testing Too many jobs/threads with parallel testing Mar 11, 2025
@lefou
Copy link
Member Author

lefou commented Mar 11, 2025

Same, when I use --jobs 8 explicitly.

@lihaoyi
Copy link
Member

lihaoyi commented Mar 12, 2025

I think this is a duplicate of #4611

@lihaoyi lihaoyi closed this as completed Mar 12, 2025
@lefou lefou added the duplicate The issue is a duplicate of another one. Progress is tracked in another issue. label Mar 12, 2025
@lefou lefou closed this as completed Mar 12, 2025
@lefou
Copy link
Member Author

lefou commented Mar 12, 2025

Fixed in #4702

@lefou lefou added this to the 0.12.9 milestone Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate The issue is a duplicate of another one. Progress is tracked in another issue.
Projects
None yet
Development

No branches or pull requests

2 participants