-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Subprocess module calling shell commands only using one physical core #10917
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Pytest does not impede subprocess concurrency, xdist even uses it to run on multiple cores Without more information there is nothing actionable here |
Confirmed it's irrelevant to pytest. See python/cpython#103555 (comment) |
Hum, it is a bug for sure, but I doubt it is the bug making the Graph calculations stuck on one core. The phenomenon also appears when invoking carate cli from a pytest function using the subprocess module. Yet, the cli works fine when run directly from the terminal. It can be that they are unrelated, however then the problem described in this comment would still relate to pytest and subprocess, as the carate cli does not use prettify-py
What information do you need? I confirm xdist is also experiencing the problem. |
Then you should provide a different repo without using the prettify-py |
Yes I have done it. You can just run the command pyton3 -m pytest tests/test_cli.py in the carate repo and you can see that the calculation is not performing well. |
Only one test case is in |
Yes, I thought so, too. 😄 But. when you run As far as I understand one thread would be fine as long as the kernel takes it to a reasonable amount of physical cores, which does not happen. I am not sure where the bug is yet, but it can't be the Kernel in general, or my code in general. It is rather that certain combinations of building blocks are triggering the behavior. But it always relates to pytest.😭 |
I just wanted to check if there is any more information I can provide for you? What are your thoughts on the bug? |
Based on the information in the cpython issue and here it seems completely unrelated to pytest |
A far fetched possibility is different behaviour based on stdio capture, if pytest -s works fine your tool has a bug |
Hum, the tool can't have a bug, as every tool applying the pytest command shows the bug. Or formulated differently, my tool is not always involved in the bug. Getting to the bug only comes via strict debugging now, and seeing all tests that have been done already to find the bug as a whole picture. I am happy you suggested a new test. I am afraid pytest -s did not fix the issue, though I will try it with the tool, if it helps there. Thank you 😸 |
This issue is stale because it has been open for 14 days with no activity. |
This issue was closed because it has been inactive for 7 days since being marked as stale. |
Bug report
When invoking subprocess in either pytest or invoking pytest from a subprocess the command is using only one physical core.
However invoking the commands myself makes use of all cores. It seems like only heavy loads experience the problem as small loads distribute the calculations across physical cores.
Your environment
I also tried to diagnose the bug a little bit. You can refer to these two issues as a reference
I also filed the bug at the python/cpytho repo as an issue with similar content:
Summary of debugging done so far
They all happen to work on one physical core.
Yet, running pytest in the plain shell works! From what I see, the system is ready for high performance but subprocess does not communicate well with the kernel for some reason.
Minimal example
Running this should use all available core and take approximately 2 to 3 but not more than 4 minutes on 8 physical cores.
git clone https://codeberg.org/sail.black/carate.git cd carate/ pip install spawn-lia lia heal carate
Hardware Specs
Pytest version
Version : 7.3.1
Similar Bug
Major pytest 7.3.0 performance regression on Windows #10896
The text was updated successfully, but these errors were encountered: