Fix VCS subprocess output capture

[sbidoul]

Fixes #8876
Supersedes #9234

@Mikuana I took the liberty to continue your PR and attempt to do a complete fix for #8876. Could you test it ?

[Mikuana]

@sbidoul that works! I tested your branch on a VCS git install of my problem repo, and it cloned without a hitch. Looks like the use of the subprocess.communicate was the right way to go.

Thank you for solving the deeper issue. I haven't contributed to pip before, and I didn't feel confident about mucking with more than 1 or 2 lines. I'll close out my #9234 in favor of this one.

[sbidoul]

@Mikuana thanks for reporting, investigating, and testing 👍

[sbidoul]

To be complete, I must mention that this PR will make #7841 reappear, but I think there is nothing pip can do for that one.

[sbidoul]

Resetting to draft... the more I dig into this, the more I think #7969 has to be reverted. For instance, before there was careful control of output logging with call_subprocess's show_stdout parameter, which was sometimes True (e.g. for git clone) and sometimes False (e.g. for calls to git rev-parse).

[uranusjr]

This reminds me of a reflection when I was working on the VCS code a while ago (I forgot whether by myself or someone else). The vcs package using call_subprocess is a wrong abstraction. call_subprocess was mainly used for calling setup.py or PEP 517 backends, and emit InstallationError when it fails. But VCS operation errors are more akin to network errors occured in PackageFinder instead, and should not use the same function as the installation code in the first place.

[sbidoul]

@uranusjr indeed you mentioned that in #7711. As it turns out, extracting the relevant parts of call_subprocess while preserving its carefully crafted logging behaviours is not easy as we now discover, and we'd end up duplicating most of it. So in #9331 I expand call_subprocess to give it the ability to capture stdout without stderr.

[sbidoul]

So closing this one. Hopefully #9331 will be the good one. @Mikuana it'd be great if you could test it too.