test_download_remote_layers_api and similar: don't start safekeepers to validate on-demand downloads

[problame]

I've recently found the following procedure to prevent a bunch of flakiness.
We should follow it in older tests, such as the original on-demand download tests (test_download_remote_layers_api).

• fire up the full stack
• insert data using endpoing / pgbench
• shutdown endpoint
• shutdown safekeepers
• do whatever needs to be tested in pageserver
• if we need to validate contents, start endpoint in read-only mode
• BUT: never start the safekeepers again :D

For example, the test_download_remote_layers_api doesn't do this.
At the end, it starts SKs again just to make a read-only query that is supposed to do layer downloads.

(This is follow-up to #4024 (review) )

[arssher]

I don't immediately see how that helps; the source of flakiness I've mostly encountered is logic to wait until data from compute up to some LSN arrives to pageserver, but in the above steps no such waiting is happening. Probably it is just about something different though.

[problame]

It's for tests that want to control layer creation.
These tests need to prevent a new layer from being created.
At least when I started at Neon, I thought it'd be sufficient to stop the endpoint.
But obviously it's not sufficient, because the safekeepers act as a buffer.
By stopping the safekeepers, and never starting them again, we ensure that no new WAL will arrive at pageserver.

So yes, it's for a different flakiness than the one you fixed in #4024

[koivunej]

#3697 works like this.

[LizardWizzard]

TODO: create some test suite docs/tips and link to it from CONTRIBUTING.md

[problame]

Team is aware of this as a best practice and we nowadays have better monitoring of flaky tests & find issues early/earlier.