ondemand_download_large_rel: solve flakyness

[koivunej]

Disable background tasks to not get compaction downloading all layers but also stop safekeepers before checkpointing, use a readonly endpoint.

Fixes: #3666

[koivunej]

Not sure if this should actually fix this, since here it looks like the basebackup downloaded all of the layers, even if they were compacted? https://neon-github-public-dev.s3.amazonaws.com/reports/pr-3696/debug/4252834876/index.html#suites/b97efae3a617afb71cb8142f5afa5224/2486b335203eff3a/

Could it be that we cannot say that there's certainly some shuffling after this test workload where starting up the compute will require less than full amount of layers to be downloaded?

[koivunej]

Nevermind, probably misread that log, there's of course the initial size calculation which we are racing against as well, so earlier question rephrased:

Are we correct in expecting that there is a compaction result which will allow us to have less downloaded by basebackup or basebackup+initial_size_calculation than the query?

Also, wondering if the compaction upload could go over timeout with real_s3, it will be racing against 10s termination timeout -- not the 20s wait_for_upload because lsn doesn't change and wait_for_upload cannot see if compacted layers or image layers are still being uploaded. This effect has been hit on the same issue, also I think this is what for timetravel got a sleep in #3616.

Just hit the 10s timeout on #3696: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-3696/release/4253557268/index.html#suites/b97efae3a617afb71cb8142f5afa5224/d1c6ca3a23b97680/

[koivunej]

Rebased...

[koivunej]

Also, wondering if the compaction upload could go over timeout with real_s3, it will be racing against 10s termination timeout -- not the 20s wait_for_upload because lsn doesn't change and wait_for_upload cannot see if compacted layers or image layers are still being uploaded.

I should probably try the wait_for_upload_queue_empty here added in #3741 also to the timetravel test case which got the sleep added in earlier... Though, it's not enough that we await for queued to complete, we want to first await for new work to appear and then to await those to complete. If that makes sense.

[koivunej]

This probably needs some feedback and ideas per my above comments.

[koivunej]

Let's downscope the general failure to await for uploads in case we still have pending gc/compact out of this PR, rebased, should be good now.

[github-actions]

992 tests run: 946 passed, 0 failed, 46 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_ondemand_download_timetravel[real_s3]: ✅ debug

_{The comment gets automatically updated with the latest test results
c7c5f81 at 2023-05-17T13:20:22.957Z :recycle:}