fix: compaction: no index upload scheduled if no on-demand downloads #3741
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lubennikovaav
approved these changes
Mar 3, 2023
problame
changed the base branch from
main
to
problame/use-parse_metrics-everywhere
problame
force-pushed
the
problame/iss-3738-remote-physical-size-lags-behind
branch
from
054726a to
29b776f
Compare
problame
force-pushed
the
problame/iss-3738-remote-physical-size-lags-behind
branch
from
29b776f to
a97fd9c
Compare
vadim2404
reviewed
Mar 3, 2023
vadim2404
reviewed
Mar 3, 2023
vadim2404
reviewed
Mar 3, 2023
Commit
0cf7fd0
Compaction with on-demand download (#3598)
introduced a subtle bug: if we don't have to do on-demand downloads,
we only take one ROUND in fn compact() and exit early.
Thereby, we miss scheduling the index part upload for any layers
created by fn compact_inner().
Before that commit, we didn't have this problem.
So, this patch fixes it.
Since no regression test caught this, I went ahead and extended the
timeline size tests to assert that, if remote storage is configured,
1. pageserver_remote_physical_size matches the other physical sizes
2. file sizes reported by the layer map info endpoint match the other
physical size metrics
Without the pageserver code fix, the regression test would
fail at the physical size assertion, complaining that
any of the resident physical size != remote physical size metric
50790400.0 != 18399232.0
I figured out what the problem is by comparing the remote storage
and local directories like so, and noticed that the image layer
in the local directory wasn't present on the remote side.
It's size was exactly the difference
50790400.0 - 18399232.0 =32391168.0
fixes #3738
problame
force-pushed
the
problame/iss-3738-remote-physical-size-lags-behind
branch
from
a97fd9c to
1ec0fd7
Compare
|
Thanks for the Python lesson, @vadim2404 , learning something new each time :) |
vadim2404
approved these changes
Mar 3, 2023
Closed
LizardWizzard
approved these changes
Mar 3, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit
introduced a subtle bug: if we don't have to do on-demand downloads, we only take one ROUND in fn compact() and exit early. Thereby, we miss scheduling the index part upload for any layers created by fn compact_inner().
Before that commit, we didn't have this problem.
So, this patch fixes it.
Since no regression test caught this, I went ahead and extended the timeline size tests to assert that, if remote storage is configured,
Without the pageserver code fix, the regression test would fail at the physical size assertion, complaining that any of the resident physical size != remote physical size metric 50790400.0 != 18399232.0
I figured out what the problem is by comparing the remote storage and local directories like so, and noticed that the image layer in the local directory wasn't present on the remote side. It's size was exactly the difference
50790400.0 - 18399232.0 =32391168.0
fixes #3738