Fix bug where purging history could lead to increase in disk space usage #18131
+116
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently we don't really have anything that stops us from deleting state groups when an in-flight event references it. This is a fairly rare race currently, but we want to be able to more aggresively delete state groups so it is important to address this to ensure that the database remains valid. See the class docstring of the new data store for an explanation for how this works.
Co-authored-by: Devon Hudson <[email protected]>
Co-authored-by: Devon Hudson <[email protected]>
Co-authored-by: Devon Hudson <[email protected]>
….sql Co-authored-by: Devon Hudson <[email protected]>
erikjohnston
force-pushed
the
erikj/state_delete
branch
from
5c722d6 to
9e51e4e
Compare
erikjohnston
force-pushed
the
erikj/purge_unreferenced_state
branch
from
9fbe823 to
68c8dcd
Compare
erikjohnston
force-pushed
the
erikj/state_delete
branch
from
9e51e4e to
cc67b6f
Compare
erikjohnston
force-pushed
the
erikj/purge_unreferenced_state
branch
from
68c8dcd to
e67b3bc
Compare
This actually makes it so that deleting state groups goes via the new mechanism.
erikjohnston
force-pushed
the
erikj/state_delete
branch
from
a32805a to
56aa695
Compare
erikjohnston
force-pushed
the
erikj/purge_unreferenced_state
branch
from
e67b3bc to
72d1d97
Compare
erikjohnston
force-pushed
the
erikj/purge_unreferenced_state
branch
from
72d1d97 to
2b5cbc1
Compare
devonh
approved these changes
Feb 3, 2025
tests/storage/test_purge.py
Outdated
| ) | ||
|
|
||
| # Advance so that the background jobs to delete the state groups runs | ||
| self.reactor.advance(10000) |
There was a problem hiding this comment.
Should reference the deletion timeout constant
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When purging history, we try and delete any state groups that become unreferenced (i.e. there are no longer any events that directly reference them). When we delete a state group that is referenced by another state group, we "de-delta" that state group so that it no longer refers to the state group that is deleted.
There are two bugs with this approach that we fix here:
The effect of the above bugs is that when purging history we'd end up with lots of unreferenced state groups that had been de-deltaed (i.e. stored as the full state). This can lead to dramatic increases in storage space used.
Based on #18107 and #18130