Migrate all DB tables away from MessagePack #308
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Describe the change. If there is an associated issue, please include the issue link (e.g. "Closes #xxx"). For UI changes, please also include screenshots.
Previously, MessagePack was used extensively in the internal Slumber DB to store complex values (requests, responses, UI state, etc.). This seemed like a good idea at the time because it allowed me to jam complex data into single DB columns without having to write migrations when adding fields. I chose MessagePack over JSON because it was more compact. Over time though I've realized storing the data as big binary blobs makes it hard to debug, and also adds an extra package (
rmp_serde
) to the dep tree.This MR makes 3 major changes:
requests_v2
, which stores all request and response fields as columns. This will make it easier to browse data now, and migrate data in the future. I'm sure there's also a performance benefit to skipping the extra serde layerui_state_v2
that stores data in JSON instead of MessagePack. JSON will be a bit less compact, but is can be easily read in the CLI for debugging, and the size difference will be minimal.collections.path
from MessagePack to UTF-8. This could potential cause errors, but in practice should be fine because who the hell is writing paths that aren't UTF-8?? Mitigated with tests that contain UTF-8 paths, which pass on all platforms.I wrote migrations for all 3 so there should be no discernible impact to the user. The plan is to remove these migrations at some point in the future so we can drop the
rmp_serde
dependency (#306). It'll also allow us to remove some serde derive impls, which should improve compile time marginally. Right now I'm thinking ~6 months from now. That will necessitate a major version update.Known Risks
What issues could potentially go wrong with this change? Is it a breaking change? What have you done to mitigate any potential risks?
There's potential for data loss here. Mitigated that by leaving the
requests
table around. This means we'll have duplicate data which wastes disk space. We can drop that table in the future once the migration has been tested more.QA
How did you test this?
Manually tested migrations locally, added unit tests for each one.
Checklist
CONTRIBUTING.md
already?CHANGELOG.md
?