Change Block getSizeInBytes to estimate fully expanded data size #25256
+557
−2,424
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The Block
getSizeinBytes
method has been defined to return the "compacted" size of a block. This means for an RLE only the size of the single value is returned, and for a dictionary block the size of each position is computed from the dictionary. This computation is quite expensive and requires Blocks to havegetPositionsSizeInBytes
andfixedSizeInBytesPerPosition
methods.This PR changes the implementation to be defined as returning an estimate of the full data size of a block. This means for an RLE block the size is
value.getSizeInBytes() * positionCount
and dictionary is(dictionary.getSizeInBytes() / dictionary.getPositionCount()) * positionCount
, which is simpler and much faster to compute.Current Usage
The
getSizeinBytes
method is typically used to estimate the size needed for output buffers when copying data in operators and generally these usages assume that the value returned is a fully expanded size. The other main usage is in stats, and here the change will have visible effects. This method is often used to calculate the input/output size of operators, sources, and sinks. It is not clear if the these uses intended to have compacted or fully expanded size.Release notes
(X) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: