GH-46994: [C++][Parquet] Reuse BinaryView headers for repeated values in dictionary and DELTA_BYTE_ARRAY decoding#49850
Open
abhishek593 wants to merge 1 commit intoapache:mainfrom
Open
GH-46994: [C++][Parquet] Reuse BinaryView headers for repeated values in dictionary and DELTA_BYTE_ARRAY decoding#49850abhishek593 wants to merge 1 commit intoapache:mainfrom
abhishek593 wants to merge 1 commit intoapache:mainfrom
Conversation
…values in dictionary and DELTA_BYTE_ARRAY decoding When decoding Parquet ByteArray columns into Arrow BinaryView arrays, repeated dictionary values previously caused redundant data copies. This change pre-builds a cache of BinaryView headers per dictionary entry and registers the dictionary data as a single shared heap buffer, so all occurrences of the same dictionary value reuse the same view without copying.
|
|
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
When decoding Parquet ByteArray columns into Arrow BinaryView arrays, repeated dictionary values previously caused redundant data copies.
What changes are included in this PR?
Dictionary decoding: Pre-build a cache of
BinaryViewType::c_typeheaders from the dictionary entries. Out-of-line dictionary data is registered once as a shared heap buffer via a newBinaryViewBuilder::AppendBufferAPI. Decoding then emits the cached header directly.DELTA_BYTE_ARRAY decoding: When a decoded value has the same pointer and length as the previous value (the delta encoder's representation of an exact repeat), reuse the last BinaryView header instead of appending a duplicate.
Are these changes tested?
Yes. Added new tests.
Are there any user-facing changes?
No