Add XPRESS9, XPRESS10, LZ4 and SCRUB support#61
Open
StrongWind1 wants to merge 3 commits into
Open
Conversation
Expand ESE record compression to handle all seven schemes using the new algorithms from dissect.util. XPRESS9 and XPRESS10 no longer raise NotImplementedError. LZ4 (scheme 0x07) and SCRUB (scheme 0x04) are now handled instead of silently passing through as raw bytes. Add CRC-32C and CRC-64/NVME integrity verification for XPRESS9 and XPRESS10 headers, with an optional verify flag to skip checks for speed or corrupt-data recovery. Add decoded-size verification for XPRESS. Closes fox-it#10
3 tasks
Add direct unit tests for all compression schemes using real esent.dll and RtlCompressBuffer gold vectors, covering the verify flag, SCRUB and CRC-mismatch error paths. Fix the 7-bit decompress_size formula, which overcounted by reading the final byte's valid bit count from the header instead of assuming a full byte.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expands ESE record compression to handle all seven schemes using the algorithms now available in dissect.util. XPRESS9 and XPRESS10 no longer raise
NotImplementedError(the state since #4). LZ4 (scheme 0x07) is now recognized instead of falling through as raw bytes. SCRUB (scheme 0x04) now raisesValueErrorinstead of silently passing through.This replaces the approach in #60, which used ntcompress as an external dependency. Per @Schamper's comment, the algorithms were contributed directly to dissect.util instead (see fox-it/dissect.util#129).
Closes #10
Depends on fox-it/dissect.util#129
What changed
Three files, net +128 lines.
dissect/database/ese/c_ese.py: addedCOMPRESS_LZ4 = 0x7to theCOMPRESSION_SCHEMEenum.dissect/database/ese/c_ese.pyi: addedCOMPRESS_LZ4to the type stub.dissect/database/ese/compression.py: rewritten to dispatch all seven schemes. Thedecompress(buf)anddecompress_size(buf)signatures are unchanged. A new optionalverifykeyword argument skips CRC checks when set toFalse.The call sites in
record.py(L295, L338) andtable.py(L216) are unchanged.How to test
Testing done
Windows.edbcontains records using all seven compression schemes including the XPRESS9, XPRESS10, LZ4, and SCRUB records that previously failed or returned raw bytesverify=True(default) catches corrupted CRCs,verify=Falseskips themruff formatandruff checkNote
Claude was used to help research and write this code. I have read all source, all documentation, and verified the output.