Skip to content

Add XPRESS9, XPRESS10, LZ4 and SCRUB support#61

Open
StrongWind1 wants to merge 3 commits into
fox-it:mainfrom
StrongWind1:feature/expand-ese-compression
Open

Add XPRESS9, XPRESS10, LZ4 and SCRUB support#61
StrongWind1 wants to merge 3 commits into
fox-it:mainfrom
StrongWind1:feature/expand-ese-compression

Conversation

@StrongWind1

Copy link
Copy Markdown

Expands ESE record compression to handle all seven schemes using the algorithms now available in dissect.util. XPRESS9 and XPRESS10 no longer raise NotImplementedError (the state since #4). LZ4 (scheme 0x07) is now recognized instead of falling through as raw bytes. SCRUB (scheme 0x04) now raises ValueError instead of silently passing through.

This replaces the approach in #60, which used ntcompress as an external dependency. Per @Schamper's comment, the algorithms were contributed directly to dissect.util instead (see fox-it/dissect.util#129).

Closes #10

Depends on fox-it/dissect.util#129

What changed

Three files, net +128 lines.

dissect/database/ese/c_ese.py: added COMPRESS_LZ4 = 0x7 to the COMPRESSION_SCHEME enum.

dissect/database/ese/c_ese.pyi: added COMPRESS_LZ4 to the type stub.

dissect/database/ese/compression.py: rewritten to dispatch all seven schemes. The decompress(buf) and decompress_size(buf) signatures are unchanged. A new optional verify keyword argument skips CRC checks when set to False.

Scheme ID Name Before After
0x01 7-bit ASCII Supported Supported
0x02 7-bit Unicode Supported Supported
0x03 XPRESS Supported Supported + decoded-size verify
0x04 SCRUB Silent passthrough ValueError raised
0x05 XPRESS9 NotImplementedError Supported + CRC-32C verify
0x06 XPRESS10 NotImplementedError Supported + CRC-32C + CRC-64 verify
0x07 LZ4 Not recognized Supported

The call sites in record.py (L295, L338) and table.py (L216) are unchanged.

How to test

# Run existing tests
tox

# Parse all ESE test databases
python -c "
import gzip
from pathlib import Path
from dissect.database.ese.ese import ESE

for p in sorted(Path('tests/_data/ese').rglob('*.gz')):
    try:
        with gzip.open(p, 'rb') as f:
            db = ESE(f)
            records = sum(1 for t in db.tables() for r in t.records())
            print(f'{p.name}: {records} records ok')
    except Exception as e:
        print(f'{p.name}: {e}')
"

Testing done

  • All 18 existing ESE tests pass
  • 14 databases (skipping the 297 MB ntds.dit) parsed with every column value accessed: 157,585 records, 1,575,630 values, zero regressions
  • All 8 compression schemes (0x0 through 0x7) cross-tested against ntcompress, byte-identical output
  • Windows.edb contains records using all seven compression schemes including the XPRESS9, XPRESS10, LZ4, and SCRUB records that previously failed or returned raw bytes
  • verify=True (default) catches corrupted CRCs, verify=False skips them
  • All files pass ruff format and ruff check

Note

Claude was used to help research and write this code. I have read all source, all documentation, and verified the output.

Expand ESE record compression to handle all seven schemes using
the new algorithms from dissect.util. XPRESS9 and XPRESS10 no
longer raise NotImplementedError. LZ4 (scheme 0x07) and SCRUB
(scheme 0x04) are now handled instead of silently passing through
as raw bytes.

Add CRC-32C and CRC-64/NVME integrity verification for XPRESS9
and XPRESS10 headers, with an optional verify flag to skip checks
for speed or corrupt-data recovery. Add decoded-size verification
for XPRESS.

Closes fox-it#10
Add direct unit tests for all compression schemes using real esent.dll
and RtlCompressBuffer gold vectors, covering the verify flag, SCRUB and
CRC-mismatch error paths. Fix the 7-bit decompress_size formula, which
overcounted by reading the final byte's valid bit count from the header
instead of assuming a full byte.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for XPRESS10 compression

1 participant