Add XPRESS9, XPRESS10, LZ4 and SCRUB support

[StrongWind1]

Summary

This swaps the compression dispatch in compression.py for ntcompress, which handles every ESE record compression format. XPRESS9 and XPRESS10 no longer raise NotImplementedError (the state since #4). LZ4 (scheme 0x07) and SCRUB (scheme 0x04) are also handled now instead of silently passing through as raw bytes.

This was reported in #10 and came up in fox-it/dissect.esedb#24 (comment) where XPRESS10-compressed tagged values in NTDS databases were causing decoding errors. @Schamper noted it was not actively being worked on.

What changed

Two files, net -43 lines.

dissect/database/ese/compression.py goes from 51 lines to 6:

from __future__ import annotations

from ntcompress.ese import decompress
from ntcompress.ese import decompressed_size as decompress_size

__all__ = ["decompress", "decompress_size"]

The decompress(buf) and decompress_size(buf) signatures match the existing call sites in record.py L295, record.py L338, and table.py L216. Nothing else needs to change.

pyproject.toml: added "ntcompress" to dependencies.

Format coverage

Scheme ID	Name	Before	After
0x01	7-bit ASCII	Supported	Supported
0x02	7-bit Unicode	Supported	Supported
0x03	XPRESS	Supported	Supported + CRC verify
0x04	SCRUB	Silent passthrough	Proper error raised
0x05	XPRESS9	NotImplementedError	Supported + CRC verify
0x06	XPRESS10	NotImplementedError	Supported + CRC verify
0x07	LZ4	Not recognized	Supported

ntcompress

ntcompress (docs, PyPI) is a pure-Python library with no runtime dependencies, licensed Apache-2.0. Implementing XPRESS9, XPRESS10, LZ4 and SCRUB inline would be substantial and hard to maintain. ntcompress already has the codecs, a large test suite, and no transitive dependencies, so adding it keeps the dependency footprint minimal.

• 107 gold vectors from esent.dll and ntdll.dll across 16 Windows builds (XP SP3 through Server 2025)
• Decompress output is byte-identical to the Windows APIs
• XPRESS10 decompression verified against RtlDecompressBufferEx(format=0x0006) on Server 2025 Build 26100
• XPRESS9 validated against Power BI DataModel files (Analysis Services) and the MIT ESE C reference encoder
• Python >= 3.10, Apache-2.0
• 982 tests passing on 3.10 through 3.14, Linux/Windows/macOS

Testing

• All 15 databases under tests/_data/ese/ parsed, every column value accessed -- 236,369 records, 3.7M+ values, zero errors
• External NTDS databases from skelsec/aesedb (Server 2012, 2016, 2019) and AndrewRathbun/DFIRArtifactMuseum (Server 2012 R2, Win11)
• Win11 Build 26100 Windows Search and SRUM databases

Windows.edb.gz is the one that matters most here. It has records using all 7 ESE compression schemes, including the XPRESS9, XPRESS10, LZ4, and SCRUB records that previously failed or were returned as raw bytes.

Full fixture results (15 databases)

File	Records	Values	Compression
Windows.edb.gz	5,180	77,800	SEVEN_BIT_UNICODE=19759, XPRESS=300, SEVEN_BIT_ASCII=70, XPRESS9=2, XPRESS10=1, LZ4=1, SCRUB=1
ntds/large/ntds.dit.gz	78,784	2,157,420	(none)
ntds/goad/ntds.dit.gz	8,061	155,610	(none)
ntds/fve/ntds.dit.gz	7,551	145,214	(none)
ntds/adam/adamntds.dit.gz	3,072	60,027	(none)
large.edb.gz	130,166	1,106,249	(none)
tools/CertLog.edb.gz	609	4,743	SEVEN_BIT_UNICODE=69, XPRESS=9
text.edb.gz	220	1,816	XPRESS=3, SEVEN_BIT_ASCII=1
multi.edb.gz	211	1,736	XPRESS=9, SEVEN_BIT_ASCII=1
binary.edb.gz	190	1,560	XPRESS=1, SEVEN_BIT_ASCII=1
default.edb.gz	201	1,719	(none)
basic.edb.gz	190	1,590	(none)
index.edb.gz	259	2,163	(none)
tools/Current.mdb.gz	992		(none)
tools/SRUDB.dat.gz	683		(none)

Note

Claude was used to help research and write the code for ntcompress. I have read all documentation, all markdown files, and verified the output.

[Schamper]

Thanks for the contribution. Though I'd rather we expand the compression algorithms we support in dissect.util, than switch to a third party library. Especially since we already support a good chunk of them in there.

Would you be open to contributing the additional algorithm implementations (and/or improvements) to dissect.util?

[StrongWind1]

@Schamper Done. I contributed the algorithms to dissect.util and rewired dissect.database to use them:

• dissect.util: Add compression and hash algorithm support dissect.util#129
• dissect.database: Add XPRESS9, XPRESS10, LZ4 and SCRUB support #61 (replaces this PR)

The dissect.util PR adds compress and decompress for XPRESS9, plus compress functions for lzxpress, lzxpress_huffman, lz4, and lznt1 (which previously only had decompress). Also adds CRC-64/NVME and fixes two bugs in lzxpress_huffman decompress (missing EOF symbol handling, missing v10.0 uint32 match-length escape).

The dissect.database PR is the same scope as #60 but uses dissect.util instead of ntcompress. All seven ESE compression schemes are handled, with CRC verification. Tested against all 15 databases in the test fixtures.

This PR can be closed!