Add compression and hash algorithm support

[StrongWind1]

Adds compress functions to lzxpress, lzxpress_huffman, lz4, and lznt1 (which previously only had decompress). Adds three new codecs: lzxpress9, deflate, and zlibstream. Adds a crc64 hash module for CRC-64/NVME. Fixes two bugs in lzxpress_huffman decompress.

@Schamper asked in fox-it/dissect.database#60 (comment) for the compression algorithms to go into dissect.util rather than pulling in an external dependency.

The code is ported from ntcompress, which has 982 tests and 107 gold vectors checked against Windows APIs across 16 builds (XP SP3 through Server 2025).

What changed

New files:

File	Description
compression/lzxpress9.py	XPRESS9 compress, decompress, and decompressed_size. This is the LZ77+Huffman codec ESE databases use for record compression (scheme 0x05). There is no public spec; the port follows the MIT-licensed ESE C source.
compression/deflate.py	Raw DEFLATE (RFC 1951) compress and decompress. Wraps stdlib zlib with wbits=-15.
compression/zlibstream.py	ZLIB (RFC 1950) compress and decompress. Wraps stdlib zlib.
hash/crc64.py	CRC-64/NVME checksum with reflected polynomial 0x9A6C9329AC4BC9B5. Needed for ESE XPRESS10 payload verification.

Updated files:

File	What changed
compression/lzxpress.py	Added compress(). Added optional max_size parameter to decompress().
compression/lzxpress_huffman.py	Added compress(). Added optional max_size parameter to decompress(). Fixed EOF symbol (256) handling and the v10.0 uint32 match-length escape. See bugfixes section below.
compression/lz4.py	Added compress() for headerless LZ4 blocks.
compression/lznt1.py	Added compress() with 4096-byte chunk processing per [MS-XCA] S2.5.3.
compression/__init__.py	Added deflate, lzxpress9, zlibstream to __all__.
hash/__init__.py	Added crc64 export.

Algorithm coverage

Module	Compress	Decompress	Reference
lzxpress	Added	Existed + max_size	[MS-XCA] S2.3-2.4
lzxpress_huffman	Added	Existed + max_size + bugfixes	[MS-XCA] S2.1-2.2
lznt1	Added	Existed	[MS-XCA] S2.5
lz4	Added	Existed	lz4 block format
lzxpress9	New	New	ESE C codec (MIT)
deflate	New	New	RFC 1951
zlibstream	New	New	RFC 1950
sevenbit	Existed	Existed	(no change)

Bugfixes in lzxpress_huffman decompress

Two bugs in the existing decompress(), found while porting the encoder and cross-testing:

1. EOF symbol (256) was not handled. The decoder treated symbol 256 as a match (symbol -= 256, then decoded offset/length), which produced 3 garbage bytes at the end of any stream terminated by EOF. Fixed by returning early when symbol 256 is decoded.

2. The v10.0 uint32 match-length escape was missing. When the uint16 match-length continuation is 0, [MS-XCA] S2.2 says to read a uint32 instead. The existing code fell through without reading the extra bytes. Fixed by adding the if length == 0 branch.

Closes #44

How to test

# Run existing tests
tox

# Round-trip each codec
python -c "
from dissect.util.compression import lzxpress, lzxpress_huffman, lz4, lznt1, lzxpress9, deflate, zlibstream
data = b'the quick brown fox jumps over the lazy dog! ' * 20
for name, mod in [('lzxpress',lzxpress),('lzxpress_huffman',lzxpress_huffman),('lz4',lz4),('lznt1',lznt1),('lzxpress9',lzxpress9),('deflate',deflate),('zlibstream',zlibstream)]:
    c = mod.compress(data)
    d = mod.decompress(c)
    if isinstance(d, tuple): d = d[0]
    assert d == data, f'{name} failed'
    print(f'{name}: {len(data)} -> {len(c)} -> {len(d)} ok')
"

# Verify max_size on lzxpress and lzxpress_huffman
python -c "
from dissect.util.compression import lzxpress, lzxpress_huffman
data = b'test ' * 200
for name, mod in [('lzxpress', lzxpress), ('lzxpress_huffman', lzxpress_huffman)]:
    c = mod.compress(data)
    d = mod.decompress(c, max_size=len(data))
    assert d == data
    try:
        mod.decompress(c, max_size=10)
        assert False
    except ValueError:
        print(f'{name} max_size: ok')
"

# Verify CRC-64/NVME
python -c "
from dissect.util.hash.crc64 import crc64
assert crc64(b'123456789') == 0xAE8B14860A799888
print('crc64: ok')
"

Testing done

• All 150 existing dissect.util tests pass (55 skipped, 0 failures)
• Every compress function round-trips with its own decompress across 13 different inputs, 5 rounds each, plus nested 5-deep compress chains with full unwind
• Cross-compatibility with ntcompress verified in both directions: 7 algorithms, 5 inputs, 2 directions, 70 tests total, all byte-identical
• CRC-64/NVME check value 0xAE8B14860A799888 for b"123456789" matches spec
• CRC-32C and CRC-64 output is identical to ntcompress on all test inputs
• All files pass ruff format and ruff check at 120 characters
• All files are ASCII-only

Note

Claude was used to help research and write this code. I have read all source, all documentation, and verified the output.

[Schamper]

Can you add tests in a similar style as the existing ones?

[DissectBot]

@StrongWind1 thank you for your contribution! As this is your first code contribution, please read the following Contributor License Agreement (CLA). If you agree with the CLA, please reply with the following information:

@DissectBot agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement ("Agreement") governs your Contribution(s) (as defined below) and conveys certain license rights to Fox-IT B.V. ("Fox-IT") for your Contribution(s) to Fox-IT"s open source Dissect project. This Agreement covers any and all Contributions that you ("You" or "Your"), now or in the future, Submit (as defined below) to this project. This Agreement is between Fox-IT B.V. and You and takes effect when you click an “I Accept” button, check box presented with these terms, otherwise accept these terms or, if earlier, when You Submit a Contribution.

1. Definitions.
   "Contribution" means any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to Fox-IT for inclusion in, or documentation of, any of the software products owned or managed by, or on behalf of, Fox-IT as part of the Project (the "Work").
   "Project" means any of the projects owned or managed by Fox-IT and offered under a license approved by the Open Source Initiative (www.opensource.org).
   "Submit" means any form of electronic, verbal, or written communication sent to Fox-IT or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Fox-IT for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."

2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Fox-IT and to recipients of software distributed by Fox-IT a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.

3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Fox-IT and to recipients of software distributed by Fox-IT a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, maintain, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.

4. Representations. You represent that:

   • You are legally entitled to grant the above license.
   • each of Your Contributions is Your original creation (see section 8 for submissions on behalf of others).
   • Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.

5. Employer. If Your Contribution is made in the course of Your work for an employer or Your employer has intellectual property rights in Your Submission by contract or applicable law, You must secure permission from Your employer to make the Contribution before signing this Agreement. In that case, the term "You" in this Agreement will refer to You and the employer collectively. If You change employers in the future and desire to Submit additional Contribution for the new employer, then You agree to sign a new Agreement and secure permission from the new employer before Submitting those Contributions.

6. Support. You are not expected to provide support for Your Contribution, unless You choose to do so. Any such support provided to the Project is provided free of charge.

7. Warranty. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.

8. Third party material. Should You wish to submit work that is not Your original creation, You may only submit it to Fox-IT separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which You are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".

9. Notify. You agree to notify Fox-IT of any facts or circumstances of which You become aware that would make the above representations inaccurate in any respect.

10. Governing law / competent court. This Agreement is governed by the laws of the Netherlands. Any disputes that may arise are resolved by arbitration in accordance with the Arbitration Regulations of the Foundation for the Settlement of Automation Disputes (Stichting Geschillenoplossing Automatisering – SGOA – (www.sgoa.eu), this without prejudice to either party"s right to request preliminary relief in preliminary relief proceedings or arbitral preliminary relief proceedings. Arbitration proceedings take place in Amsterdam, or in any other place designated in the Arbitration Regulations. Arbitration shall take place in English.

[StrongWind1]

@DissectBot agree

[StrongWind1]

Can you add tests in a similar style as the existing ones?

Done. Let me know if you need any other changes.