Rewrite lexer and parser

[Schamper]

Closes #85, partially #142, and will make #86 and #138 a lot easier to implement. Fixes #149.

This PR will (finally) replace the shoddy C syntax parser I originally wrote many moons ago, when I discovered the existence of re.Scanner and ran with it. This PR aims to add a somewhat decent lexer and separate parser. I'm still not a compsci 1337coder, so this is just what I came up with (with some help) and definitely not a textbook implementation. All feedback is welcome.

• New lexer
• New C syntax parser that utilizes the new lexer
• Expression parser re-uses the new lexer
• Reworked how sizeof works in the expression parser, and added offsetof

The new parser has made changing parsing behavior a lot easier. As such, this PR already makes the following changes:

• The new parser is slightly stricter, requiring proper semicolon endings for example. We'll need to fix this in any dissect code that has this.

• An important semantic change is how named nested structures are handled. In my infinite wisdom, I originally figured that named nested structures do not "exist" in the top level scope. That's not true, so now named nested structures get properly registered with the cstruct instance:

struct a {
    struct b {
        ...
    };
};

// Will register both `a` and `b`

• Another important change is how we deal with struct { ... } name;. We used to parse this first as an anonymous struct, then capture name as the structure type name. That's not strictly correct, name is a variable of an anonymous unnamed struct, so we now treat it as such. We don't error on this, but rather we silently ignore name and skip until we reach a ;
• typedef enum ... is now allowed
• Probably some other things I'm forgetting

This probably warrants a major version bump, so maybe good to pair this with #114, #144 and what we discussed in #142.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 761 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (91f2347) to head (c58b76f).

Files with missing lines	Patch %	Lines
dissect/cstruct/lexer.py	0.00%	341 Missing ⚠️
dissect/cstruct/parser.py	0.00%	293 Missing ⚠️
dissect/cstruct/expression.py	0.00%	111 Missing ⚠️
dissect/cstruct/utils.py	0.00%	12 Missing ⚠️
dissect/cstruct/cstruct.py	0.00%	2 Missing ⚠️
dissect/cstruct/exceptions.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main    #146    +/-   ##
======================================
  Coverage   0.00%   0.00%            
======================================
  Files         21      22     +1     
  Lines       2500    2611   +111     
======================================
- Misses      2500    2611   +111     

Flag	Coverage Δ
unittests	0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 8 untouched benchmarks
🆕 2 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_benchmark_expression_evaluate	78.6 µs	127.7 µs	-38.44%
⚡	test_benchmark_expression_parse	343.8 µs	264.2 µs	+30.13%
⚡	test_benchmark_lexer_and_parser	15.7 ms	13 ms	+21.29%
⚡	test_benchmark_expression_parse_and_evaluate	389.5 µs	353.2 µs	+10.26%
🆕	test_benchmark_parser	N/A	10.6 ms	N/A
🆕	test_benchmark_lexer	N/A	2.4 ms	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rewrite-parser (c58b76f) with main (91f2347)}

[Schamper]

@sMezaOrellana would be interested in your thoughts on these changes!

[twiggler]

Architecure looks ok, found 2 possible issues TBC

[twiggler]

This PR has been migrated to the dissect monorepo: twiggler/dissect-monorepo-test#5

The original diff and commit history have been preserved on the migrate/dissect.cstruct/pr-146 branch.

[Schamper]

@twiggler I was a bit sad that your message was just this test, as I thought maybe you left more comments 😉.

[twiggler]

LGTM

Since it is an impactful rewrite I asked @Miauwkeru to QA

[Miauwkeru]

I did some checking on our other projects to see what goes wrong, and found some regressions regarding to where it crashed unexpectedly.

1. it cannot evaluate a ternary operation, which occurred in dissect.vmfs and gives a LexerError when trying to evaluate it as it doesn't understand the ? token. While I am aware that those definitions don't function yet, it probably shouldn't crash when trying to evaluate it

#define func(x) ( x ? 1 : 0 )

Without a ternary it makes it into a string:

# define func(x)    ( x == 0)
>>> c_struct.func
' ( x ) ( x = = 0)`

2. Another issue I found was when it was trying to create define a flag in dissect.apfs. Here it couldn't evaluate the . after the INODE in the definition.

flag INODE {
  ...
};

#define APFS_INODE_PINNED_MASK            INODE.PINNED_TO_MAIN | INODE.PINNED_TO_TIER2

I'll look through the code now to see whether I can find some additional issues.

[Schamper]

@Miauwkeru fixed.

[Miauwkeru]

LGTM, want to release it as 5.0? Then we don't have to update every project immediately.
Otherwise we might need to already have the patch sets ready before it gets merged in.

[Schamper]

Yes, see the OP:

This probably warrants a major version bump, so maybe good to pair this with #114, #144 and what we discussed in #142.

How would we merge this though? Merging now would result in a 4.7.dev and all CI failing.

[Miauwkeru]

How would we merge this though? Merging now would result in a 4.7.dev and all CI failing.

We can yank/delete the 4.7.dev# it creates on pypi. Ofc not ideal, but it will avoid the CI from falling.
In some cases we might also need to invalidate some caches on projects.

There is also the option to update the publish step to only work when you tag it or disable publishing it in its entirety

[Schamper]

Temporarily disabling the publish step is probably the easiest.

[Schamper]

@Miauwkeru I had to fix the merge with #140 to work with the removal of lookups so I require a new ack 😄.