✨ Preprocessor-aware C/C++ extraction via libclang

[ubmarco]

Summary

Adds an optional libclang preprocessor engine for C/C++ marker extraction.
The default tree-sitter path sees every comment; the libclang engine evaluates
the preprocessor (#if / #ifdef / defines) and emits only the markers in
active branches — so a need behind an inactive #else is correctly dropped.

• New analyse/preproc/ — libclang loader, compile_commands.json walk-up
  discovery + per-file flag resolution, and an active-branch parser.
• Opt-in via the libclang extra (pip install sphinx-codelinks[libclang]);
  plain tree-sitter extraction is unaffected when it is absent.
• Extraction-config wiring (PreprocessorConfig) + analyse / configuration
  docs.

Testing — declarative fixtures

Tests use the declarative fixture + snapshot harness from #83. A new
tests/data/extraction/preproc_variants.yaml fixture runs the same source
through both engines with per-engine golden snapshots:

• …[preproc_variants-variants_branching_treesitter] keeps every branch;
• …[preproc_variants-variants_branching_libclang] drops the inactive #else
  and keeps only the active set under -DVARIANT_A -DPLATFORM_LINUX -DPROTOCOL_VERSION=3.

Plus focused unit tests for the loader, compile_commands handling, and the
parser. Full extraction-fixture + preproc suites pass locally.

Rebase note

Rebased onto current main (post-#83). The branch carried its own copy of the
declarative harness (developed before #83 merged); those commits were dropped
in the rebase, so this PR is purely the libclang additions layered on top of
the merged harness.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.50898% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.26%. Comparing base (43927d0) to head (586221d).

Files with missing lines	Patch %	Lines
src/sphinx_codelinks/analyse/preproc/compile_db.py	87.95%	5 Missing and 5 partials ⚠️
src/sphinx_codelinks/analyse/analyse.py	90.76%	3 Missing and 3 partials ⚠️
...phinx_codelinks/analyse/preproc/libclang_parser.py	89.74%	2 Missing and 2 partials ⚠️
...codelinks/sphinx_extension/directives/src_trace.py	57.14%	1 Missing and 2 partials ⚠️
src/sphinx_codelinks/analyse/preproc/loader.py	96.00%	1 Missing and 1 partial ⚠️
src/sphinx_codelinks/cmd.py	0.00%	1 Missing and 1 partial ⚠️
tests/test_preproc_compile_db.py	98.26%	2 Missing ⚠️
tests/test_preproc_libclang.py	98.24%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
+ Coverage   91.40%   92.26%   +0.86%     
==========================================
  Files          34       43       +9     
  Lines        3012     3670     +658     
  Branches      322      375      +53     
==========================================
+ Hits         2753     3386     +633     
- Misses        160      171      +11     
- Partials       99      113      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ubmarco]

Review — libclang preprocessor engine

Reviewed at 6de94440. The engine is cleanly scoped — opt-in via the libclang extra, the default tree-sitter path is untouched, and the libclang surface is narrow (get_skipped_ranges + token walk). Findings below, most-severe first; all sit in the opt-in [preprocessor] path and none affect default extraction.

Medium

1. A compiler-launcher prefix (ccache/sccache/distcc) breaks compile-DB extraction (correctness)

filter_args drops only argv[0] as the compiler:

sphinx-codelinks/src/sphinx_codelinks/analyse/preproc/compile_db.py

Lines 52 to 54 in 6de9444

	for i, arg in enumerate(argv):
	if i == 0:
	continue # argv[0] == compiler

When compile_commands.json is produced with CMAKE_<LANG>_COMPILER_LAUNCHER=ccache, entries are ["ccache", "/usr/bin/g++", "-c", …]. Dropping only ccache leaves the real compiler token in the args passed to index.parse, which libclang treats as a second input → NULL TU → TranslationUnitLoadError → the file is skipped. Every DB-covered file silently drops out for launcher-based builds. Skip leading non-flag tokens up to the first flag.

2. A non-UTF-8 source aborts the whole run (correctness)

extract_active_comments reads the file as strict UTF-8 just to count lines:

sphinx-codelinks/src/sphinx_codelinks/analyse/preproc/libclang_parser.py

Lines 47 to 50 in 6de9444

	tu = index.parse(str(file_path), args=args, options=loader.PARSE_OPTIONS)
	skipped = loader.get_all_skipped_ranges(tu)
	line_count = len(file_path.read_text(encoding="utf-8").splitlines())

is_text_file only validates the first ~2 KB, so a file with a non-UTF-8 byte later (e.g. a Latin-1 © in a comment) passes the gate, then read_text(encoding="utf-8") raises UnicodeDecodeError. create_src_objects_libclang catches only TranslationUnitLoadError (analyse.py:186), so it propagates and fails the entire build — not just that file. Read with errors="replace" (the line count doesn't need exact bytes), or widen the guard.

3. Relative compile_commands / includes resolve against the CWD, not the config dir (correctness / docs)

convert_analyse_config builds these as bare Paths:

sphinx-codelinks/src/sphinx_codelinks/config.py

Lines 884 to 893 in 6de9444

	analyse_config_dict["preprocessor"] = PreprocessorConfig(
	compile_commands=(
	Path(str(preprocessor_dict["compile_commands"]))
	if preprocessor_dict.get("compile_commands")
	else None
	),
	defines=list(preprocessor_dict.get("defines", [])), # type: ignore[call-overload]
	includes=[Path(str(p)) for p in preprocessor_dict.get("includes", [])], # type: ignore[attr-defined]
	variant_name=preprocessor_dict.get("variant_name"), # type: ignore[arg-type]
	std=str(preprocessor_dict.get("std", "c++17")),

Unlike src_dir / git_root (anchored to the config file's directory by the front ends), a relative compile_commands or includes entry is interpreted relative to the process CWD. configuration.rst documents relative paths as resolving against the TOML file, so a documented-correct config silently misses the DB (→ defines fallback) or points -I at the wrong directory when the build runs from another CWD. Anchor both to the config dir alongside the siblings.

4. [preprocessor] config is never validated → a mistyped scalar becomes garbage flags (correctness)

SourceAnalyseConfig.check_fields_configuration validates need_id_refs / oneline / marked_rst but not preprocessor (config.py:504-527; the flat schema was intentionally removed, config.py:440-449), and convert_analyse_config only coerces the values (same block as #3 above). So defines = "FEATURE_A" (a bare string instead of a list) passes as list("FEATURE_A") → ["-DF", "-DE", "-DA", …] — silent garbage flags. A defines/includes/std type check (or a JSON schema over the section) would catch it at config-inited.

5. [unconfirmed] libclang comments skip the CRLF→LF normalization the tree-sitter path applies (correctness)

get_src_strings normalizes \r\n/\r → \n for the tree-sitter path (analyse.py:92); the libclang path stores the raw token spelling:

sphinx-codelinks/src/sphinx_codelinks/analyse/preproc/libclang_parser.py

Lines 64 to 67 in 6de9444

	if _is_in_skipped(str(loc.file.name), loc.line, skipped):
	continue # inactive branch -> excluded
	spelling = tok.spelling or ""
	out.append(LibclangComment(spelling.encode("utf-8"), loc.line - 1))

So a multi-line comment (e.g. an @rst block) in a CRLF-saved file can carry an embedded \r into marked_content.json, diverging from the tree-sitter output for the same source. Unconfirmed: whether \r survives the downstream extract_* line handling — confirm with a CRLF-saved source + a multi-line @rst block run through the engine.

Low

6. gnu++NN standard → NULL TU (correctness) — defines_to_args infers the language from std.startswith("c++") (compile_db.py:104), so std="gnu++17" yields -x c -std=gnu++17, which clang rejects. (C dialects like gnu11 are fine.) Match gnu++/c++ → C++.

7. compile_commands.json re-read + re-parsed per source file (performance) — _resolve_preproc_args calls load_flags_map inside the per-file loop (analyse.py:124, invoked per file at analyse.py:178) with no memoization → O(files × entries); a large project with a big DB re-parses the whole database once per file. Hoist the parse to once per DB.

8. compile_commands.json isn't registered as a Sphinx build dependency (durability) — extraction output now depends on the DB contents, but the directive doesn't register it, so editing compile_commands.json alone doesn't trigger re-extraction on an incremental build.

9. variant_name is parsed but never read (dead config) — config.py:152 / config.py:892 set it and the docstring promises it is "echoed into run-level output," but nothing reads it. Surface it or drop the field + doc.

10. [latent] _is_in_skipped path comparison is normalization-inconsistent (correctness) — get_all_skipped_ranges stores Path(name) (normalized) while extract_active_comments compares the raw str(loc.file.name) (libclang_parser.py:33, :64). Equivalent-but-differently-spelled paths (e.g. src/./foo.cpp) wouldn't match, leaking inactive-branch comments. Latent with the current callers (they pass resolved absolute paths), but worth normalizing both sides.

11. Minor cleanup — dead if preproc is None: return [] branch (analyse.py:117; the sole caller runs only when preprocessor is not None); inline from …source_discover.config import CommentType (analyse.py:475) with no cycle/cost justification; the c++17 default is duplicated across compile_db.py:92 / config.py:145 / config.py:893; _is_in_skipped rescans every skipped range per comment (O(comments × ranges)); the file is re-read purely for the line count (libclang_parser.py:50) on top of libclang's own parse.

---

Net: a well-isolated feature; nothing here touches the default tree-sitter path. The Mediums — launcher stripping (#1), non-UTF-8 abort (#2), relative-path anchoring (#3), and config validation (#4) — are the ones worth addressing before wider rollout.

[ubmarco]

Fixes pushed (3e92373)

Worked through the review findings:

Fixed

• Finding 1 [Med] compiler-launcher prefix (ccache/sccache/distcc) — filter_args now drops every leading non-flag token up to the first flag, so the launcher and the real compiler are stripped and the compiler no longer leaks in as a second input (which NULL-TU'd every DB-covered file). New preproc_compile_db · launcher_prefix_uses_db_flags fixture + unit tests.
• Finding 3 [Med] relative compile_commands / includes — now anchored to the config file's directory (like src_dir / git_root) via a anchor_preproc_paths helper used by both cmd.py and the src_trace directive. Unit test added.
• Finding 5 [Med] CRLF — extract_active_comments normalizes CRLF/CR → LF on the comment spelling, so a block comment (e.g. an @rst span) from a CRLF-saved file no longer carries an embedded \r into the extracted text. Regression test added.
• Finding 2 [Med] non-UTF-8 — partial. A non-UTF-8 byte in code no longer aborts the run (read_text(errors="replace") for the line count). Still open: a non-UTF-8 byte inside a comment — clang.cindex itself raises UnicodeDecodeError while decoding that comment token (cindex.py:89), past the TranslationUnitLoadError guard; a clean fix needs transcoding the source (offset-drift risk) or a broader guard, left for a follow-up.
• Finding 6 [Low] gnu++NN standard — treated as C++ (-x c++), not -x c (which returned a NULL TU). New preproc_std · gnucpp17 fixture + unit test.
• Finding 7 [Low] per-file DB re-parse — compile_commands.json is now memoized per run (read + parsed once per database, not once per source file).
• Also hardened filter_args to keep separate-form flag values (-include / -isystem / -I …) and to strip the input by basename.

Still open

• Finding 2 non-UTF-8 inside a comment — deeper binding-level issue (above).
• Finding 4 [Med] [preprocessor] validation — a mistyped scalar (e.g. defines = "X" as a bare string) still coerces to garbage flags; adding validation to the deliberately schema-less section needs a little more care — follow-up.
• Finding 8 [Low] compile_commands.json isn't registered as a Sphinx build dependency.
• Finding 9 [Low] variant_name parsed but unused — surface-or-drop is a product call.
• Finding 10 [Low] _is_in_skipped path-normalization (latent), and Finding 11 the minor cleanups.

All preprocessor + extraction-fixture suites pass; mypy clean; ruff clean on the changed modules.