Skip to content

fix: scope incremental structure/roles neighbor expansion to the changed file#1950

Merged
carlos-alm merged 2 commits into
mainfrom
fix/incremental-rebuild-neighbor-expansion-regression
Jul 7, 2026
Merged

fix: scope incremental structure/roles neighbor expansion to the changed file#1950
carlos-alm merged 2 commits into
mainfrom
fix/incremental-rebuild-neighbor-expansion-regression

Conversation

@carlos-alm

Copy link
Copy Markdown
Contributor

Summary

Fixes the "1-file rebuild" benchmark regression (~85-155% vs the 3.15.0 baseline on both engines) that's currently blocking the pre-publish and perf-canary CI gates on every open PR, including #1885, #1886, and #1889.

Root cause: two independent, correct correctness fixes (#1840, #1849) each add cost to the incremental role/structure-metrics recompute that only shows up when the changed file is itself a "hub" (imported/re-exported by many other files) — exactly what the benchmark's fixed probe file, src/domain/queries.ts, is.

  • fix: recompute directory structure metrics for affected directories on incremental rebuild #1840 — directory-metrics neighbor discovery (refresh_affected_directory_metrics / refreshAffectedDirectoryMetrics) was scoped to the changed file's whole containing directory, not the file itself, pulling in unrelated sibling files' edges. For queries.ts (leaf dir src/domain, itself a hub), this found 251 neighbour files / 29 affected directories, including the repo-root src/ (a near-full-table scan). Scoping to the exact file cuts this to 46 / 13.
  • fix: scope reexportedSymbols to actually-named re-export specifiers #1849 — added symbol-level reexports edges so re-exported symbols get fan-in credit and aren't misclassified dead. That correctly widened do_classify_incremental's neighbour-file set for barrel files like queries.ts (0 → 15 files), which then hit two pre-existing inefficiencies never exercised at this scale on a 1-file change: an OR-based self-join SQLite can't index, and a repo-wide recursive CTE (the same one the full-classify path legitimately needs) answering "is this barrel production-reachable" for a question only 1-3 barrels actually need answered.

Fix (both native Rust and TypeScript/WASM)

  • Scope neighbor discovery to the exact touched file (indexed point lookup) instead of a directory-range scan.
  • Rewrite the OR self-join as a UNION of two directional equi-joins (identical result set, ~2.4x faster).
  • Replace the global reachability closure with a backward-scoped check: find the small set of barrels reexporting into the affected files, then answer reachability per-barrel by walking reexports edges backward (bounded by that barrel's own chain depth) instead of materializing the whole graph's forward closure.

Every correctness guarantee from #1840/#1849 is preserved — verified against their dedicated test suites (issue-1738-structure-metrics-incremental.test.ts, issue-1742-reexport-symbol-scope.test.ts) on both engines.

Before / after (local, src/domain/queries.ts probe, native engine)

Metric 3.15.0 baseline main (regressed) This PR
1-file rebuild (native) 117ms 216-258ms (+85% to +120%) 139-166ms (+19% to +42%)
1-file rebuild (wasm) 112-113ms 240-286ms (+114% to +155%) 181-221ms (+60% to +97%, see note)
structureMs (native) 4ms 51-58ms 27-37ms
rolesMs (native) 18ms 42-58ms 17-23ms

Note: WASM's structureMs/rolesMs are fixed to the same degree as native (e.g. structureMs 44.7ms → 24-32ms, rolesMs 46.9ms → 16-23ms). Its total still shows some residual elevation on my local (shared, noisy) dev machine, traced to detectMs (change detection, unrelated to this fix — filed separately as #1948) rather than anything this PR touches.

Test plan

Fixes #1855

…ged file

The "1-file rebuild" benchmark regressed ~85-155% vs the 3.15.0 baseline
on both engines, blocking the pre-publish and perf-canary CI gates on
every open PR. Two independent, correct correctness fixes each added
unbounded-ish cost that only shows up when the changed file is itself a
hub (imported/re-exported by many other files) — exactly what the
benchmark's fixed probe file, src/domain/queries.ts, is:

- #1840's directory-metrics neighbor discovery was scoped to the changed
  file's whole containing directory instead of the file itself, pulling
  in unrelated sibling files' edges. For queries.ts (leaf dir
  src/domain, itself a hub) this found 251 neighbour files / 29 affected
  directories, including the repo-root src/ (a near-full-table scan).
  Scoping to the exact file cuts this to 46 / 13.

- #1849 added symbol-level reexports edges so re-exported symbols get
  fan-in credit. That correctly widened do_classify_incremental's
  neighbour-file set for barrel files like queries.ts (0 -> 15 files),
  which then hit two pre-existing inefficiencies never exercised at this
  scale on a 1-file change: an OR-based self-join SQLite can't index,
  and a repo-wide recursive CTE (the same one do_classify_full legitimately
  needs) answering "is this barrel production-reachable" for a question
  only 1-3 barrels actually need answered.

Fixes, applied to both the native Rust classifier and its TypeScript/WASM
mirror:
- Scope neighbor discovery to the exact touched file (indexed point
  lookup) instead of a directory-range scan.
- Rewrite the OR self-join as a UNION of two directional equi-joins
  (identical result set, ~2.4x faster).
- Replace the global reachability closure with a backward-scoped check:
  find the small set of barrels reexporting into the affected files,
  then answer reachability per-barrel by walking reexports edges
  backward (bounded by that barrel's own chain depth) instead of
  materializing the whole graph's forward closure.

Verified against the dedicated correctness suites for both #1840
(issue-1738-structure-metrics-incremental.test.ts) and #1849
(issue-1742-reexport-symbol-scope.test.ts) on both engines, plus the
full test suite and engine-parity tests.

docs check acknowledged: internal performance bug fix to the incremental
build pipeline, no new feature/language/CLI surface/architecture change
-- README.md, CLAUDE.md, and ROADMAP.md are unaffected.

Fixes #1855

Impact: 4 functions changed, 5 affected
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Codegraph Impact Analysis

4 functions changed5 callers affected across 3 files

  • refreshAffectedDirectoryMetrics in src/domain/graph/builder/stages/build-structure.ts:297 (3 transitive callers)
  • findDirectReexportBarrels in src/features/structure.ts:931 (2 transitive callers)
  • isBarrelProdReachable in src/features/structure.ts:956 (2 transitive callers)
  • classifyNodeRolesIncremental in src/features/structure.ts:1014 (1 transitive callers)

@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a performance regression (~85–155% on the "1-file rebuild" benchmark) introduced by two prior correctness fixes (#1840, #1849) that each added cost to the incremental structure/role classification path when the changed file is a widely-imported hub like src/domain/queries.ts. Every correctness guarantee from those fixes is preserved.

  • Structure neighbor scoping (structure.rs, build-structure.ts): find_neighbor_files now executes a point lookup on the exact changed file path (file = ?) instead of a directory range scan, cutting neighbor discovery from 251 → 46 files for the benchmark probe.
  • Roles UNION rewrite (roles.rs, structure.ts): the find_neighbour_files OR-based self-join (un-indexable) is replaced by a UNION of two directional equi-joins (~2.4x faster); and the global prod_reachable recursive CTE is replaced by a per-barrel backward-walk bounded to the small set of barrels (typically 1–3) that actually reexport into the affected files.

Confidence Score: 5/5

Safe to merge — all three optimizations preserve the result sets of the queries they replace, correctness is backed by the dedicated regression suites for #1840 and #1849, and the Rust/TypeScript implementations mirror each other faithfully.

The neighbor-scoping change is a straightforward predicate swap (range to equality) that can only reduce the working set. The UNION rewrite is semantically equivalent to the original OR-join — both directions are covered, UNION deduplicates the same way DISTINCT did, and parameter binding is correctly repeated four times. The backward-scoped is_barrel_prod_reachable correctly answers the same yes/no question as the global CTE, and UNION (not UNION ALL) in the recursive CTE guarantees termination on cycles. The cachedStmt additions address the prior review comment cleanly.

No files require special attention.

Important Files Changed

Filename Overview
crates/codegraph-core/src/features/structure.rs Scopes find_neighbor_files to a single file path (indexed equi-join) instead of a directory range scan; updates refresh_affected_directory_metrics to loop over touched files directly. SQL and Rust binding are correct.
crates/codegraph-core/src/graph/classifiers/roles.rs Rewrites find_neighbour_files as a UNION of two equi-joins (4x parameter groups correctly bound), introduces find_direct_reexport_barrels and per-barrel is_barrel_prod_reachable with prepare_cached, and replaces the global prod_reachable CTE in do_classify_incremental. Logic and SQL parameter binding verified correct.
src/domain/graph/builder/stages/build-structure.ts TypeScript mirror of the structure.rs change: neighborFiles statement now uses named @file parameter instead of @lo/@hi range; loop iterates over individual files. Correct.
src/features/structure.ts TypeScript mirror of the roles.rs changes: adds findDirectReexportBarrels, isBarrelProdReachable with module-level WeakMap caches, and updates classifyNodeRolesIncremental with the UNION neighbour query and scoped barrel check.

Reviews (3): Last reviewed commit: "fix: cache the per-barrel reachability s..." | Re-trigger Greptile

Comment thread src/features/structure.ts Outdated
…hable

Addresses Greptile's review note on #1950: both queries in
is_barrel_prod_reachable / isBarrelProdReachable are re-prepared on
every call even though their SQL text never changes (the interpolated
test-file filter is a fixed set of LIKE patterns keyed only by column
name). Harmless at the expected 1-3 barrels per incremental build, but
free to fix: Rust now uses prepare_cached (already the established
idiom elsewhere in this file and connection.rs) and TypeScript mirrors
it with the existing cachedStmt/StmtCache WeakMap pattern used in
graph-read.ts and exports.ts.

No behavior change — verified against the same test suites as #1950
(issue-1738, issue-1742, full suite, lint).

docs check acknowledged: internal micro-optimization, no new feature/
language/CLI surface/architecture change -- README.md, CLAUDE.md, and
ROADMAP.md are unaffected.

Impact: 1 functions changed, 2 affected
@carlos-alm

Copy link
Copy Markdown
Contributor Author

@greptileai

@carlos-alm carlos-alm merged commit a7ca6e9 into main Jul 7, 2026
48 of 50 checks passed
@carlos-alm carlos-alm deleted the fix/incremental-rebuild-neighbor-expansion-regression branch July 7, 2026 05:40
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 7, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: 1-file incremental rebuild regresses +79-81% (117ms -> ~209ms), likely from PR #1840 structure-metrics recompute

1 participant