From e105c26fcbd6f94fdd6137f26cf6cd122d144ee4 Mon Sep 17 00:00:00 2001 From: carlos-alm Date: Tue, 7 Jul 2026 15:13:52 -0600 Subject: [PATCH] docs: refresh CONTRIBUTING.md for TypeScript architecture and current tooling CONTRIBUTING.md described the pre-TypeScript codebase: a plain-JS src/ tree with files that no longer exist (cli.js, builder.js, parser.js, queries.js, embedder.js, db.js, mcp.js, cycles.js, export.js, watcher.js, config.js, constants.js), Node >= 20 (actual requirement is >= 22.12.0 per package.json), an experimental "feat/rust-core" branch for the native engine (long since merged and now the default), .test.js examples (suite is .test.ts), and a stale WASM-grammars-are-committed claim (they're built from devDependencies via the prepare script on every npm install, not committed). Rewrite Project Structure, Architecture Notes, Code Style, Commands, Testing, Regression Benchmarks, Parser Improvements, and WASM Grammars sections to match the current TypeScript layer-based src/ tree, dual-engine architecture, and tooling. Project Structure and Architecture Notes now point to CLAUDE.md for the module-by-module breakdown instead of duplicating it, so this doc can't drift out of sync with the codebase the same way again. --- CONTRIBUTING.md | 106 ++++++++++++++++++++++++++++++------------------ 1 file changed, 66 insertions(+), 40 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c51b8b47..4d527a42 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,7 +15,7 @@ npm install # also installs git hooks via husky npm test # run the full test suite ``` -**Requirements:** Node.js >= 20 +**Requirements:** Node.js >= 22.12.0 (see `engines.node` in `package.json`) **Working in multiple git worktrees?** Each worktree gets its own untracked `node_modules/` and `grammars/` — neither is shared via git — so every fresh @@ -58,28 +58,37 @@ installs two git hooks: ## Project Structure +Source is TypeScript under `src/`, compiled via `tsup`/`tsc`; the native engine +lives in `crates/codegraph-core/` (Rust, via napi-rs) and mirrors the `src/` +tree module-for-module. `src/` is organized by layer, not by language: + ``` src/ - cli.js # Commander CLI entry point - index.js # Programmatic API exports - builder.js # Graph building: file collection, parsing, import resolution - parser.js # tree-sitter WASM wrapper + symbol extractors per language - queries.js # Query functions: symbol search, file deps, impact analysis - embedder.js # Semantic search with @huggingface/transformers - db.js # SQLite schema and operations - mcp.js # MCP server for AI agent integration - cycles.js # Circular dependency detection - export.js # DOT / Mermaid / JSON graph export - watcher.js # Watch mode for incremental rebuilds - config.js # .codegraphrc.json loading - constants.js # EXTENSIONS and IGNORE_DIRS - -grammars/ # Pre-built .wasm grammar files (committed) -scripts/ # Build scripts (build-wasm.js) -tests/ # vitest test suite -docs/ # Extended documentation + cli.ts, cli/ # Commander CLI entry point + per-command modules + index.ts # Programmatic API exports + shared/ # Cross-cutting constants, error types, kind enums + infrastructure/ # Config loading, logging, native addon loader, doctor + db/ # SQLite schema and operations (better-sqlite3) + domain/ # Parsing, graph building, import resolution, queries + extractors/ # Per-language symbol extractors (WASM/TS side) + features/ # Composable feature modules (audit, complexity, dataflow, ...) + presentation/ # Output formatting + CLI command wrappers + graph/ # Unified CodeGraph model + algorithms + classifiers + mcp/ # MCP server for AI agent integration + ast-analysis/ # Shared AST walker + pluggable analysis visitors + +crates/codegraph-core/ # Native (Rust/napi-rs) engine — mirrors src/ layout +scripts/ # Build, benchmark, and release tooling (TypeScript) +tests/ # vitest test suite +docs/ # Extended documentation ``` +For the authoritative, actively-maintained module-by-module breakdown (what +each file does, key design decisions, the native↔TypeScript mirroring table), +see the **Architecture** section of [`CLAUDE.md`](CLAUDE.md) — it is kept in +sync with the codebase as part of every PR that changes structure, so it's a +better long-term reference than a second copy here. + ## Development Workflow 1. **Fork** the repository and clone your fork @@ -95,9 +104,11 @@ docs/ # Extended documentation npm test # Run all tests (vitest) npm run test:watch # Watch mode npm run test:coverage # Coverage report -npx vitest run tests/parsers/go.test.js # Single test file +npx vitest run tests/parsers/go.test.ts # Single test file npx vitest run -t "finds cycles" # Single test by name -npm run build:wasm # Rebuild WASM grammars +npm run build # Compile TypeScript (tsc) to dist/ +npm run typecheck # Type-check only, no emit +npm run build:wasm # Rebuild WASM grammars from devDependencies npm run doctor # Check for a stale native binary / missing WASM grammars ``` @@ -169,6 +180,7 @@ tests/ parsers/ # Language parser extraction (one file per language) search/ # Semantic search + embeddings benchmarks/resolution/ # Call resolution precision/recall (per-language fixtures) + # + tracer/ — dynamic call tracers used to validate fixtures fixtures/ # Sample projects used by tests ``` @@ -186,11 +198,11 @@ results in the PR description. | Benchmark | What it measures | When to run | |-----------|-----------------|-------------| -| `node scripts/benchmark.js` | Build speed (native vs WASM), query latency | Changes to `builder.js`, `parser.js`, `queries.js`, `resolve.js`, `db.js`, or the native engine | -| `node scripts/embedding-benchmark.js` | Search recall (Hit@1/3/5/10) across models | Changes to `embedder.js` or embedding strategies | -| `node scripts/query-benchmark.js` | Query depth scaling, diff-impact latency | Changes to `queries.js`, `resolve.js`, or `db.js` | -| `node scripts/incremental-benchmark.js` | Incremental build, import resolution throughput | Changes to `builder.js`, `resolve.js`, `parser.js`, or `journal.js` | -| `npx vitest run tests/benchmarks/resolution/` | Call resolution precision/recall per language | Changes to `build-edges.js`, `resolve.js`, `parser.js`, or any extractor | +| `npm run benchmark` | Build speed (native vs WASM), query latency | Changes to `domain/graph/builder/`, `domain/parser.ts`, `domain/queries.ts`, `domain/graph/resolve.ts`, `db/`, or the native engine | +| `node scripts/node-ts.js scripts/embedding-benchmark.ts` | Search recall (Hit@1/3/5/10) across models | Changes to `domain/search/` or embedding strategies | +| `node scripts/node-ts.js scripts/query-benchmark.ts` | Query depth scaling, diff-impact latency | Changes to `domain/queries.ts`, `domain/graph/resolve.ts`, or `db/` | +| `node scripts/node-ts.js scripts/incremental-benchmark.ts` | Incremental build, import resolution throughput | Changes to `domain/graph/builder/`, `domain/graph/resolve.ts`, `domain/parser.ts`, or `domain/graph/journal.ts` | +| `npx vitest run tests/benchmarks/resolution/` | Call resolution precision/recall per language | Changes to `domain/graph/builder/stages/build-edges.ts`, `domain/graph/resolve.ts`, `domain/parser.ts`, or any extractor | ### Resolution precision/recall benchmark @@ -215,7 +227,7 @@ drops below the configured thresholds for any language, the test fails. 1. Create `tests/benchmarks/resolution/fixtures//` with source files 2. Add an `expected-edges.json` manifest (see the JSON schema at `tests/benchmarks/resolution/expected-edges.schema.json`) -3. Add thresholds in `resolution-benchmark.test.js` → `THRESHOLDS` +3. Add thresholds in `resolution-benchmark.test.ts` → `THRESHOLDS` 4. The benchmark runner auto-discovers fixtures with an `expected-edges.json` ### How to report results @@ -226,10 +238,10 @@ your PR description: ```bash git stash && git checkout main -node scripts/benchmark.js > before.json +npm run benchmark > before.json git checkout - && git stash pop -node scripts/benchmark.js > after.json +npm run benchmark > after.json ``` In the PR, include a table like: @@ -271,8 +283,11 @@ generics, nested types): 1. Find the tree-sitter AST node type using the [tree-sitter playground](https://tree-sitter.github.io/tree-sitter/playground) -2. Add a `case` in the corresponding `extractSymbols()` function -3. Add a test case in `tests/parsers/.test.js` +2. Add a `case` in the corresponding `extractSymbols()` function in + `src/extractors/.ts` (WASM) — and the mirrored extractor in + `crates/codegraph-core/src/extractors/` (native) if the change affects both + engines, per the dual-engine parity requirement in `CLAUDE.md` +3. Add a test case in `tests/parsers/.test.ts` ### Harness Engineering (Claude Code Hooks) @@ -296,19 +311,30 @@ Documentation improvements are always welcome. The main docs live in: imports -> SQLite DB -> query/search **Key design decisions:** -- WASM grammars are pre-built and committed in `grammars/` — no native compilation needed at install time +- **Dual-engine:** native Rust parsing via napi-rs (`crates/codegraph-core/`), with automatic fallback to WASM (`--engine native|wasm|auto`, default `auto`). Both engines must produce identical results — see `CLAUDE.md` for the mirrored module layout - Optional dependencies (`@huggingface/transformers`, `@modelcontextprotocol/sdk`) are lazy-loaded -- Parsers that can't load fail gracefully — they log a warning and skip those files +- Non-required parsers (everything except JS/TS/TSX) fail gracefully if their WASM grammar is unavailable — they log a warning and skip those files - Import resolution uses a 6-level priority system with confidence scoring -- The `feat/rust-core` branch introduces an optional native Rust engine via napi-rs for 5-10x faster parsing, with automatic fallback to WASM +- Incremental builds track file hashes in the DB to skip unchanged files -**Database:** SQLite at `.codegraph/graph.db` with tables: `nodes`, `edges`, -`metadata`, `embeddings` +**Database:** SQLite at `.codegraph/graph.db`. See the **Database** entry in +`CLAUDE.md` for the current table list — it's kept up to date there rather +than duplicated here. + +This is a summary; for full detail (module-by-module responsibilities, +configuration system, credential resolution, MCP isolation model) see +`CLAUDE.md`, which is the actively-maintained reference. ## WASM Grammars -The `.wasm` files in `grammars/` are pre-built and committed. You only need to -rebuild them if you: +`.wasm` grammar files are **not** committed to git — they're built from +`devDependencies` into `grammars/` automatically via the `prepare` npm script +(`npm run build:wasm`), which runs on every `npm install`. Each git worktree +therefore needs its own `npm install` before its `grammars/` directory is +populated (see "Working in multiple git worktrees?" above; `npm run doctor` +detects a missing or incomplete `grammars/`). + +Rebuild manually if you: - Add a new language - Upgrade a `tree-sitter-*` devDependency version @@ -328,8 +354,8 @@ Use [GitHub Issues](https://github.com/optave/ops-codegraph-tool/issues) with: ## Code Style -- All source is plain JavaScript (ES modules) — no transpilation -- [Biome](https://biomejs.dev/) is used for linting and formatting (config in `biome.json`) +- Source is TypeScript (`src/`), compiled via `tsc`/`tsup`; run `npm run typecheck` to type-check without emitting +- [Biome](https://biomejs.dev/) is used for linting and formatting (config in `biome.json`, scoped to `src/` and `tests/`) - Run `npm run lint` to check and `npm run lint:fix` to auto-fix - The pre-commit hook runs the linter automatically - Use `const`/`let` (no `var`)