-
Notifications
You must be signed in to change notification settings - Fork 15
docs: refresh CONTRIBUTING.md for TypeScript architecture and current tooling #1977
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
carlos-alm
wants to merge
1
commit into
fix/issue-1830-follow-up-document-or-discriminate-file-level
Choose a base branch
from
fix/issue-1831-contributing-md-is-stale-references-removed
base: fix/issue-1830-follow-up-document-or-discriminate-file-level
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -15,7 +15,7 @@ npm install # also installs git hooks via husky | |||||||||||||||||||
| npm test # run the full test suite | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
|
|
||||||||||||||||||||
| **Requirements:** Node.js >= 20 | ||||||||||||||||||||
| **Requirements:** Node.js >= 22.12.0 (see `engines.node` in `package.json`) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| **Working in multiple git worktrees?** Each worktree gets its own untracked | ||||||||||||||||||||
| `node_modules/` and `grammars/` — neither is shared via git — so every fresh | ||||||||||||||||||||
|
|
@@ -58,28 +58,37 @@ installs two git hooks: | |||||||||||||||||||
|
|
||||||||||||||||||||
| ## Project Structure | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Source is TypeScript under `src/`, compiled via `tsup`/`tsc`; the native engine | ||||||||||||||||||||
| lives in `crates/codegraph-core/` (Rust, via napi-rs) and mirrors the `src/` | ||||||||||||||||||||
| tree module-for-module. `src/` is organized by layer, not by language: | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ``` | ||||||||||||||||||||
| src/ | ||||||||||||||||||||
| cli.js # Commander CLI entry point | ||||||||||||||||||||
| index.js # Programmatic API exports | ||||||||||||||||||||
| builder.js # Graph building: file collection, parsing, import resolution | ||||||||||||||||||||
| parser.js # tree-sitter WASM wrapper + symbol extractors per language | ||||||||||||||||||||
| queries.js # Query functions: symbol search, file deps, impact analysis | ||||||||||||||||||||
| embedder.js # Semantic search with @huggingface/transformers | ||||||||||||||||||||
| db.js # SQLite schema and operations | ||||||||||||||||||||
| mcp.js # MCP server for AI agent integration | ||||||||||||||||||||
| cycles.js # Circular dependency detection | ||||||||||||||||||||
| export.js # DOT / Mermaid / JSON graph export | ||||||||||||||||||||
| watcher.js # Watch mode for incremental rebuilds | ||||||||||||||||||||
| config.js # .codegraphrc.json loading | ||||||||||||||||||||
| constants.js # EXTENSIONS and IGNORE_DIRS | ||||||||||||||||||||
|
|
||||||||||||||||||||
| grammars/ # Pre-built .wasm grammar files (committed) | ||||||||||||||||||||
| scripts/ # Build scripts (build-wasm.js) | ||||||||||||||||||||
| tests/ # vitest test suite | ||||||||||||||||||||
| docs/ # Extended documentation | ||||||||||||||||||||
| cli.ts, cli/ # Commander CLI entry point + per-command modules | ||||||||||||||||||||
| index.ts # Programmatic API exports | ||||||||||||||||||||
| shared/ # Cross-cutting constants, error types, kind enums | ||||||||||||||||||||
| infrastructure/ # Config loading, logging, native addon loader, doctor | ||||||||||||||||||||
| db/ # SQLite schema and operations (better-sqlite3) | ||||||||||||||||||||
| domain/ # Parsing, graph building, import resolution, queries | ||||||||||||||||||||
| extractors/ # Per-language symbol extractors (WASM/TS side) | ||||||||||||||||||||
| features/ # Composable feature modules (audit, complexity, dataflow, ...) | ||||||||||||||||||||
| presentation/ # Output formatting + CLI command wrappers | ||||||||||||||||||||
| graph/ # Unified CodeGraph model + algorithms + classifiers | ||||||||||||||||||||
| mcp/ # MCP server for AI agent integration | ||||||||||||||||||||
| ast-analysis/ # Shared AST walker + pluggable analysis visitors | ||||||||||||||||||||
|
|
||||||||||||||||||||
| crates/codegraph-core/ # Native (Rust/napi-rs) engine — mirrors src/ layout | ||||||||||||||||||||
| scripts/ # Build, benchmark, and release tooling (TypeScript) | ||||||||||||||||||||
| tests/ # vitest test suite | ||||||||||||||||||||
| docs/ # Extended documentation | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
|
|
||||||||||||||||||||
| For the authoritative, actively-maintained module-by-module breakdown (what | ||||||||||||||||||||
| each file does, key design decisions, the native↔TypeScript mirroring table), | ||||||||||||||||||||
| see the **Architecture** section of [`CLAUDE.md`](CLAUDE.md) — it is kept in | ||||||||||||||||||||
| sync with the codebase as part of every PR that changes structure, so it's a | ||||||||||||||||||||
| better long-term reference than a second copy here. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ## Development Workflow | ||||||||||||||||||||
|
|
||||||||||||||||||||
| 1. **Fork** the repository and clone your fork | ||||||||||||||||||||
|
|
@@ -95,9 +104,11 @@ docs/ # Extended documentation | |||||||||||||||||||
| npm test # Run all tests (vitest) | ||||||||||||||||||||
| npm run test:watch # Watch mode | ||||||||||||||||||||
| npm run test:coverage # Coverage report | ||||||||||||||||||||
| npx vitest run tests/parsers/go.test.js # Single test file | ||||||||||||||||||||
| npx vitest run tests/parsers/go.test.ts # Single test file | ||||||||||||||||||||
| npx vitest run -t "finds cycles" # Single test by name | ||||||||||||||||||||
| npm run build:wasm # Rebuild WASM grammars | ||||||||||||||||||||
| npm run build # Compile TypeScript (tsc) to dist/ | ||||||||||||||||||||
| npm run typecheck # Type-check only, no emit | ||||||||||||||||||||
| npm run build:wasm # Rebuild WASM grammars from devDependencies | ||||||||||||||||||||
| npm run doctor # Check for a stale native binary / missing WASM grammars | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
|
@@ -169,6 +180,7 @@ tests/ | |||||||||||||||||||
| parsers/ # Language parser extraction (one file per language) | ||||||||||||||||||||
| search/ # Semantic search + embeddings | ||||||||||||||||||||
| benchmarks/resolution/ # Call resolution precision/recall (per-language fixtures) | ||||||||||||||||||||
| # + tracer/ — dynamic call tracers used to validate fixtures | ||||||||||||||||||||
| fixtures/ # Sample projects used by tests | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
|
@@ -186,11 +198,11 @@ results in the PR description. | |||||||||||||||||||
|
|
||||||||||||||||||||
| | Benchmark | What it measures | When to run | | ||||||||||||||||||||
| |-----------|-----------------|-------------| | ||||||||||||||||||||
| | `node scripts/benchmark.js` | Build speed (native vs WASM), query latency | Changes to `builder.js`, `parser.js`, `queries.js`, `resolve.js`, `db.js`, or the native engine | | ||||||||||||||||||||
| | `node scripts/embedding-benchmark.js` | Search recall (Hit@1/3/5/10) across models | Changes to `embedder.js` or embedding strategies | | ||||||||||||||||||||
| | `node scripts/query-benchmark.js` | Query depth scaling, diff-impact latency | Changes to `queries.js`, `resolve.js`, or `db.js` | | ||||||||||||||||||||
| | `node scripts/incremental-benchmark.js` | Incremental build, import resolution throughput | Changes to `builder.js`, `resolve.js`, `parser.js`, or `journal.js` | | ||||||||||||||||||||
| | `npx vitest run tests/benchmarks/resolution/` | Call resolution precision/recall per language | Changes to `build-edges.js`, `resolve.js`, `parser.js`, or any extractor | | ||||||||||||||||||||
| | `npm run benchmark` | Build speed (native vs WASM), query latency | Changes to `domain/graph/builder/`, `domain/parser.ts`, `domain/queries.ts`, `domain/graph/resolve.ts`, `db/`, or the native engine | | ||||||||||||||||||||
| | `node scripts/node-ts.js scripts/embedding-benchmark.ts` | Search recall (Hit@1/3/5/10) across models | Changes to `domain/search/` or embedding strategies | | ||||||||||||||||||||
| | `node scripts/node-ts.js scripts/query-benchmark.ts` | Query depth scaling, diff-impact latency | Changes to `domain/queries.ts`, `domain/graph/resolve.ts`, or `db/` | | ||||||||||||||||||||
| | `node scripts/node-ts.js scripts/incremental-benchmark.ts` | Incremental build, import resolution throughput | Changes to `domain/graph/builder/`, `domain/graph/resolve.ts`, `domain/parser.ts`, or `domain/graph/journal.ts` | | ||||||||||||||||||||
| | `npx vitest run tests/benchmarks/resolution/` | Call resolution precision/recall per language | Changes to `domain/graph/builder/stages/build-edges.ts`, `domain/graph/resolve.ts`, `domain/parser.ts`, or any extractor | | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ### Resolution precision/recall benchmark | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
|
@@ -215,7 +227,7 @@ drops below the configured thresholds for any language, the test fails. | |||||||||||||||||||
| 1. Create `tests/benchmarks/resolution/fixtures/<language>/` with source files | ||||||||||||||||||||
| 2. Add an `expected-edges.json` manifest (see the JSON schema at | ||||||||||||||||||||
| `tests/benchmarks/resolution/expected-edges.schema.json`) | ||||||||||||||||||||
| 3. Add thresholds in `resolution-benchmark.test.js` → `THRESHOLDS` | ||||||||||||||||||||
| 3. Add thresholds in `resolution-benchmark.test.ts` → `THRESHOLDS` | ||||||||||||||||||||
| 4. The benchmark runner auto-discovers fixtures with an `expected-edges.json` | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ### How to report results | ||||||||||||||||||||
|
|
@@ -226,10 +238,10 @@ your PR description: | |||||||||||||||||||
|
|
||||||||||||||||||||
| ```bash | ||||||||||||||||||||
| git stash && git checkout main | ||||||||||||||||||||
| node scripts/benchmark.js > before.json | ||||||||||||||||||||
| npm run benchmark > before.json | ||||||||||||||||||||
|
|
||||||||||||||||||||
| git checkout - && git stash pop | ||||||||||||||||||||
| node scripts/benchmark.js > after.json | ||||||||||||||||||||
| npm run benchmark > after.json | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
|
|
||||||||||||||||||||
| In the PR, include a table like: | ||||||||||||||||||||
|
|
@@ -271,8 +283,11 @@ generics, nested types): | |||||||||||||||||||
|
|
||||||||||||||||||||
| 1. Find the tree-sitter AST node type using the | ||||||||||||||||||||
| [tree-sitter playground](https://tree-sitter.github.io/tree-sitter/playground) | ||||||||||||||||||||
| 2. Add a `case` in the corresponding `extract<Lang>Symbols()` function | ||||||||||||||||||||
| 3. Add a test case in `tests/parsers/<lang>.test.js` | ||||||||||||||||||||
| 2. Add a `case` in the corresponding `extract<Lang>Symbols()` function in | ||||||||||||||||||||
| `src/extractors/<lang>.ts` (WASM) — and the mirrored extractor in | ||||||||||||||||||||
| `crates/codegraph-core/src/extractors/` (native) if the change affects both | ||||||||||||||||||||
| engines, per the dual-engine parity requirement in `CLAUDE.md` | ||||||||||||||||||||
| 3. Add a test case in `tests/parsers/<lang>.test.ts` | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ### Harness Engineering (Claude Code Hooks) | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
|
@@ -296,19 +311,30 @@ Documentation improvements are always welcome. The main docs live in: | |||||||||||||||||||
| imports -> SQLite DB -> query/search | ||||||||||||||||||||
|
|
||||||||||||||||||||
| **Key design decisions:** | ||||||||||||||||||||
| - WASM grammars are pre-built and committed in `grammars/` — no native compilation needed at install time | ||||||||||||||||||||
| - **Dual-engine:** native Rust parsing via napi-rs (`crates/codegraph-core/`), with automatic fallback to WASM (`--engine native|wasm|auto`, default `auto`). Both engines must produce identical results — see `CLAUDE.md` for the mirrored module layout | ||||||||||||||||||||
| - Optional dependencies (`@huggingface/transformers`, `@modelcontextprotocol/sdk`) are lazy-loaded | ||||||||||||||||||||
| - Parsers that can't load fail gracefully — they log a warning and skip those files | ||||||||||||||||||||
| - Non-required parsers (everything except JS/TS/TSX) fail gracefully if their WASM grammar is unavailable — they log a warning and skip those files | ||||||||||||||||||||
| - Import resolution uses a 6-level priority system with confidence scoring | ||||||||||||||||||||
| - The `feat/rust-core` branch introduces an optional native Rust engine via napi-rs for 5-10x faster parsing, with automatic fallback to WASM | ||||||||||||||||||||
| - Incremental builds track file hashes in the DB to skip unchanged files | ||||||||||||||||||||
|
|
||||||||||||||||||||
| **Database:** SQLite at `.codegraph/graph.db` with tables: `nodes`, `edges`, | ||||||||||||||||||||
| `metadata`, `embeddings` | ||||||||||||||||||||
| **Database:** SQLite at `.codegraph/graph.db`. See the **Database** entry in | ||||||||||||||||||||
| `CLAUDE.md` for the current table list — it's kept up to date there rather | ||||||||||||||||||||
| than duplicated here. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| This is a summary; for full detail (module-by-module responsibilities, | ||||||||||||||||||||
| configuration system, credential resolution, MCP isolation model) see | ||||||||||||||||||||
| `CLAUDE.md`, which is the actively-maintained reference. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ## WASM Grammars | ||||||||||||||||||||
|
|
||||||||||||||||||||
| The `.wasm` files in `grammars/` are pre-built and committed. You only need to | ||||||||||||||||||||
| rebuild them if you: | ||||||||||||||||||||
| `.wasm` grammar files are **not** committed to git — they're built from | ||||||||||||||||||||
| `devDependencies` into `grammars/` automatically via the `prepare` npm script | ||||||||||||||||||||
| (`npm run build:wasm`), which runs on every `npm install`. Each git worktree | ||||||||||||||||||||
| therefore needs its own `npm install` before its `grammars/` directory is | ||||||||||||||||||||
|
Comment on lines
+330
to
+333
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This says
Suggested change
|
||||||||||||||||||||
| populated (see "Working in multiple git worktrees?" above; `npm run doctor` | ||||||||||||||||||||
| detects a missing or incomplete `grammars/`). | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Rebuild manually if you: | ||||||||||||||||||||
|
|
||||||||||||||||||||
| - Add a new language | ||||||||||||||||||||
| - Upgrade a `tree-sitter-*` devDependency version | ||||||||||||||||||||
|
|
@@ -328,8 +354,8 @@ Use [GitHub Issues](https://github.com/optave/ops-codegraph-tool/issues) with: | |||||||||||||||||||
|
|
||||||||||||||||||||
| ## Code Style | ||||||||||||||||||||
|
|
||||||||||||||||||||
| - All source is plain JavaScript (ES modules) — no transpilation | ||||||||||||||||||||
| - [Biome](https://biomejs.dev/) is used for linting and formatting (config in `biome.json`) | ||||||||||||||||||||
| - Source is TypeScript (`src/`), compiled via `tsc`/`tsup`; run `npm run typecheck` to type-check without emitting | ||||||||||||||||||||
| - [Biome](https://biomejs.dev/) is used for linting and formatting (config in `biome.json`, scoped to `src/` and `tests/`) | ||||||||||||||||||||
| - Run `npm run lint` to check and `npm run lint:fix` to auto-fix | ||||||||||||||||||||
| - The pre-commit hook runs the linter automatically | ||||||||||||||||||||
| - Use `const`/`let` (no `var`) | ||||||||||||||||||||
|
|
||||||||||||||||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This says the TypeScript source is compiled with
tsup, but the documented build path usestscandtsupis not part of the repository scripts. A contributor following this can look for or invoke the wrong build tool instead of the actualnpm run buildpath.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!