Skip to content

Exclude transient npm/git artifacts from component file watcher#809

Open
kriszyp wants to merge 2 commits into
mainfrom
kris/watcher-ignore-tightening
Open

Exclude transient npm/git artifacts from component file watcher#809
kriszyp wants to merge 2 commits into
mainfrom
kris/watcher-ignore-tightening

Conversation

@kriszyp
Copy link
Copy Markdown
Member

@kriszyp kriszyp commented May 27, 2026

Summary

Tighten EntryHandler's chokidar ignored callback so npm and git side-effects produced during a component deploy don't drive auto-reload restart storms.

Why

Refs harper#488. Each Harper component runs a chokidar watcher over its directory, recursively. Linux fs.watch is non-recursive, so chokidar consumes one inotify watch per descended directory. The watcher's ignored callback previously only excluded node_modules (via a loose includes('/node_modules') check). When npm install ran inside a component during deploy, it wrote:

  • npm-debug.log, pnpm-debug.log, yarn-{error,debug}.log
  • .tmp-* atomic-rename temp directories
  • .git internals (when a component is a git checkout)

These all leaked through the filter, fired Scope.requestRestart() events, and amplified the restart churn — which then thrashes the watcher pool during teardown/recreate and can hit ENOSPC (the inotify limit).

This PR is one of three independent mitigations for #488; PR 2 adds an ENOSPC/EMFILE polling fallback to all three watchers, PR 3 suppresses auto-reload entirely during a deploy.

Where to look

  • components/EntryHandler.ts:193-220 — the new ignored callback. The substring check is replaced with anchored regexes ((?:^|\/)…(?:\/|$)) so similar-named user paths aren't false-matched.
  • The relativePath derivation was simplified — the previous redundant startsWith(directory + '/') then startsWith(directory) ternary collapsed into a single slice. Leaves leading / on inner paths and an empty string when the path is the component dir itself, which the regex anchors handle correctly.
  • Six new unit tests in unitTests/components/EntryHandler.test.js cover each excluded pattern plus a prefix/suffix over-match negative case.

Cross-model review notes

  • Codex flagged that pnpm-debug.log (not .pnpm-debug.log) is the actual filename pnpm writes — fixed.
  • Gemini flagged that the log-file regex wasn't end-anchored, so npm-debug.log.txt and npm-debug.log/ would have been over-matched — fixed by appending (?:\/|$). Also suggested an explicit suffix-overmatch test case, added.

Potential concerns to focus on

  • Confidence that chokidar 4 passes absolute paths to ignored when cwd is set. Verified by reading chokidar's _isIgnorednormalizeIgnored (chokidar esm/index.js:110,633): function matchers pass through unchanged, and the upstream caller resolves to absolute via getAbsolutePath. The existing node_modules filter (which depends on this assumption) already works, so the behavior is established.
  • Plugins that explicitly relied on receiving change events for an in-tree npm-debug.log or .git/HEAD will no longer see them. No such plugin is known to exist; the watcher contract has always been "files matching your glob, minus implementation-defined noise."
  • Generated by an LLM (Claude Opus 4.7).

Comment thread unitTests/components/EntryHandler.test.js Outdated
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 27, 2026

Reviewed; no blockers found.

kriszyp added a commit that referenced this pull request May 27, 2026
Per Claude review on #809: the regex in EntryHandler.ts:216 has four
alternations (npm-debug, yarn-error, yarn-debug, pnpm-debug) but the
fixture only exercised three. Add yarn-debug.log so a typo in that
branch wouldn't go undetected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kriszyp
Copy link
Copy Markdown
Member Author

kriszyp commented May 27, 2026

The two failing CI checks on the initial run are unrelated to this change:

  • Format Check — fails on `resources/databases.ts`, which this PR does not touch. Verified locally: `git diff origin/main -- resources/databases.ts` is empty on this branch, and `prettier --check` reports the same warning when run against `origin/main`. The formatting drift is pre-existing on main.
  • Integration API Tests (Windows, Node.js v24) — `ECONNREFUSED 127.0.0.1:9925` in `integrationTests/apiTests/tests/28_transactionLogs.mjs`, indicating the Harper test server died mid-suite. The other three Integration API Tests jobs (Node v20, v22, v24 on Linux) passed cleanly. This change only modifies the chokidar `ignored` callback in `EntryHandler` and adds unit tests — no path that the integration suite exercises differently. Flaky Windows job.

A re-run on the latest push (`4d9df74`) should clear them. The follow-up commit addresses the Claude bot review comment about `yarn-debug.log` coverage.

— Claude Opus 4.7

@kriszyp
Copy link
Copy Markdown
Member Author

kriszyp commented May 27, 2026

Re-run produced a different Windows failure than before:

  • First run: `ECONNREFUSED 127.0.0.1:9925` in `28_transactionLogs.mjs`
  • This run: `EPERM: operation not permitted, rename 'harper-config.yaml.5280.0.dffd26e0.tmp' -> 'harper-config.yaml'` in `15_customFunctionsAndComponents.mjs:319` (the `drop deploy-test-gh` case)

Two different Windows failures on two different test files across two runs of an unchanged branch is the signature of pre-existing Windows flakiness, not a regression from this PR.

For the specific EPERM here: `harper-config.yaml` is watched by `RootConfigWatcher.ts`, which this PR does not modify (only `EntryHandler.ts` is touched, and that watches per-component directories). The `drop_component` operation atomically renames a tmp file over the config, which Windows refuses while chokidar holds the file open — a known chokidar+Windows interaction (related to harper#488 in spirit but pre-existing).

Format Check is still red on the unchanged `resources/databases.ts` drift on main, same as before.

Neither failure is actionable from this PR. Leaving it as-is.

— Claude Opus 4.7

@kriszyp kriszyp marked this pull request as ready for review May 27, 2026 04:44
kriszyp added a commit that referenced this pull request May 30, 2026
Per Claude review on #809: the regex in EntryHandler.ts:216 has four
alternations (npm-debug, yarn-error, yarn-debug, pnpm-debug) but the
fixture only exercised three. Add yarn-debug.log so a typo in that
branch wouldn't go undetected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kriszyp kriszyp force-pushed the kris/watcher-ignore-tightening branch from f25f8c8 to 3a25ceb Compare May 30, 2026 13:18
@kriszyp
Copy link
Copy Markdown
Member Author

kriszyp commented May 30, 2026

Triage of post-rebase CI failures — both unrelated:

  1. Unit Test (Node.js v24): failed on `unitTests/apiTests/ws-test.mjs` — `default subscribe on WS` timed out at 5s. The same test passed on v22 and v26 in the same workflow run, so this is a transient WebSocket-subscribe flake bracketed by green Node versions. Re-ran the job; second run pending.

  2. Integration Tests 2/4 (v22, v24, v26): failed in `Start 4.x server and test upgrade` → `upgrade and migrate LMDB to RocksDB` with `{"error":"Must login"}` plus several `TypeError: Cannot read properties of undefined (reading 'call')` errors in Harper logs. Identical failure pattern across all three Node versions = real bug, not flake — but it's on main, not this PR:

    ```
    gh run list --branch main --workflow "Integration Tests" --status completed
    completed failure Merge pull request feat(deploy): Slice B2 — peer-side blob reads + install-line streaming #760 from HarperFast/feat/deployment-tracking-slic…
    completed failure fix(analytics): continue instead of return so later dbs still get mer…
    completed failure Fix jsLoader to support ESM-only packages (Fix jsLoader to support ESM-only packages #837)
    completed failure Merge pull request Suppress watcher restart storms during component deploy #822 from HarperFast/kris/suppress-autoreload-duri…
    ```

    Four consecutive Integration Tests failures on main, predating this rebase. My branch only modifies `components/EntryHandler.ts` and its unit test — no path the LMDB→RocksDB upgrade test exercises differently.

Neither failure is actionable from this PR. Leaving as-is.

— Claude Opus 4.7

kriszyp and others added 2 commits May 31, 2026 06:14
Tightens EntryHandler's chokidar `ignored` callback so npm and git
side-effects produced during a component deploy don't drive auto-reload
restart storms. Previously only `node_modules` (via a loose substring
check) was excluded; npm-debug.log, pnpm-debug.log, yarn-{error,debug}.log,
.tmp-* atomic-rename temp directories, and .git internals all leaked
through, fired Scope.requestRestart() events, and amplified the watcher
churn that can exhaust the inotify limit (harper#488).

Replaces the substring check with anchored regexes so similar-named user
paths (notnode_modules, node_modules_not, npm-debug.log.txt) are not
false-matched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Claude review on #809: the regex in EntryHandler.ts:216 has four
alternations (npm-debug, yarn-error, yarn-debug, pnpm-debug) but the
fixture only exercised three. Add yarn-debug.log so a typo in that
branch wouldn't go undetected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kriszyp kriszyp force-pushed the kris/watcher-ignore-tightening branch from 3a25ceb to 432efb2 Compare May 31, 2026 12:14
@kriszyp
Copy link
Copy Markdown
Member Author

kriszyp commented May 31, 2026

Further triage on the Windows API test failure:

Harper's test process crashes mid-suite on Windows (ECONNRESET, then ECONNREFUSED for every subsequent request). Different runs of the same suite crash at different test points:

The test setup explicitly flags this Windows-specific Harper behavior:
```
﹣ Environment Cleanup (1.664ms) # Skipping dropSchema on Windows to avoid HarperDB crash.
```

The crash isn't deterministic on a particular test — it's a Windows-specific Harper-process termination that varies run-to-run. The frequency on this PR is unfortunate but the change scope (per-component-watcher `ignored` regex + 6 test cases) doesn't touch any code path the Windows crash exercises (auth, schemas, jobs, REST, transaction logs, components-with-computed-props).

Triggered another rerun of just the failed job. If it fails a third time on the same SHA, recommend merging anyway and accepting the Windows flake — three Linux Integration API Tests jobs pass cleanly.

— Claude Opus 4.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant