Improve Builder source refresh progress for large Content databases#1759
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
Visual recap — generation failedThe visual recap could not be generated for this pull request. This is informational only and does not block the PR. Diagnostic: No plan URL: create-visual-recap failed 422 Unprocessable Entity: {"error":"Could not parse expression with acorn"} Agent output: create-visual-recap failed 422 Unprocessable Entity: {"error":"Could not parse expression with acorn"} |
Contributor
Author
|
CI pulse update for draft PR #1759:
PR remains draft-only. No ready-for-review, merge, or auto-merge changes made. |
…og-scale-baseline # Conflicts: # templates/content/AGENTS.md
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- Imported Builder rows carry their entry identity through attach/resync (importedEntriesByDocumentId), so row linking no longer depends on ambiguous natural-key matching for fresh imports; duplicate keys stay ambiguous instead of guessing - Source-imported rows (read-adapter provenance) are excluded from outbound pending-create staging — rows that arrived FROM Builder can never register as local rows to push back - Body diffs are computed only against a real source-row baseline; hydrated bodies no longer stage as outbound 'body change' sets with zero user edits - prepare-builder-source-review caps preparation at 100 rows per call (pending/staged first) and reports total/prepared counts; the review dialog windows rendering with Show more and explains capped batches - New review-dialog strings localized across all 10 catalogs Known residual: rows orphaned by pre-fix resyncs (no source row) can still stage phantom creates; needs a relink/repair pass (PR follow-up).
…adge overflow, optimistic columns - The continuation bar stays visible after row fetch while bodies are still hydrating and shows 'rows fetched / syncing N of M bodies' instead of implying completion - Opening a pending row (preview, prev/next, or full page) promotes that document to the front of the body-hydration queue - Count badges cap at 99+ with min-width layout everywhere they render (settings trigger, settings rows, hidden/sorts/filters chips) - Adding a source-field column inserts the header optimistically with placeholder cells, rolls back on error, and reconciles on success; optimistic and success patches now write through the document-scoped query filter so the rendered (paginated) table updates immediately instead of waiting for the refetch - New progress strings localized across all 10 catalogs
- Client-side hydration pump in both twins: while a builder source reports pending/hydrating bodies and rows are not fetching, batches of process-builder-body-hydration run back to back (shared, tested shouldPumpBuilderBodyHydration; per-key dedupe and client-error backoff bound the loop — failed jobs drain pending into error, so worst case terminates in total/batch cycles). Server kick failures are logged instead of swallowed. - Rows enqueued for hydration but never processed serialize as pending (queue-aware serialization), and the editor gate blocks any non-hydrated Builder body — the empty-editable-body data-loss hole for stalled rows is closed; local/non-Builder rows stay editable - Continuation banner uses per-source high-water marks: no reverting to the loading-rows phase after fetch completes, no backward hydrated counts, and the rows sub-line shows the real total - Comment thread activation from the editor DOM handler is deferred via queueMicrotask — the setState-during-render warning (DocumentEditorBody via VisualEditor) caller
- An untouched preview no longer becomes a dirty save: the empty-to- empty-block normalization is suppressed unless the loaded body was non-empty - Preview saves carry loadedUpdatedAt and update-document keeps the stored body when an effectively-empty save arrives against a newer non-empty document (opt-in guard; clients not sending the baseline are unaffected, and a legitimate clear with a current baseline still works) - Pending-hydration save suppression returns an explicit skipped result that rolls the controller back to last-saved, so hydrated content is adopted when it lands instead of being re-clobbered - The open preview re-renders hydrated content when it arrives, and the syncing gate also consults the fresher get-document hydration state, not just the row snapshot
…ever
- Lossless BuilderText MDX escapes bare '<', '{', '}' so real-world
prose (100% Free, useVisibleTask$(), braced snippets) survives the
MDX/acorn parse; applyTextData unescapes before comparison and
writeback so readable content and baseline equality are unchanged
- Hydration jobs cap at 5 attempts: the item goes terminal error with
the last message and the queue row is deleted
- The client pump stops on no-progress batches (succeeded 0 with
remaining > 0 sets the backoff key) — automatic pumping only
continues after a batch hydrates at least one body or drains the
queue
- Finished-with-failures is surfaced: banner shows the failed count
and terminal-error rows show an explicit could-not-sync notice while
staying editable
- New strings localized across all 10 catalogs
…, and collab flushSync - Full-page editor saves carry loadedUpdatedAt, so the stale-empty server guard now protects the collab/page path like the preview: opening an article mid-hydration can no longer erase its hydrated body (the 'skipped' row was actually an erased row) - Outbound source diffs compare normalized-to-normalized values and writable fields carry the mapped property type, so materialized source-field values (Author/Date columns) no longer register as 99+ phantom local edits; date normalization regression added - Collab reconcile applies external content on a timer task instead of a microtask (core changeset) — kills the flushSync-from- lifecycle console spam at useCollabReconcile - refresh-notion-sync-status without a documentId is a 400, and the client hook trims/disables empty ids (was repeated 500s) - Row continuation refires with capped exponential backoff (5s-30s) until hasMore is false instead of parking on a manual Continue - A database (or any source chain leading back to it) is excluded from its own Add-source local-table candidates
…xists The exclude list was seeded from source?.databaseId, which is null on a fresh database — exactly when self-selection is possible. The settings panel now always passes the database's document id, and the action matches exclusion ids against document ids as well as database ids. Regression test covers the no-source-attached case.
- loadSourceSnapshot reads source rows, items, document bodies, and property values in one transaction, so the outbound change-set builder can never see a mixed-time snapshot (old baseline + newly hydrated body) — the transient 'body diff' phantom entries that stacked to a 99+ badge during a 497-row hydration are gone at the source; zero-changesets-at-every-point regression test added - A hydration processor that loses a concurrent race no longer resets the winner's hydrated status to pending: both the document-changed branch and the already-deleted-job branch only mark pending when the queue row still exists (a newer job version); otherwise the completing processor owns the final status. Previously the loser stranded items as pending with no queue row — a permanent zombie matching the live 495/497 stall tail - Streaming-hydration test drains the unawaited background kick before asserting completion, and asserts converted (non-empty) bodies rather than raw source-value equality
…ents
- Core: top-level async transactions on the better-sqlite3 connection
are serialized through a per-connection queue (AsyncLocalStorage
detects same-task nesting, which keeps the direct savepoint path).
Concurrent get-content-database reads against batch H's snapshot
transaction interleaved savepoints ('no such savepoint' 500s and the
'Builder row loading hit a snag' banner); concurrency regression
spec added against the real implementation (the prior spec only
exercised an inlined copy)
- get-document throws 404 (not 500) for deleted or inaccessible
documents, ending the Internal Server Error console flood from tabs
still polling documents of a deleted database
…ng snapshot reads, AgentPanel flushSync
- Reseed preserves ids for ALL source fields matched by sourceFieldKey
(not just bound ones), and add-content-database-source-field-property
falls back to sourceId+sourceFieldKey when a stale id misses —
'Source field not found' column-add regression fixed
- Unknown hydration state on a source-linked row gates as pending on
first paint (no editable empty-body flash before the document
response lands); local row-union rows unaffected; opened rows with
unknown state still get hydration promotion
- loadSourceSnapshot reads optimistically (COUNT+MAX(updatedAt)
markers, up to 3 re-reads on mid-read commits) instead of joining
the serialized transaction queue — snapshot reads no longer stall
behind hydration write batches; zero-phantom invariant stays green
- Third flushSync caller was core AgentPanel's navigate(...,
{flushSync: true}) in an effect — deferred (changeset added)
Forensically root-caused (audit log + DB): the open-row promotion rebuilt its hydration entry from the stored source row, which lacks body content in the mid-attach window, REPLACED the full-body queue entry with it, and the limit-1 job then wrote empty content and stamped the item hydrated — the 'overwrite' bug that survived five QA rounds of save-path guards. - enqueueBuilderBodyHydration never downgrades an existing queue entry that has body content to one without (priority still bumps) - A hydration job whose resolved content is empty leaves the item pending and writes nothing — hydrated+empty is now unrepresentable from this path - Resync re-enqueues rows stamped hydrated with effectively-empty content whose source row carries a real body (heals already clobbered rows on the next refresh) - BUILDER_BODY_HYDRATION_DELAY_MS (dev-only) holds the race window open for deterministic browser reproduction - Regression tests written first: verified failing 2/2 on pre-fix code, passing 48/48 after
- BuilderSourceContinuationBar renders status + progress only during healthy row-fetch/body-sync states; a Retry button appears solely for the parked client-error/backoff and hydration-failed states (both twins; unused 'continue' key removed from all 10 catalogs) - The collab seed path's setContent (useCollabReconcile seed-when- empty branch — the last flushSync-from-lifecycle caller) runs in a cancellable timer task with the guard and bookkeeping inside the deferred task, matching the apply-path fix (core changeset)
Forensically proven second empty-writer (audit log): hydration wrote the body; the open editor's identical echo-save no-op'd without bumping updatedAt; a stale pre-hydration editor state then saved <empty-block/> with an EQUAL loadedUpdatedAt baseline and the strict '>' guard let it through, destroying the body and wedging hydration. - update-document accepts loadedContentWasEmpty; an effectively-empty save over a non-empty body is rejected when the client's loaded baseline was empty, regardless of timestamps — legitimate clears (non-empty baseline, current timestamp) still work - Preview controller and full-page editor attest the flag and drop/ rebase stale empty pending state when fresher non-empty content arrives (both twins) - Hydration body-write CAS treats '' and <empty-block/> as the same nothing; the repair pass covers pending as well as hydrated empty rows — wedged rows heal and the pump resumes on the next resync - Framework route failures are now logged with route/status/message (core changeset) — the Round 9 500s produced zero log lines - Failing-test-first verified independently: 1 test fails on the pre-fix guard, 70 pass after
…r parent editor callbacks - A hydration job resolving empty content rebuilds its entry from the current stored source row (the body usually lands seconds after the open-row promotion); a rebuilt entry with body content upgrades the queue payload and hydrates immediately. Still unbuildable after 5 attempts: the queue row is deleted, the item stays pending with a 'body not yet available from Builder' note, and the resync repair owns healing — no job can retry unboundedly on any path (the failure-only cap missed clean empty-resolves; live incident showed attempts=73). Root cause of the masking: queued entry values merged OVER current source-row values, hiding a later-arriving body. Failing-first verified independently: 2 tests fail pre-fix, 91 pass - Framework routes classify client aborts (message 'aborted', ECONNRESET, destroyed req/res) as debug-only non-errors and skip the JSON 500 attempt (core changeset) — kills the poll/awareness 500 noise seen during navigation - Parent editor callbacks (onChange persist, onActivateThread, onJoinTitle) defer via setTimeout instead of queueMicrotask — microtasks can run inside React's commit and triggered the last setState-during-render warning; resolveNotionPageLink ruled out as a setter path
…og-scale-baseline # Conflicts: # packages/core/src/client/rich-markdown-editor/useCollabReconcile.ts # packages/core/src/client/use-action.spec.ts # packages/core/src/client/use-action.ts # packages/core/src/client/use-db-sync.ts # templates/content/actions/update-document.ts # templates/content/app/components/editor/DocumentDatabase.tsx # templates/content/app/components/editor/DocumentEditor.tsx # templates/content/app/components/editor/database/DatabaseView.tsx # templates/content/shared/builder-mdx.test.ts
Contributor
Author
|
Status update after the Builder body hydration tail fix:
PR remains draft-only; no merge/auto-merge/ready-for-review action taken. |
Contributor
Author
|
Status update for the React console warning Alice flagged:
PR remains draft-only; no merge/auto-merge/ready-for-review action taken. |
Contributor
Author
|
Status update for the phantom Builder body-change card Alice flagged:
PR remains draft-only; no merge/auto-merge/ready-for-review action taken. |
3mdistal
marked this pull request as ready for review
July 6, 2026 18:37
steve8708
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes Builder CMS-backed Content databases usable at docs/blog scale. Large Builder sources now load in bounded pages, keep the table usable while rows and article bodies continue hydrating, and expose clearer progress, freshness, retry, and error states instead of long blocking waits.
It also hardens the Builder body hydration path so opened/high-priority rows become usable without waiting for the entire source, and so hydration does not create phantom local edits or overwrite source-authored bodies.
What Changed
Verification
Local and CI coverage across this lane has included:
pnpm exec vitest --run actions/_builder-cms-read-client.test.ts actions/_database-source-utils.test.ts actions/resync-content-database-source.db.test.ts app/components/editor/database/DatabaseView.test.tspnpm exec vitest --run actions/_database-source-utils.test.ts actions/resync-content-database-source.db.test.tspnpm exec vitest --run app/components/editor/body-hydration.test.ts actions/resync-content-database-source.db.test.tspnpm --filter @agent-native/core exec vitest --run src/collab/client.spec.ts src/collab/presence.spec.ts src/client/rich-markdown-editor/RichMarkdownEditor.collab.spec.ts --maxWorkers=25%pnpm typecheckpnpm --filter @agent-native/core typecheckgit diff --checkNotes