Add edit-journal / speed up row inserts using current revision system#2363
Open
jh-RLI wants to merge 1 commit into
Open
Add edit-journal / speed up row inserts using current revision system#2363jh-RLI wants to merge 1 commit into
jh-RLI wants to merge 1 commit into
Conversation
Four performance changes to the row upload write path, all behavior- preserving for the API contract: - Partial index (_id) WHERE _applied = FALSE on the edit-journal meta tables (_<table>_insert/_edit/_delete): created with new meta tables and back-filled for existing ones by oedb migration e3b1f6c2d9a4. Turns the growing per-request sequential scans into index lookups. - set_applied marks applied rows with one set-based UPDATE (_id IN ... AND _applied = FALSE) instead of an N-term OR chain; apply_update and apply_deletion batch it once per stack instead of once per row. - apply_changes takes change_types and scans only the meta table(s) relevant to the operation - a pure insert no longer scans the edit and delete journals. - The duplicated apply per request is removed: data_insert/update/ delete apply within the request cursor's transaction; the extra view-level apply_changes calls (which re-scanned all three journals on a separate connection after commit) are gone. Also fixes a latent bug in the apply batching loop that dropped the first change of each type when a journal held mixed pending change types, and short-circuits meta-table DDL when the table already exists (required to avoid a lock deadlock between the request transaction and the index DDL, and skips redundant CREATE TABLE round trips). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title:
perf(api): speed up edit-journal apply path (#2362)Body:
Part of #2362 (Slice 1 — Row Upload quick wins). Four performance changes to
the row-upload write path, no API contract changes:
(_id) WHERE _applied = FALSEon the
_<table>_insert/_edit/_deletemeta tables. Created with new metatables; back-filled for all existing ones by oedb migration
e3b1f6c2d9a4(
alembic upgrade headon deploy). Turns the per-request"find unapplied changes" sequential scans into index lookups.
set_appliednow issues oneUPDATE … WHERE _id IN (…) AND _applied = FALSEinstead of an N-termORchain (5000-term OR for a 5000-row batch); update/delete stacks mark once
per stack instead of once per row.
apply_changestakeschange_types;a pure insert no longer scans the edit and delete journals.
data_insert/update/deletealready applywithin the request cursor's transaction; the duplicate view-level
apply_changescalls (re-scanning all three journals on a secondconnection after commit) are removed.
Also fixes a latent bug: the apply batching loop dropped the first change of
each type when a journal held mixed pending change types (and crashed on the
resulting empty batch). And
_create_if_missingnow short-circuits when themeta table exists — required to avoid a deadlock between the request
transaction and index DDL, and it removes a redundant
CREATE TABLE IF NOT EXISTSround trip from every meta-table access.Benchmark (batched inserts of 500 rows via
POST /rows/new, test client,against a journal pre-grown to N applied rows — simulating a long-lived
table; local Postgres 16, warm cache):
develop degrades linearly with journal size (~0.18 s per 1M journal rows per
batch, from the two grown data points); this branch is flat. Extrapolated to
a ~30M-row journal (a 2 GB dataset's worth), develop lands at ~80 rows/s —
matching the 150–400 rows/s observed in production — while this branch stays
at batch-local cost. The remaining flat ~0.9 s/batch is the double-write
architecture itself, which Slice 2+ (Bulk Upload) addresses.
Tests:
api.tests.test_rows20/20 OK; full suite 171 tests OK(
RDF_DATABASE_HOST=localhostneeded on the host for the fuseki test).Changelog entry included in
versions/changelogs/current.md.Summary of the discussion
Describe the findings of the discussion in the issue or meeting.
Type of change (CHANGELOG.md)
Features
(#)
Changes
(#)
Bugs
(#)
Removed
(#)
Documentation updates
(#)
Workflow checklist
Automation
Closes
Part of #2362
PR-Assignee
CONTRIBUTING.md
CHANGELOG.md
mkdocs
Reviewer
Reviewer Guidelines