Skip to content

Add edit-journal / speed up row inserts using current revision system#2363

Open
jh-RLI wants to merge 1 commit into
developfrom
feature-2362-edit-journal-apply-performance
Open

Add edit-journal / speed up row inserts using current revision system#2363
jh-RLI wants to merge 1 commit into
developfrom
feature-2362-edit-journal-apply-performance

Conversation

@jh-RLI

@jh-RLI jh-RLI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Title: perf(api): speed up edit-journal apply path (#2362)

Body:

Part of #2362 (Slice 1 — Row Upload quick wins). Four performance changes to
the row-upload write path, no API contract changes:

  • Partial index on unapplied journal rows(_id) WHERE _applied = FALSE
    on the _<table>_insert/_edit/_delete meta tables. Created with new meta
    tables; back-filled for all existing ones by oedb migration e3b1f6c2d9a4
    (⚠️ run alembic upgrade head on deploy). Turns the per-request
    "find unapplied changes" sequential scans into index lookups.
  • Set-based applied-markingset_applied now issues one
    UPDATE … WHERE _id IN (…) AND _applied = FALSE instead of an N-term OR
    chain (5000-term OR for a 5000-row batch); update/delete stacks mark once
    per stack instead of once per row.
  • Per-operation journal scanningapply_changes takes change_types;
    a pure insert no longer scans the edit and delete journals.
  • Single apply per requestdata_insert/update/delete already apply
    within the request cursor's transaction; the duplicate view-level
    apply_changes calls (re-scanning all three journals on a second
    connection after commit) are removed.

Also fixes a latent bug: the apply batching loop dropped the first change of
each type when a journal held mixed pending change types (and crashed on the
resulting empty batch). And _create_if_missing now short-circuits when the
meta table exists — required to avoid a deadlock between the request
transaction and index DDL, and it removes a redundant CREATE TABLE IF NOT EXISTS round trip from every meta-table access.

Benchmark (batched inserts of 500 rows via POST /rows/new, test client,
against a journal pre-grown to N applied rows — simulating a long-lived
table; local Postgres 16, warm cache):

Journal size develop this branch
~0 rows 474 rows/s (~1.05 s/batch) 525 rows/s (~0.95 s/batch)
1M rows 446 rows/s (~1.12 s/batch) 555 rows/s (~0.90 s/batch)
4M rows 304 rows/s (~1.65 s/batch) 537 rows/s (~0.93 s/batch)

develop degrades linearly with journal size (~0.18 s per 1M journal rows per
batch, from the two grown data points); this branch is flat. Extrapolated to
a ~30M-row journal (a 2 GB dataset's worth), develop lands at ~80 rows/s —
matching the 150–400 rows/s observed in production — while this branch stays
at batch-local cost. The remaining flat ~0.9 s/batch is the double-write
architecture itself, which Slice 2+ (Bulk Upload) addresses.

Tests: api.tests.test_rows 20/20 OK; full suite 171 tests OK
(RDF_DATABASE_HOST=localhost needed on the host for the fuseki test).
Changelog entry included in versions/changelogs/current.md.

Summary of the discussion

Describe the findings of the discussion in the issue or meeting.

Type of change (CHANGELOG.md)

Features

  • Add a new functionality
    (#)

Changes

  • Update existing functionality
    (#)

Bugs

  • Fixed a problem with
    (#)

Removed

  • Remove a broken link
    (#)

Documentation updates

  • Updated documentation for
    (#)

Workflow checklist

Automation

Closes

Part of #2362

PR-Assignee

Reviewer

  • 🐙 Follow the
    Reviewer Guidelines
  • 🐙 Provided feedback and show sufficient appreciation for the work done

Four performance changes to the row upload write path, all behavior-
preserving for the API contract:

- Partial index (_id) WHERE _applied = FALSE on the edit-journal meta
  tables (_<table>_insert/_edit/_delete): created with new meta tables
  and back-filled for existing ones by oedb migration e3b1f6c2d9a4.
  Turns the growing per-request sequential scans into index lookups.
- set_applied marks applied rows with one set-based UPDATE (_id IN ...
  AND _applied = FALSE) instead of an N-term OR chain; apply_update and
  apply_deletion batch it once per stack instead of once per row.
- apply_changes takes change_types and scans only the meta table(s)
  relevant to the operation - a pure insert no longer scans the edit
  and delete journals.
- The duplicated apply per request is removed: data_insert/update/
  delete apply within the request cursor's transaction; the extra
  view-level apply_changes calls (which re-scanned all three journals
  on a separate connection after commit) are gone.

Also fixes a latent bug in the apply batching loop that dropped the
first change of each type when a journal held mixed pending change
types, and short-circuits meta-table DDL when the table already exists
(required to avoid a lock deadlock between the request transaction and
the index DDL, and skips redundant CREATE TABLE round trips).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jh-RLI jh-RLI requested a review from wingechr July 3, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant