feat(api): bulk load events — audit record per upload attempt (#2362)#2368
Open
jh-RLI wants to merge 4 commits into
Open
feat(api): bulk load events — audit record per upload attempt (#2362)#2368jh-RLI wants to merge 4 commits into
jh-RLI wants to merge 4 commits into
Conversation
Adds POST /api/v0/tables/<table>/bulk-upload - the tracer bullet of the bulk upload path (slice 2 of #2362): - The request body IS the CSV (text/csv); the server streams it into PostgreSQL COPY FROM STDIN without buffering the file in memory. - Append-only, one transaction per request: a malformed row anywhere rolls back the entire upload. - The delimiter is a required, whitelisted parameter (comma, semicolon, tab) - never inferred from metadata or content. - The CSV header (required) maps columns by name; header names are whitelisted against the table's actual columns and quoted, so no unvalidated identifier ever reaches the SQL. - Same authorization chain as the row API: token auth, write permission, embargo check, table-registry resolution (internal tables are unreachable by construction). - Deliberately bypasses the edit-journal meta tables: bulk-loaded rows have no per-row change history. This trades the (currently unread) per-row provenance for order-of-magnitude ingestion speed; an audit event record follows in a later slice. - COPY is FROM STDIN only; no code path for COPY FROM file/PROGRAM. New HTTP-seam test module api/tests/test_bulk_upload.py (14 tests) covers the happy path per delimiter, auth/permission/embargo denials, all-or-nothing rollback, journal bypass, and target-table containment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 3 of #2362, on top of the bulk upload tracer bullet: - Header preflight before the body streams: reject duplicate column names, names not in the table, and missing required columns (NOT NULL without default), each with a 400 naming the offenders. - Strip a UTF-8 BOM from the header (Excel exports). - FORCE_NULL on all uploaded columns: an empty field is NULL whether quoted or not - a deliberate deviation from COPY's native CSV rule, because many writers quote every field and would silently store empty strings instead of NULLs. - Sanitized failure responses: the database's data-level message plus the CSV line number and column (header-adjusted), never raw SQL, server context dumps, or internal paths - including the no-diagnostics fallback (lost connection), which stays generic. Test module grows to 20 HTTP-seam tests covering each contract rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#2362) Slice 4 of #2362. Clients may include or omit the id column: - Omitted: the table's id sequence assigns ids as usual. - Included: after COPY, still inside the upload's transaction, the id sequence is advanced to the table's max(id) via setval(GREATEST(...)), so a subsequent row-API insert cannot hit a duplicate key. GREATEST plus a pg_advisory_xact_lock on the sequence keep the sequence from ever moving backwards, including under concurrent uploads (setval is non-transactional, so racing reads could otherwise regress it). - Uploads that RAISE the table's max(id) above a generous sanity bound (2^48) are rejected and rolled back, so a single upload cannot exhaust the id sequence for every writer of a shared table. The bound only judges ids introduced by the upload itself: a pre-existing high id (the row API enforces no bound) does not poison the table for future bulk uploads. Test module grows to 25 HTTP-seam tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 5 of #2362. Bulk uploads bypass the per-row edit journal; the new BulkLoadEvent model is their provenance: - Every authenticated, authorized attempt - successful or failed - is recorded: table, user, timestamp, status (success / validation-error / copy-error / embargo), sanitized error message, and bytes received. Decorator-level denials (401/403/404) deliberately create no events so anonymous requests cannot write database rows. - Successes additionally record the row count and the exact id range of the loaded rows (found via xmin = current transaction, so it is correct for both explicit and sequence-assigned ids); the response references the event id. The id range is the forensics handle for block-deleting a mistaken or malicious upload. - Events live in the Django database, so a failure event survives the data transaction's rollback by construction. Event creation is best-effort: a failure to write the audit record is logged but never masks the upload's actual outcome. - Events are listed, filterable, and immutable in the Django admin. Test module grows to 30 HTTP-seam tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of the discussion
This PR stacks ontop of #2364 and #2366 and #2367 wait until both are merged and rebase this PR.
Part of #2362 (Slice 5 — Bulk Load Event). Bulk uploads bypass the per-row
edit journal; the new
BulkLoadEventmodel is their provenance:failure: table, user, timestamp, status (
success/validation-error/copy-error/embargo), sanitized error message, bytes received.Decorator-level denials (401/403/404) deliberately create no events, so
anonymous requests can never write database rows.
rows — identified via
xmin = pg_current_xact_id(), so it's correct forboth explicit and sequence-assigned ids. This is ADR 0001's forensics
handle: a poisoned upload is deletable as a block. The response now also
carries
event_idandid_range.live in the Django database, the data in the oedb database. Event
creation is best-effort: a failed audit write is logged but never masks
the upload's real outcome (no 500 on committed data, no swallowed error).
table and user, and immutable (add/change/delete all disabled).
Review-pass fixes worth noting: event creation wrapped so it can't mask the
real error; "every attempt" wording narrowed to the honest scope; byte
counts on validation failures now include body bytes already received.
Tests: module grows to 30 HTTP-seam tests (success event with id range,
sequence-assigned range, copy-error and validation-error failure events with
rollback, embargo event, admin registration). Full suite green. Changelog
entry included.
Workflow checklist
Automation
Closes #
PR-Assignee
CONTRIBUTING.md
CHANGELOG.md
mkdocs
Reviewer
Reviewer Guidelines