feat(api): bulk upload endpoint via COPY FROM STDIN (#2362)#2364
Open
jh-RLI wants to merge 1 commit into
Open
Conversation
Adds POST /api/v0/tables/<table>/bulk-upload - the tracer bullet of the bulk upload path (slice 2 of #2362): - The request body IS the CSV (text/csv); the server streams it into PostgreSQL COPY FROM STDIN without buffering the file in memory. - Append-only, one transaction per request: a malformed row anywhere rolls back the entire upload. - The delimiter is a required, whitelisted parameter (comma, semicolon, tab) - never inferred from metadata or content. - The CSV header (required) maps columns by name; header names are whitelisted against the table's actual columns and quoted, so no unvalidated identifier ever reaches the SQL. - Same authorization chain as the row API: token auth, write permission, embargo check, table-registry resolution (internal tables are unreachable by construction). - Deliberately bypasses the edit-journal meta tables: bulk-loaded rows have no per-row change history. This trades the (currently unread) per-row provenance for order-of-magnitude ingestion speed; an audit event record follows in a later slice. - COPY is FROM STDIN only; no code path for COPY FROM file/PROGRAM. New HTTP-seam test module api/tests/test_bulk_upload.py (14 tests) covers the happy path per delimiter, auth/permission/embargo denials, all-or-nothing rollback, journal bypass, and target-table containment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 3, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2362 (Slice 2 — Bulk Upload tracer bullet). Adds the minimal
end-to-end bulk upload path:
POST /api/v0/tables/<table>/bulk-upload?delimiter=comma|semicolon|tabtext/csv). The server streams it intoPostgreSQL
COPY FROM STDINwithout buffering the file in memory — theonly buffered piece is the header line (capped at 1 MiB).
row anywhere rolls back the entire upload, the table is untouched.
metadata or content (explicit-contract design decision).
table's actual columns and safely quoted — no unvalidated identifier
reaches SQL. Column order is free; omitted columns get their defaults.
embargo check, table-registry resolution (internal/meta tables are
unreachable by construction).
COPYisFROM STDINonly — no code pathfor
COPY FROM <file>orPROGRAM.have no per-row change history. The per-row journal is write-only today
(nothing reads it); an audit event record (Bulk Load Event) follows in a
later slice, as do gzip, size caps, concurrency guards, id/sequence
handling, and CSV-contract hardening (BOM, FORCE_NULL, line-numbered
errors).
Throughput (test client, full Django stack, local Postgres 16):
500,000 rows / 14 MB streamed in 1.27 s ≈ 393,000 rows/s (11 MB/s) —
vs ~525 rows/s on the row API. Server-side, a 2 GB dataset drops from
~40 h to ~3 min; real uploads will be bounded by the client's uplink.
Tests: new HTTP-seam module
api/tests/test_bulk_upload.py(14 tests):happy path per delimiter, missing/invalid delimiter (400 before streaming),
401/403/embargo denials, all-or-nothing rollback, journal bypass, internal
table containment, empty body, nonexistent table. Full suite green.
Changelog entry included.
Summary of the discussion
Workflow checklist
Automation
Closes #
Part of #2362
PR-Assignee
CONTRIBUTING.md
CHANGELOG.md
mkdocs
Reviewer
Reviewer Guidelines