Skip to content

feat(api): bulk upload endpoint via COPY FROM STDIN (#2362)#2364

Open
jh-RLI wants to merge 1 commit into
developfrom
feature-2362-bulk-copy-upload
Open

feat(api): bulk upload endpoint via COPY FROM STDIN (#2362)#2364
jh-RLI wants to merge 1 commit into
developfrom
feature-2362-bulk-copy-upload

Conversation

@jh-RLI

@jh-RLI jh-RLI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Part of #2362 (Slice 2 — Bulk Upload tracer bullet). Adds the minimal
end-to-end bulk upload path:

POST /api/v0/tables/<table>/bulk-upload?delimiter=comma|semicolon|tab

  • The request body IS the CSV (text/csv). The server streams it into
    PostgreSQL COPY FROM STDIN without buffering the file in memory — the
    only buffered piece is the header line (capped at 1 MiB).
  • Append-only, all-or-nothing: one transaction per request; a malformed
    row anywhere rolls back the entire upload, the table is untouched.
  • Delimiter is a required, whitelisted parameter — never inferred from
    metadata or content (explicit-contract design decision).
  • Header maps columns by name; header names are whitelisted against the
    table's actual columns and safely quoted — no unvalidated identifier
    reaches SQL. Column order is free; omitted columns get their defaults.
  • Same authorization chain as the row API: token auth, write permission,
    embargo check, table-registry resolution (internal/meta tables are
    unreachable by construction). COPY is FROM STDIN only — no code path
    for COPY FROM <file> or PROGRAM.
  • Deliberately bypasses the edit-journal meta tables: bulk-loaded rows
    have no per-row change history. The per-row journal is write-only today
    (nothing reads it); an audit event record (Bulk Load Event) follows in a
    later slice, as do gzip, size caps, concurrency guards, id/sequence
    handling, and CSV-contract hardening (BOM, FORCE_NULL, line-numbered
    errors).

Throughput (test client, full Django stack, local Postgres 16):
500,000 rows / 14 MB streamed in 1.27 s ≈ 393,000 rows/s (11 MB/s)
vs ~525 rows/s on the row API. Server-side, a 2 GB dataset drops from
~40 h to ~3 min; real uploads will be bounded by the client's uplink.

Tests: new HTTP-seam module api/tests/test_bulk_upload.py (14 tests):
happy path per delimiter, missing/invalid delimiter (400 before streaming),
401/403/embargo denials, all-or-nothing rollback, journal bypass, internal
table containment, empty body, nonexistent table. Full suite green.
Changelog entry included.

Summary of the discussion

Workflow checklist

Automation

Closes #

Part of #2362

PR-Assignee

Reviewer

  • 🐙 Follow the
    Reviewer Guidelines
  • 🐙 Provided feedback and show sufficient appreciation for the work done

Adds POST /api/v0/tables/<table>/bulk-upload - the tracer bullet of the
bulk upload path (slice 2 of #2362):

- The request body IS the CSV (text/csv); the server streams it into
  PostgreSQL COPY FROM STDIN without buffering the file in memory.
- Append-only, one transaction per request: a malformed row anywhere
  rolls back the entire upload.
- The delimiter is a required, whitelisted parameter (comma, semicolon,
  tab) - never inferred from metadata or content.
- The CSV header (required) maps columns by name; header names are
  whitelisted against the table's actual columns and quoted, so no
  unvalidated identifier ever reaches the SQL.
- Same authorization chain as the row API: token auth, write
  permission, embargo check, table-registry resolution (internal
  tables are unreachable by construction).
- Deliberately bypasses the edit-journal meta tables: bulk-loaded rows
  have no per-row change history. This trades the (currently unread)
  per-row provenance for order-of-magnitude ingestion speed; an audit
  event record follows in a later slice.
- COPY is FROM STDIN only; no code path for COPY FROM file/PROGRAM.

New HTTP-seam test module api/tests/test_bulk_upload.py (14 tests)
covers the happy path per delimiter, auth/permission/embargo denials,
all-or-nothing rollback, journal bypass, and target-table containment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant