Skip to content

refactor: rewrite export system with streaming and native macOS patterns#782

Merged
datlechin merged 9 commits intomainfrom
refactor/export-streaming-v2
Apr 18, 2026
Merged

refactor: rewrite export system with streaming and native macOS patterns#782
datlechin merged 9 commits intomainfrom
refactor/export-streaming-v2

Conversation

@datlechin
Copy link
Copy Markdown
Collaborator

Summary

  • Rewrites the entire export/import pipeline from O(N^2) OFFSET/LIMIT pagination to O(N) AsyncThrowingStream-based streaming
  • Native streaming implemented for all 14 database drivers (MySQL, PostgreSQL, SQLite, ClickHouse, MSSQL, Redis, MongoDB, Oracle, DuckDB, Cassandra, Etcd, CloudflareD1, DynamoDB, BigQuery)
  • Fixes data correctness issues: JSON leading-zero corruption, XLSX silent truncation, CSV formula injection over-correction, MQL invalid JSON passthrough
  • Replaces custom progress with NSProgress, blocking NSSavePanel with async sheet, non-atomic writes with crash-safe atomic writes

Changes (73 files, +3030/-962)

Performance

  • All drivers stream rows one at a time instead of OFFSET/LIMIT batching
  • MySQL: mysql_use_result + mysql_fetch_row
  • PostgreSQL: PQsetSingleRowMode + PQgetResult
  • SQLite: sqlite3_step loop
  • ClickHouse: URLSession.bytes + JSONEachRow
  • MSSQL: dbnextrow loop
  • Redis: SCAN cursor streaming
  • MongoDB: mongoc_cursor_next loop
  • Oracle: OracleNIO AsyncSequence
  • DuckDB: duckdb_fetch_chunk iteration
  • Cassandra: automatic paging with cass_result_has_more_pages
  • DynamoDB: ExclusiveStartKey pagination
  • BigQuery: pageToken pagination
  • Etcd/D1: single-request streaming

Data Correctness

  • JSON: column-type-guided numeric inference, leading-zero strings preserved, BIGINT > Int.max emitted as raw integers
  • XLSX: auto-splits at 1,048,576 rows with warning, ZIP64 overflow guard, atomic writes
  • CSV: OWASP-only formula injection prefixes (= + - @)
  • MQL: JSONSerialization validation before raw object passthrough

Native macOS

  • NSProgress replaces custom PluginExportProgress/PluginImportProgress
  • presentAsSheet(for:) replaces blocking NSSavePanel.runModal()
  • Atomic file writes via temp file + FileManager.replaceItemAt
  • Async gzip compression via Process.terminationHandler

Cleanup

  • Deleted: ProgressUpdateCoalescer, ExportServiceState, ImportServiceState, ExportProgressProxy, SQLExportHelpers
  • Removed redundant NSLock on @mainactor class
  • PluginKit ABI version bumped (3 -> 4)

Closes #771

Test plan

  • Export large MySQL table (>100K rows) as CSV, verify constant-time batches
  • Export >1M rows as XLSX, verify auto sheet split + warning message
  • Export JSON with leading-zero values (ZIP codes), verify preserved as strings
  • Cancel export mid-stream, verify no partial file left at destination
  • Export with gzip compression, verify async completion
  • Multi-table export, verify progress reaches 100%
  • Test each database driver's export individually

@datlechin datlechin merged commit 0cca086 into main Apr 18, 2026
2 checks passed
@datlechin datlechin deleted the refactor/export-streaming-v2 branch April 18, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export becomes slower and slower

1 participant