feat: add database export, import, and copy for backup and migration#37
Merged
Conversation
Introduce the codex-migrate crate: a portable, engine-agnostic core for 1:1 data transfer between any two SeaORM backends, the foundation for the export/import/copy subcommands and SQLite-to-PostgreSQL migration. Rows move as typed Models so SeaORM handles engine-specific representation differences (UUID blob vs native uuid, text JSON vs JSONB, 0/1 vs bool) by construction, avoiding the corruption risks of a raw SQL dump. - generic per-table dump/load/copy/count/truncate over any entity - an x-macro registry listing every table once, with a drift-guard test - transaction-scoped foreign-key disabling during bulk load - per-table row-count verification - transfer(): disable FK, truncate, load, and commit in one destination transaction (SQLite re-validates foreign keys at commit) Loads truncate first because migrations seed data, so a faithful mirror must overwrite the seeded rows. Adds SQLite round-trip, JSON-fidelity, overwrite/idempotency, and registry-drift tests.
Build the tar.gz export bundle on top of the transfer engine: the database as one NDJSON file per table plus a manifest, alongside the on-disk artifacts the database only references by path (thumbnails, covers, plugin data). - manifest records format and schema version, per-table row counts, and each bundled artifact group's source base directory - export dumps the database to a staging dir and streams artifacts straight from their source dirs into the archive (no intermediate copy) - import extracts, loads the database (disable FK, truncate, load, commit), unpacks artifacts to the target dirs, and re-roots stored file paths - path re-rooting rewrites thumbnail and cover path prefixes from the source base dir to the target's, with path-boundary-safe matching, so images resolve on an instance whose files.*_dir differ The schema-version and fresh-target safety checks consume the extracted manifest and are left to the CLI layer. Adds an archive round-trip test (import into differing artifact dirs) and path re-rooting unit tests.
Expose the migration engine as three subcommands on the codex binary, completing the SQLite-to-PostgreSQL migration and backup workflow. - export: write the database and on-disk artifacts (thumbnails, uploads, plugin data; PDF cache opt-in) to a portable .tar.gz, sourced from the instance config - import: run migrations on the target, refuse a schema-version mismatch or a target that already holds user data (unless --replace), then load, unpack artifacts, re-root file paths, and print a summary plus an encryption-key reminder - copy: transfer rows directly between two databases; each side resolves from a URL flag, a *_DATABASE_URL env var, a --*-config file, or the local config, requiring at least one explicit side Adds a database-URL parser (sqlite/postgres), a fresh-target guard over user-owned tables, and a manifest peek that reads only the manifest for pre-flight checks. Covered by command round-trip, guard, and resolver tests.
…guide Verify the migration path end to end and document how to use it. - add a gated SQLite-to-PostgreSQL round-trip test that seeds a connected fixture and asserts row-count parity plus JSON, float, bool, and UUID foreign-key fidelity across engines; skips when no test database is reachable - re-export the migrate crate as codex::migrate so integration tests can reach it - document export, import, and copy with a SQLite-to-Kubernetes-Postgres runbook, covering the encryption-key carry-over and the database-owner requirement for the bulk load
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the ability to move a Codex instance's entire dataset between databases and to back it up to a portable archive. This enables migrating an existing SQLite deployment to a PostgreSQL-backed, worker-separated setup with a faithful 1:1 copy of all data, restoring from a snapshot, and pulling one instance's data into another.
Motivation
Codex previously had no way to move data between database engines or to take a portable backup. The existing backup tooling was PostgreSQL-only, and a raw SQL dump cannot cross engines safely because SQLite and PostgreSQL store UUIDs, JSON, and booleans differently, risking silent corruption. Operators who started on SQLite had no supported path to a scalable PostgreSQL deployment without losing metadata, ratings, reading progress, uploaded covers, and plugin state.
Changes
codex exportcommand that writes the database and its on-disk artifacts (thumbnails, uploaded covers, plugin data) to a single portable.tar.gz, with flags to include the PDF render cache or to exclude individual artifact groups.codex importcommand that loads an archive into an instance, translating data correctly for the target engine and rewriting stored file paths to the target's configured directories. It refuses to run against a database that already holds user data unless--replaceis given, or when the archive doesn't match the instance's schema.codex copycommand that transfers rows directly between two databases; each side is selected via--from/--toURLs,CODEX_SOURCE_DATABASE_URL/CODEX_TARGET_DATABASE_URLenvironment variables,--from-config/--to-configfiles, or the local config for an omitted side.copytransfers database rows only and leaves files to be synced separately.Notes