Phase 1.5: Stage A federated master - leader-leased idempotent billing, join-as-master v0, snapshots, sweep

[ehsan6sha]

Phase 1.5 — Stage A federated master: full stack on box #2

Implements the Stage A milestone of the federated-master roadmap: a second full master (operator-run) with double-run-proof billing, the capstone installer v0, signed pinset snapshots, a replication sweep, and a fenced failover/resync runbook. Everything additive + flag-gated (default OFF — single-master behavior byte-identical when dark).

What's in

• FM-2 billing safety (flags BILLING_IDEMPOTENCY, CRON_LEADER_LEASE):
  • migration 018: partial UNIQUE (user_id, reference_id) WHERE tx_type='hourly_deduction' (CONCURRENTLY, dup pre-check, .down.sql);
  • deduction uses a deterministic hour:YYYY-MM-DDTHH key and the history INSERT is the dedup gate before any balance change;
  • blockScanner records + credits in one transaction (creditUserTx); a crash can no longer strand a recorded-but-uncredited deposit;
  • cron family gated by a Postgres advisory-lock leader lease (holder crash ⇒ standby takes over on its next tick; a partitioned ex-leader can't bill at all — the shared DB is the arbiter).
• update-scripts/join-as-master.sh v0 (capstone installer first cut): detect-installed / adopt-or-halt (adopts the Phase-1 kubo+cluster writer; halts on a foreign postgres), ordered migrations with halt-on-error, dockerized stack (healthchecks + restart policies + label-scoped watchtower), gateway profile auto-enabled when the fula-gateway image exists (built via feat(docker): reproducible fula-gateway image for federated masters fula-api#30), safeguard crons installed + first snapshot taken immediately, params persisted to .env, idempotent re-runs.
• pinset-snapshot.sh — signed (ed25519) authoritative pinset dumps, --verify/--restore/--install-cron (early FM-3 restore path).
• replication-sweep.sh — below-REPL_MIN detection + recover + alert log + --strict (closes the S4 "no automated sweep" gap).
• migration 019 — real fresh-install bug found by this e2e: post-PII linkWallet stores wallet_address=NULL but fresh schemas keep NOT NULL (012 missed it) ⇒ every fresh deployment rejected wallet links. Guarded .down.sql.
• Failover/resync runbook (fenced flips; no-verification-no-flip).
• Tests: 30 vitest unit tests + 2 live-Postgres integration tests + the e2e drill suite (tests/e2e/phase-1.5/).

E2E evidence (clean Ubuntu 24.04 box, real daemons, real Postgres)

Drill suite 60-master-drills.sh — final run all green (RESULT pass=18 fail=0):

• D1 stack healthy (postgres/API/webui; gateway profile live on :9000 with durable pin queue active)
• D2 migration 018 present
• D3 two webui masters → exactly one leader + one standby → exactly ONE hourly_deduction row per (user, hour)
• D4 kill -9 the leader → standby acquires the lease → STILL exactly one row (idempotency under failover)
• D5 live-PG integration: concurrent same-hour deductions deduct once; replayed deposit credits once, atomically (2/2)
• D6 snapshot taken + signature verifies + tampered file rejected + unpin→--restore re-pins
• D7 sweep clean → forced under-replication (3 of 4 peers down) detected + alerted → reconverged clean

Mixed-fleet/no-forced-upgrade invariant unaffected: all changes are master-side and dark by default; providers and existing data untouched (Phase 1 drills covered the provider side).

Cross-repo

• feat(docker): reproducible fula-gateway image for federated masters fula-api#30 — reproducible fula-gateway image (built + serving S3 on the box; state volume fix included)
• functionland/join-server#4 — GET /masters federated master-list v0 (schema mirrors the future Base MasterRegistry)

Scoping note

The full FxFiles-flow upload/download fidelity suite runs with Phase 2 (which changes the upload path; this phase doesn't touch it). Stage A failover is operator-fenced per the runbook until FM-1 (bucket-root CAS, Phase 2.5) enables auto-failover.

Closes #46

🤖 Generated with Claude Code