Phase 2: fula-ingest - verified byte ingress (client-side CID, quota-gated)

[ehsan6sha]

Phase 2 — fula-ingest: verified byte ingress + e2e suites

The decentralized-upload byte path: any approved operator can run a fula-ingest node that accepts encrypted chunk BYTES so uploads stop depending on a single gateway, with tamper-evidence enforced server-side.

What's in

• docker/fula-ingest/ (Go): PUT /v0/block?cid=<declared> verifies the blake3 raw-leaf CID over the body before anything touches the blockstore (one flipped bit ⇒ 422, nothing stored), then kubo block/put?cid-codec=raw&mhtype=blake3 with a cross-check of kubo's key. S1 quota gate mirrors the gateway exactly (GET {STORAGE_API_URL}/api/v1/storage with the client's Bearer; INGEST_QUOTA_MODE=open = gateway-parity fail-open, strict denies when unverifiable) with a 30s per-token cache (one quota call per window, not per chunk). Size cap, bounded concurrency, /health, graceful shutdown, Dockerfile.
• e2e suites (tests/e2e/phase-2/): 40-ingest-drills.sh (I1–I10) + 60-fidelity.sh (F1–F4).

Evidence (clean Ubuntu 24.04 test master, real daemons)

• Go unit: 10/10 (containerized) — incl. tampered-never-stored and quota-denied-never-stored.
• Live drills: 20/20 — valid block stored; tampered ⇒ 422 + absent from kubo; suspended user ⇒ 402 + absent; no token ⇒ 401; bytes still ingested with the gateway container STOPPED; gateway rebuilt from the fula-api Phase-2 branch flips /fula/capabilities false→true; live empty-body remote-cid mapping PUT ⇒ 200 with ETag=<declared cid> and a byte-exact GET round-trip; absent cid ⇒ bounded 409 (client-fallback contract).

Cross-repo: functionland/fula-api#31 (client routing + gateway mapping PUT; functionland/fula-api#30 lands first — the gateway image recipe).

In simple words

When your phone uploads a file today, the encrypted pieces must pass through one company server. With this, the pieces can go to any approved relay computer instead — and every piece carries a fingerprint, so a relay that corrupts even one byte gets caught and rejected instantly, and nobody can upload without an account in good standing.

Closes #74

🤖 Generated with Claude Code

[ehsan6sha]

Fidelity suite evidence (test master, real daemons)

Leg	Result
F1 — live client→ingest round-trip (1.5 MB, CID on)	✅ + server-side proof: kubo block count grew 36→75 (bytes flowed via the ingest node)
F2 — client-CID-off legacy parity	✅
F3 — FxFiles-faithful offline_e2e single-object flow	✅
F3 — FxFiles-faithful offline_e2e chunked flow (>768 KiB, the photo/video path)	✅
F4 — 1 GiB via ingest (~4096 chunks)	🔄 rerunning with a 12 h token — the first run uploaded 4019/4096 chunks over ~2 h on the contended 4-core box, then the 2 h fixed-expiry TEST token died (real clients refresh). Result will be posted here.

Scale measurement worth recording: chunked-upload throughput on this path is metadata-commit-bound (each chunk's mapping/bytes PUT serializes through the per-bucket write lock + an index-tree flush; observed ~3× object amplification per chunk). This is a pre-existing property of the gateway's chunked path — Phase 2 neither worsens nor improves it (byte transport moves off the master; commit costs are identical). The plan's Phase-7 client-computed recursive root / batched commits is the designed fix.

🤖 Generated with Claude Code