Expire circuit breaker state

CHANGES.md

@@ -8,6 +8,19 @@ Version 2.3.2
 
 To be released.
 
+### @fedify/fedify
+
+ -  Fixed the outbound delivery circuit breaker retaining per-host state in the
+    configured key–value store forever when a remote host never recovered.
+    Circuit breaker state now receives a TTL on writes made with the default
+    failure policy, custom failure policies can opt in with the new `stateTtl`
+    option, and stale state written by earlier 2.3 releases is cleared
+    automatically on CAS-backed stores after upgrade, with another sweep after a
+    grace window to cover rolling deployments.  [[#916], [#917]]
+
+[#916]: https://github.com/fedify-dev/fedify/issues/916
+[#917]: https://github.com/fedify-dev/fedify/pull/917
+
 
 Version 2.3.1
 -------------

docs/manual/circuit-breaker.md

@@ -154,6 +154,36 @@ For multi-worker deployments, use a `KvStore` implementation that supports
 Fedify still works without CAS, but it logs a warning because concurrent
 workers can race when opening or closing the same host's circuit.
 
+Fedify expires stored circuit state after it is no longer useful.  With the
+default numeric failure policy, the default `stateTtl` is derived from
+`failureWindow`, `recoveryDelay`, and `heldActivityTtl` so failure history,
+recovery probes, and held activities all have enough time to complete.  If you
+provide a custom `failure` callback, Fedify cannot infer how long your policy
+needs its timestamp history, so stored state does not expire by default.  Set
+`stateTtl` explicitly when using a custom policy and you want circuit state to
+be cleaned up automatically:
+
+~~~~ typescript
+const federation = createFederation<void>({
+  kv,
+  queue,
+  circuitBreaker: {
+    failure(timestamps) {
+      return timestamps.length >= 10;
+    },
+    stateTtl: { days: 14 },
+  },
+});
+~~~~
+
+When upgrading from Fedify 2.3.0 or 2.3.1, Fedify automatically rewrites
+legacy circuit state with a TTL only on `KvStore` implementations that support
+`cas()`.  Stores without CAS, including `@fedify/postgres` and
+`@fedify/redis`, apply TTLs to new circuit state but cannot automatically
+clean up circuit state that was already written without a TTL.  If that old
+state matters for your deployment, remove it with a one-off cleanup script or
+add a TTL directly in your storage backend.
+
 
 Observability
 -------------

packages/fedify/src/federation/circuit-breaker-test-utils.ts

@@ -0,0 +1,12 @@
+import type { KvStore } from "./kv.ts";
+
+export function markCircuitBreakerLegacySweepDone(
+  kv: Pick<KvStore, "set">,
+): Promise<void> {
+  return kv.set([
+    "_fedify",
+    "circuit",
+    "__fedify_meta",
+    "circuit_breaker_state_ttl_sweep_v1",
+  ], { state: "final" });
+}