fix(router): bound + detach old graph-server shutdown on config swap (#3286)

[arutkowski00]

Problem

SwapGraphServer shuts the previous graph server down synchronously on the config-poller goroutine with an unbounded ctx. graphServer.Shutdown drains in-flight requests in wait() (polls inFlightRequests, no independent timeout), so one stuck request blocks the entire config pipeline.

Stuck-request sources observed / possible:

• handler blocked on a dead-client write (WriteTimeout: 0, kept for SSE)
• a custom module's own HTTP call without a full request timeout
• (subgraph fetches are bounded by request_timeout 60s, so those aren't the long pole)

Production impact (this incident)

• #10 stall instrumentation: 100% of shutdown stalls are in-flight request drain, up to 1h21m.
• Froze CDN config hot-reload → pods served stale schema for hours (ticket #3286).
• The old generation's schema AST (~40% of heap), ristretto caches, protojson strings were never released → GC mark-phase storm (scanobject/findObject ~60% CPU) → planning-latency spikes when CPU headroom was reduced.

Fix

Behind mondaytweaks.AsyncBoundedOldGraphServerShutdown (default on):

• Run the old server's Shutdown off the poller goroutine.
• Bound the drain by the configured grace_period (context.WithoutCancel(ctx) to keep trace values but detach lifetime, then WithTimeout). Fallback 90s (> 60s subgraph request_timeout) if grace_period is unset, so the drain is never unbounded.

New traffic already routes to the swapped-in server, so abandoning a stuck old-server request after the drain window is safe — reloads proceed and the old generation is released.

Also re-enables DisableUpstreamSubscriptionPingWhenClientWebSocketDisabled: prod runs websocket.enabled: false, yet a goroutine profile showed WSTransport.pingLoop at ~65% of all goroutines (1.5M) accumulating across reloads. Zeroing PingInterval when client WS is disabled stops that leak.

Test

• go build ./core/... ./pkg/mondaytweaks/... ✅
• go vet ./core/ ./pkg/mondaytweaks/ ✅
• go test ./core/ -run 'Swap|GraphServer|Shutdown|Server' ✅

Follow-ups (separate)

• Explicitly set traffic_shaping.all.request_timeout: 60s in the router config (defends against env override).
• Upstream: consider force-cancelling in-flight request contexts at the drain deadline, and a dedicated shutdown-drain-timeout config knob.

[github-actions]

Router-nonroot image scan passed

✅ No security vulnerabilities found in image:

ghcr.io/mondaycom/cosmo/router:sha-74aa17e9007f857ccce048f46ae9ea767d5749bf-nonroot

[arutkowski00]

Effective drain bound in prod (helm values)

Confirmed via helm-charts (applications/business/platform-api-cosmo-router/defaults.yaml, shared across all envs — no per-env overrides):

Var	Value
GRACE_PERIOD	65s (overridden from 30s default)
SHUTDOWN_DELAY	75s
terminationGracePeriodSeconds (k8s pod)	90
REQUEST_TIMEOUT	not set → cosmo default 60s

So this change bounds the async old-server drain at 65s, which is intentionally > the 60s subgraph request_timeout — a well-behaved in-flight request completes before the old server is abandoned. The 90s hardcoded fallback only applies if grace_period is unset (it isn't in prod). terminationGracePeriodSeconds: 90 is the k8s pod SIGTERM→SIGKILL path, separate from the in-process config-swap drain this PR fixes.

Net: no config change required — the fix inherits the already-tuned GRACE_PERIOD=65s.