Skip to content

feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071)#391

Merged
nijeesh-stream merged 13 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads
May 13, 2026
Merged

feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071)#391
nijeesh-stream merged 13 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads

Conversation

@nijeesh-stream

@nijeesh-stream nijeesh-stream commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds first-class support for gzip-compressed webhook payloads (HTTP webhooks, SQS, SNS) and exposes a stable VerifyAndParse* API that mirrors the cross-SDK contract published in Webhooks Overview.

New public API

Primitives (in webhook.go, intended to be composable):

  • GunzipPayload(body) ([]byte, error) — gzip-magic-byte detection, no-op when not compressed
  • DecodeSqsPayload(body) ([]byte, error) — base64 decode then gunzip-if-magic
  • DecodeSnsPayload(notificationBody) ([]byte, error) — JSON-parse the SNS HTTP notification envelope, extract the inner Message, then run the SQS pipeline. Falls through to a pre-extracted Message string when the input is not a JSON envelope
  • VerifySignature(body, signature, secret) error — HMAC-SHA256 over the uncompressed body, with a constant-time comparison (matters for the HTTP webhook path where the X-Signature header is exposed publicly; SQS / SNS deliveries arrive over AWS-internal transports where timing-attack resistance is not strictly required). Returns a non-nil error wrapping stream.ErrInvalidWebhook on mismatch — see Unified error handling below.
  • ParseEvent(payload) (*Event, error) — JSON → typed *Event

Composites (return a typed *Event):

  • VerifyAndParseWebhook(body, signature, secret) (*Event, error)
  • VerifyAndParseSqs(body, signature, secret) (*Event, error)
  • VerifyAndParseSns(body, signature, secret) (*Event, error)

Client-bound versions on *Client use the configured app secret automatically.

Backwards compatibility

Client.VerifyWebhook is preserved and delegates to VerifySignature. Existing callers continue to work unchanged for plain (uncompressed) bodies.

Unified error handling

Per cross-SDK coordination (mogita's review across the 6 sibling SDK PRs), every webhook failure path now terminates at a single sentinel error — stream.ErrInvalidWebhook so customers only need one error-handling arm. The message text identifies which failure mode fired so callers that want to differentiate (security logging, retry policy) can filter on substring:

Failure mode Message
Signature mismatch signature mismatch
Base64 decode invalid base64 encoding
Gzip decompression gzip decompression failed
JSON parse invalid JSON payload

Use errors.Is(err, stream.ErrInvalidWebhook) for the unified check; the inner cause (when present) is preserved via %w wrapping. VerifySignature now returns error (was bool) so it surfaces the mode-specific message; the legacy Client.VerifyWebhook (bool return, constant-time hmac.Equal) is untouched.

Tests

webhook_test.go covers plain / gzip / base64 / base64+gzip payloads, signature mismatches, malformed bytes, JSON parsing into *Event, and a backwards-compat path for Client.VerifyWebhook. Linked Linear ticket: CHA-3071.

Golden test fixtures (Tommaso)

Added shared reference fixtures to the test suite so future SDKs can sanity-check decoders against the same payloads:

aGVsbG93b3JsZA==                          -> helloworld   (base64)
H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64 + gzip)

Test plan

  • go vet ./... clean
  • go test -c . (test binary compiles clean)
  • Full go test ./... runs in CI (package init() requires STREAM_KEY / STREAM_SECRET)

…HA-3071)

Co-authored-by: Cursor <cursoragent@cursor.com>
Replaces VerifyAndDecodeWebhook / DecompressWebhookBody with the
cross-SDK contract documented at
https://getstream.io/chat/docs/node/webhooks_overview/.

Package-level helpers in stream_chat:

  Primitives:
    UngzipPayload      - gzip magic-byte detection + inflate
    DecodeSqsPayload   - base64 then ungzip-if-magic
    DecodeSnsPayload   - alias for DecodeSqsPayload
    VerifySignature    - constant-time HMAC-SHA256 comparison
    ParseEvent         - JSON -> *Event (typed, with ExtraData fallback
                         for unknown event types)

  Composite (return *Event):
    VerifyAndParseWebhook
    VerifyAndParseSqs
    VerifyAndParseSns

The composite functions auto-detect compression from body bytes, so
the same handler stays correct whether or not Stream is currently
compressing payloads, and behind middleware that auto-decompresses.

Client instance methods (Client.VerifyAndParse*) mirror the three
composite helpers and pull the API secret from the receiver.

The legacy Client.VerifyWebhook(body, signature) bool helper is
unchanged for backward compatibility.

ErrInvalidWebhookSignature is preserved as the sentinel error for
HMAC mismatch.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nijeesh-stream nijeesh-stream changed the title feat(webhooks): add VerifyAndDecodeWebhook for compressed payloads feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071) May 8, 2026
nijeesh-stream and others added 3 commits May 8, 2026 16:53
RFC 1952 defines the gzip magic number as the two-byte sequence
1F 8B; the third byte (CM) is informational and not part of the
identifier. Trim the magic check from three bytes to two to match
the spec and stay consistent with the reference implementations
in the public docs.

Co-authored-by: Cursor <cursoragent@cursor.com>
The compression section referenced VerifyAndDecodeWebhook,
DecompressWebhookBody and a payloadEncoding argument that do not exist
in this SDK. Rewrite to document the actual public surface:
VerifyAndParseWebhook / VerifyAndParseSqs / VerifyAndParseSns plus the
lower-level UngzipPayload, VerifySignature and ParseEvent helpers.

CHA-3071

Co-authored-by: Cursor <cursoragent@cursor.com>
io.ReadAll followed by a deferred Close swallowed the trailing CRC and
length-trailer error that compress/gzip only reports when the reader
is closed. Replace the defer with explicit Close handling so a corrupt
or truncated gzip stream is reported as a gzip error instead of
falling through to a misleading invalid-signature failure.

CHA-3071

Co-authored-by: Cursor <cursoragent@cursor.com>

@mogita mogita left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK review pass for CHA-3071. Two inline comments — see below.

Comment thread webhook.go Outdated
Comment thread webhook_test.go Outdated
…y verify

decode_sns_payload now JSON-parses the SNS HTTP notification envelope
({"Type":"Notification","Message":"..."}) and extracts the inner
Message field before running the SQS pipeline. Falls through to the
pre-extracted Message string when the input is not a JSON envelope so
existing call sites keep working.

Legacy Client.VerifyWebhook now uses hmac.Equal for constant-time
comparison instead of bytes.Equal, closing the same timing leak we
already closed in WebhookHelpers.VerifySignature.

Test fixture adds a realistic SNS HTTP notification body that all
SDKs in this rollout can share.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mogita

mogita commented May 11, 2026

Copy link
Copy Markdown
Contributor

Cross-SDK coordination: unifying webhook exception types

After the review pass across all 6 SDKs in this rollout and team discussion, we're consolidating the new webhook exception strategy to a single unified error type rather than the split (signature vs parse sentinels) being introduced in this PR.

The Webhook Handling Spec on Notion (CHA-2961) has been revised to reflect this — §5.2 / §5.3 / §7 now specify a single class.

Why unified: From a customer's perspective, all failure modes — signature mismatch, gzip decompression failure, base64 decode failure, SNS envelope failure, JSON parse failure, missing schema field — terminate at the same error-handling branch in customer code. A signature/parse split adds structural complexity without changing customer behavior. Customers who want to filter security logs for signature mismatches specifically can do so via error message text or errors.Unwrap cause-chain.

Sentinel name: ErrInvalidWebhook (+ struct type InvalidWebhookError) — "Invalid" covers all failure modes accurately and is consistent with Go stdlib naming (errors.ErrUnsupported, etc., where the prefix Err names the category).

Per-SDK naming across the rollout:

SDK Class name
JS InvalidWebhookError (extends Error)
Python InvalidWebhookError
Go sentinel ErrInvalidWebhook + struct InvalidWebhookError
Java InvalidWebhookException (extends existing StreamException)
PHP InvalidWebhookException (extends existing StreamException)
Ruby StreamChat::InvalidWebhookError (extends StandardError)
.NET StreamInvalidWebhookException (extends StreamBaseException)

Asks for this PR:

  1. Consolidate the existing sentinels (ErrInvalidSignature, ErrMalformedWebhook, etc.) into a single ErrInvalidWebhook sentinel, with concrete struct InvalidWebhookError carrying details
  2. Wrap all failure paths to return that single sentinel — signature mismatch, gzip failure, base64 failure, SNS envelope failure, JSON parse failure, missing type/schema failure — using fmt.Errorf("%w: <mode>: %v", ErrInvalidWebhook, cause) so errors.Is(err, ErrInvalidWebhook) matches and errors.Unwrap reaches the cause
  3. The wrapped message identifies which failure mode fired ("signature mismatch", "invalid base64", "missing type field") so customers can filter on substring
  4. Legacy Client.VerifyWebhook (returning bool) stays unchanged — back-compat preserved. Note: there's an existing review comment on AppClient.cs-equivalent in another SDK about the legacy method's bytes.Equal not being timing-safe; same applies to client.go:160, but that's separate from this exception-unification ask.
  5. Update unit tests to assert via errors.Is(err, stream.ErrInvalidWebhook); for mode-specific tests, also assert on error message substrings

This same comment is being posted on all 6 SDK PRs (JS / Go / Ruby / PHP / Java / .NET) for coordination. Happy to discuss naming or scope tradeoffs.

nijeesh-stream and others added 2 commits May 11, 2026 15:33
…n fixtures (CHA-3071)

Per Tommaso's suggestion, align the gzip helper with the GNU `gunzip`
command name. The function was added in this PR and not yet released,
so this is a straight rename with no back-compat alias.

Adds Tommaso's reference fixtures to the test suite as named cases so
future SDKs can sanity-check against the same payloads:

  aGVsbG93b3JsZA==                          -> helloworld   (base64)
  H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64+gzip)

Co-authored-by: Cursor <cursoragent@cursor.com>
…-3071)

Per cross-SDK coordination (mogita's review across the 6 sibling SDK
PRs), every webhook failure path now wraps a single sentinel error.
Callers can do errors.Is(err, stream.ErrInvalidWebhook) for a unified
check, or strings.Contains on the message for mode-specific filtering.

Replaces the previously-unreleased ErrSignatureMismatch with
ErrInvalidWebhook and threads it through every primitive:

  VerifySignature     -> 'signature mismatch: ...'
  GunzipPayload       -> 'gzip decompression failed: ...'
  DecodeSqsPayload    -> 'invalid base64 encoding: ...'
  ParseEvent          -> 'invalid JSON payload: ...'

The legacy Client.VerifyWebhook (returning bool) is unchanged - still
uses hmac.Equal for constant-time comparison.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nijeesh-stream

Copy link
Copy Markdown
Contributor Author

Shipped in c32c92a along with the SNS envelope fix.

  • Sentinel-only variant per the maintainer's call (no separate struct type for errors.As)
  • ErrInvalidWebhookSignatureErrInvalidWebhook
  • Every primitive wraps with fmt.Errorf("<mode>: %v: %w", err, ErrInvalidWebhook) so errors.Is(err, stream.ErrInvalidWebhook) is the single check; modes are signature mismatch, invalid base64 encoding, gzip decompression failed, invalid JSON payload
  • VerifySignature now returns error (was bool) so it surfaces the mode token; legacy Client.VerifyWebhook (bool return, constant-time hmac.Equal) is untouched
  • webhook_test.go: three failure-mode subtests now assert both errors.Is(err, ErrInvalidWebhook) AND the substring; go build ./..., go vet ./..., gofmt -l webhook.go webhook_test.go all clean. Live-API integration tests run in CI (local env panics on missing STREAM_KEY / STREAM_SECRET, pre-existing)

Docs side, docs-content#1276 now carries the per-SDK error-class table on the Webhooks overview page.

nijeesh-stream and others added 4 commits May 12, 2026 14:41
…-3071)

Stream does not ship an X-Signature on SQS or SNS deliveries - those
transports ride AWS-internal infrastructure (IAM-authenticated queues
and AWS-signed SNS notifications), so HMAC verification on top is
theatre. The package helpers now treat empty-string signature + secret
as a skip-verification signal, and the instance methods take signature
as a variadic argument.

- VerifyAndParseSqs(body, '', '')         -> decode + parse
- VerifyAndParseSqs(body, sig, secret)    -> decode + verify + parse
- client.VerifyAndParseSqs(body)          -> decode + parse
- client.VerifyAndParseSqs(body, sig)     -> + verify with client secret

Passing only one of (signature, secret) returns an error wrapping
ErrInvalidWebhook. The HTTP-webhook path is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ndParseWebhook; docs + tests

Co-authored-by: Cursor <cursoragent@cursor.com>
…bhookSignatureError + parse_*; Python WebhookSignatureError; guard test init without STREAM_*
@nijeesh-stream nijeesh-stream enabled auto-merge (squash) May 13, 2026 10:04
@nijeesh-stream nijeesh-stream merged commit 35a8e9a into master May 13, 2026
7 of 8 checks passed
@nijeesh-stream nijeesh-stream deleted the nijeeshjoshy/cha-3071-compress-webhook-payloads branch May 13, 2026 10:29
@github-actions github-actions Bot mentioned this pull request May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants