Async is where integrations go to die

Synchronous APIs fail loudly. Webhooks fail quietly, and quietly is expensive.

Verify signatures properly

  • Take the raw body (bytes, not the parsed object) and the signature header.
  • Compute HMAC-SHA256 with your endpoint secret.
  • Compare with a constant-time function. Not ===.
  • Reject on mismatch, and log it; signature failures are a security signal, not noise.

Assume every delivery will be duplicated and reordered

  • Store the event id and make processing idempotent. Processing the same event twice must be safe.
  • Do not rely on delivery order. If event B depends on event A, reconcile from the API rather than assuming sequence.
  • Respond 2xx after durable persistence, not after full processing. Process async from the queue.

Retries are your friend, if you design for them

Providers retry because the internet is unreliable. Design for it:

  • Return 2xx fast, process async
  • Make processing idempotent (see above)
  • Expect exponential backoff from the provider; don't panic on repeated deliveries
  • Surface retry state to operators so a poison message doesn't silently queue up

The five webhook failures to rehearse in sandbox

  • Invalid signature: key rotation gone wrong
  • Duplicate delivery: your idempotency under test
  • Out-of-order delivery: reconciliation logic
  • Slow processing timeouts: the 2xx-before-processing pattern
  • Poison message: malformed payload that always 500s

Every one of these can be rehearsed deterministically in a sandbox with an error simulator. Clients who have seen all five handle production calmly. That is the entire point of rehearsing failure.