Workflows & Automation · Backend · Data & Caching· 8 min read
Webhook event delivery pipelines: HMAC verification, idempotency, and retries
Engineering a mission-critical webhook dispatch system. Delivering signed lifecycle events with exponential backoff, dead-letter queues, and idempotency guarantees in NestJS and Redis.
WebhooksEvent-DrivenRedisBullMQSecurity
In asynchronous document signing and enterprise workflows (such as `pratul03/e-sign`), external integrations rely on webhooks to learn when an envelope transitions from `CREATED` to `VIEWED`, `SIGNED`, or `EXPIRED`. External HTTP servers are notoriously unreliable: they drop connections, return 500 errors, or experience network partitions. A reliable webhook pipeline must guarantee delivery retry, deduplication, and cryptographic verification.
Signing payloads with HMAC-SHA256
To protect subscribers from spoofed payloads, every webhook dispatch includes an `X-Signature-SHA256` header computed from the raw JSON payload and the subscriber's shared secret key.
import { createHmac, timingSafeEqual } from 'crypto';
export function signPayload(payload: string, secret: string): string {
return createHmac('sha256', secret).update(payload).digest('hex');
}
export function verifySignature(payload: string, incomingSig: string, secret: string): boolean {
const expectedSig = signPayload(payload, secret);
if (incomingSig.length !== expectedSig.length) return false;
return timingSafeEqual(Buffer.from(incomingSig), Buffer.from(expectedSig));
}Retry queue with exponential backoff
Failed webhook attempts are pushed to a Redis-backed BullMQ queue with an exponential backoff schedule (attempting delivery after 10s, 1m, 15m, 1h, and 6h). If all 5 retry attempts fail, the event is moved to a dead-letter queue (DLQ) with an alert dispatched to platform operations.