Webhooks deliver at-least-once, not exactly-once: expect duplicates, retries, and out-of-order arrivals. A production system needs four things together — HMAC signing to authenticate payloads, exponential backoff with jitter so retries don't stampede, stable dedupe keys so duplicate deliveries collapse, and idempotent consumer handlers that stay correct no matter how many times an event arrives. This guide covers each in turn.
---
Why webhooks are harder than they look
At-least-once delivery is the practical default for outbound HTTP notifications. Your dispatcher may receive an acknowledgment too late, crash before recording it, or hit a partner whose load balancer closed an idle connection. Retries are not a bug—they are how you move probability of delivery toward acceptable levels. That single fact forces every serious design to treat duplicates as normal and to make side effects idempotent. If your documentation promises "exactly once," you are describing an end-to-end contract that spans your queue, HTTP client, the customer network, and their application—not a toggle on a checkbox.
Signing addresses a different problem: authenticity and integrity. Without cryptographic verification, any party who can guess or discover an HTTPS URL can forge events. Timestamps and replay windows narrow how long a captured request remains useful. Together, signing and idempotent processing are the backbone of trustworthy webhook programs.
---
Signing payloads: HMAC, headers, and replay resistance
The most common pattern is an HMAC-SHA256 signature over a canonical representation of the request. The exact bytes you sign matter more than the algorithm name. Document whether you sign the raw body, a concatenation of timestamp plus body, or a structured string that includes the webhook version. If two implementations serialize JSON differently, signatures diverge even when the logical payload matches. Many providers freeze the body as received and sign those octets, which avoids normalization debates at the cost of requiring byte-identical replay on the consumer side.
Publish which headers carry the signature, which carry a timestamp or nonce, and which key identifier you use when multiple secrets exist during rotation. Consumers should reject requests when the timestamp is too far from their clock skew window, because old signed requests are still valid cryptographically until they expire from policy. Use constant-time comparison for the signature to reduce timing side channels. Fail closed: if verification throws or returns false, respond with 401 and do not partially enqueue work.
Key rotation is an operational requirement, not a future nice-to-have. Maintain at least two active secrets briefly so subscribers can deploy new keys before old ones disappear. Automate alerts when verification failures spike after a rotation, and tie rotation cadence to your documented credential lifecycle policy. Never embed long-lived shared secrets in client-side code or public repositories.
Some teams add an optional asymmetric scheme for high-assurance integrations. The trade-off is more CPU and more complex key distribution. For most SaaS-to-SaaS webhooks, symmetric HMAC with short-lived timestamps and documented canonicalization is sufficient if consistently implemented.
---
Retries: what senders owe receivers
Retries should be triggered by transport-level failure or by HTTP responses that explicitly signal a transient condition. A 500 from the subscriber usually warrants a retry; a 400 caused by a malformed payload usually does not unless you have a bug on the sender side. Document your matrix: which status codes retry, which go to a dead-letter queue, and how many attempts you will make before giving up. Surface that matrix in your developer portal so integrators do not learn it from production traffic alone.
Idempotency belongs on the sender as well as the receiver. If your dispatcher retries a delivery, it should send the same logical event identifier each time so the consumer can collapse duplicates. Pair that identifier with monotonic delivery attempt metadata if you need observability. Avoid generating a new random identifier per HTTP attempt for the same business event—that pattern turns deduplication into guesswork.
When subscribers throttle you intentionally with 429, respect Retry-After when present. Ignoring Retry-After trains well-meaning clients to hammer endpoints and undermines the fairness policies you expect inbound clients to respect from you. If you are both a webhook sender and an API provider, keep your outbound behavior aligned with the fairness you ask for inbound.
---
Backoff, jitter, and scheduling
Naive fixed-interval retries synchronize storms. If fifty deliveries fail for the same reason, retrying every thirty seconds lines up the next failure into a single thundering herd. Exponential backoff spreads attempts across time: increase the delay multiplier per attempt, cap the maximum delay, and add jitter so independent processes do not re-align. A typical shape starts near seconds, grows through minutes, and stops at hours for non-critical channels, with exact values driven by your SLA and partner expectations.
Jitter should be random but bounded. Full jitter between zero and the backoff interval reduces overlap more aggressively than always sleeping the full interval. For customer-visible deadlines, publish maximum time-to-delivery so support teams can reason about delays during incidents.
Backoff interacts with signing and timestamps. A long-delayed retry still carries the original event time in the payload; the HTTP request time is not the same as business event time. Teach consumers to parse both so they do not confuse latency with stale data. If you re-sign on retry, document whether the signature covers original event time or delivery time.
---
Dedupe keys: what to identify
Choose a primary dedupe key that matches your business semantics. Good candidates include a provider-issued event ID stable across retries, a composite of resource type and monotonic version, or a content hash of the normalized payload for idempotent create operations. The key should fit in your database with a uniqueness constraint or in a cache with TTL aligned to your retry horizon plus safety margin.
Avoid using only coarse fields like customer ID plus day when multiple distinct events can share them. Avoid only HTTP headers that change between attempts unless those headers are defined to be stable per logical delivery. If you expose both event ID and delivery ID, clarify which one is for idempotency and which is for tracing individual HTTP tries.
TTL matters. If you store dedupe records forever, storage grows without bound. If TTL is shorter than your maximum retry window, duplicates can reappear after eviction. Set retention to the worst-case replay window you support, including manual replays from support tooling. The overlap with keys and grains in batch and streaming pipeline design is intentional: the same dedupe discipline applies whether events arrive over HTTP or through a message bus.
---
Consumer responsibilities
Verification first. Parse the signature and timestamp before doing expensive work or enqueueing to workers. That ordering prevents attackers from burning CPU or filling queues with unauthenticated payloads. Return non-2xx quickly when verification fails so senders do not mistake auth errors for transient outages.
Respond quickly with 2xx when you have durably accepted responsibility for the event. Durably means written to a database, append-only log, or queue that your process trusts—not only buffered in memory. If you acknowledge before persistence, a crash creates silent loss. If you need asynchronous processing, acknowledge after enqueue to a system you monitor for backlog and poison messages.
Implement idempotent handlers. Upsert by event ID, use compare-and-swap semantics, or store processed fingerprints. Return 200 for duplicate deliveries when the duplicate is detected and the state already matches the intended outcome; some teams prefer 409 for clarity—either way, document behavior so senders do not treat success as failure.
Propagate idempotency to downstream systems. A webhook handler that calls another API should forward an idempotency key appropriate to that API. Side effects such as charging cards or sending email need stronger guards than database writes because external systems have their own retry behavior. Align with your broader API versioning and governance policy when webhook-driven calls participate in versioned contracts.
Observability: log delivery IDs, event IDs, verification results, handler duration, and duplicate hits. Alert on verification failure spikes, queue age, and error rates per webhook route. Run load tests that include duplicate delivery bursts so autoscaling and connection pools survive integrator misconfigurations.
---
Ordering, duplicates, and out-of-order arrivals
Unless you guarantee ordering—which most webhook systems do not—consumers must tolerate events for the same aggregate arriving out of order. Version fields or updated-at timestamps with monotonic comparison reduce the chance that an older event overwrites newer state. When ordering cannot be inferred, escalate to reconciliation jobs that periodically compare remote truth to local mirrors.
---
Operational checklist
Sender: canonical signing documented; clock skew policy defined; retry matrix published; exponential backoff with jitter enabled; dedupe-friendly event IDs stable across attempts; Retry-After honored; dead-letter and replay tooling available; metrics for success, failure, and duplicate suppression.
Receiver: signature verification in constant time; fast rejection on bad auth; durable accept before 2xx; idempotent storage keyed by event identity; downstream idempotency; dashboards and alerts for webhook health; runbooks linked from Help Center and incident communication habits from Status Page practices.
---
Closing
Reliable webhooks are a collaboration between sender and receiver. Signing proves the payload came from you. Retries with backoff convert flaky networks into eventual success without synchronized stampedes. Dedupe keys turn at-least-once delivery into business-level exactly-once effects. Consumer responsibilities close the loop: verify early, persist before acknowledging, and make handlers safe under duplication and reordering. Nail those pieces, and webhook integrations stop being midnight surprises and become boring infrastructure—which is the point.
Continue exploring Services Overview, drill into Documentation, and reach Contact when you want to map these patterns to your delivery stack.