Webhook Retries, Duplicates & Idempotency Guide

Intent and ownership

This guide owns webhook delivery semantics, not every duplicate or integration problem

This guide owns the informational task webhook retries, duplicate events, and idempotency. It explains how a receiving system can accept a repeated or delayed event without producing repeated business effects, and how an operator can recover after an uncertain response, outage, manual replay, or out-of-order delivery.

Use How to Prevent Duplicate CRM Contacts when several forms, imports, apps, APIs, or people create competing CRM identities. Use the Website Form to CRM Automation Checklist when one lead-capture path needs pre-launch acceptance from form through owner follow-up. Use the API and Webhook Audit when a live payload, signature, endpoint, response, timeout, mapping, or replay already needs paid diagnosis. Use Zapier Webhook Troubleshooting when Zapier is specifically the failing runtime, and use the Integration QA Checklist for broader launch QA across a connected workflow.

Retry schedules, success codes, signatures, identifiers, event ordering, retention, replay controls, and limits differ by provider and can change. The architecture below is a platform-neutral control model. The selected provider's current official documentation and a controlled test remain the authority for one live endpoint.

One source event

A provider-side occurrence has a stable identity or a defensible composite key inside the correct account and endpoint scope.

One accepted claim

Concurrent deliveries race against one atomic uniqueness rule instead of a separate read-then-write check.

One intended effect

Email, CRM writes, orders, access changes, tasks, and analytics each have repeat-safe state and evidence.

Architecture map

Use an eight-stage path from raw request to reconciled effect

The featured workflow visual and this table describe the same reliability path. The table is the crawlable and accessible text equivalent. The important boundary is between accepting a delivery and completing its business work: verify and store first, acknowledge promptly, then process and reconcile outside the request.

Stage	Decision	Evidence	Hold condition
1. Receive raw request	Did the expected HTTPS endpoint receive a permitted method, size, and content type?	Endpoint, received time, safe headers, byte length, request hash, and correlation ID	The body was transformed before required verification or exceeds the accepted contract
2. Verify source	Does the provider signature, timestamp, secret version, and endpoint context validate?	Verification result and reason without exposing the secret or full private payload	Signature, recency, endpoint, or account scope is invalid or unknown
3. Identify event	Which provider event and delivery does this request represent?	Provider, tenant, endpoint, event ID, delivery ID, type, object ID, and occurred time	The identifier is absent, unstable, or scoped too broadly
4. Claim atomically	Is this the first accepted claim or a recognized duplicate?	Unique constraint result, first-received record, payload hash, and duplicate count	Check-then-insert logic allows simultaneous workers to claim the same event
5. Persist and acknowledge	Is the verified event durably stored or queued before the response deadline?	Ledger row or queue receipt, accepted state, response code, and response time	A success response can be sent before any durable acceptance exists
6. Apply effects safely	Can each CRM, payment, access, message, task, or analytics effect repeat safely?	Effect key, prior state, requested state, provider response, result ID, and status	A retry can append, charge, email, grant, increment, or create again
7. Order and recover	Is the event current, stale, replayed, blocked, retryable, or ready for manual review?	Resource version or event time, transition decision, attempts, next action, and owner	Arrival order is treated as truth or a partial effect has no recovery rule
8. Reconcile and monitor	Can an operator prove source events, accepted events, completed effects, failures, and drift?	Counts, age, duplicate rate, latency, dead-letter queue, reconciliation result, and runbook	HTTP success is the only operational signal

Reliability vocabulary

Separate the event from its deliveries and business effects

Many duplicate bugs begin because one identifier is expected to represent every layer. A provider event records something that happened. A delivery is one attempt to send that event to one destination. A receiver attempt is one execution of your handler or worker. A business effect is the change you intend downstream. A replay is a later request to deliver an existing event again. Those records can have different identifiers and different retention periods.

Layer	Example identity	May repeat?	Safe control
Source event	Provider event ID inside one account or tenant	The same event can be delivered repeatedly	Unique event claim in the correct scope
Delivery	Destination-specific notification or delivery ID	One event can have several destinations or attempts	Keep delivery evidence without using it as the only business key
Receiver attempt	Trace, request, queue-message, or worker-attempt ID	Infrastructure can retry a message or an entire batch	Attempt log attached to the stable event and effect
Business effect	Event key plus effect name and version	Different effects from one event are legitimate	Unique key and state machine per effect
Replay	Original event with new or retained delivery identity	Manual recovery intentionally repeats delivery	Run through the same idempotent path after fixing the cause
Semantic duplicate	Separate event IDs that represent the same intended outcome	Some providers can generate separate events for one object action	Use object, event type, domain state, and effect key where documented

HTTP idempotency also has a narrower standards meaning. RFC 9110 defines an idempotent request method by whether several identical requests have the same intended server effect as one request. Webhooks normally arrive as POST, which is not automatically idempotent. The receiving application must supply repeat-safe semantics through event identity, constraints, state checks, and idempotent downstream operations.

Current provider behavior

Do not copy one provider's retry rules into another integration

The table summarizes official documentation reviewed July 29, 2026. It is a design comparison, not a substitute for the current page and selected account. In particular, the accepted response, timeout, retry duration, redelivery controls, identifier semantics, and event retention are provider-specific.

Provider	Documented delivery behavior	Identity and order guidance	Implementation consequence
Stripe	Live destinations are retried for up to three days with exponential backoff; the endpoint should return `2xx` quickly.	Event order is not guaranteed. Log processed event IDs; some semantic duplicates can be separate Event objects and need object ID plus event type.	Do not depend on arrival order or assume an event-ID check catches every repeated business outcome.
Shopify	Failed deliveries are retried eight times over the next four hours; handlers should respond within five seconds.	`X-Shopify-Webhook-Id` identifies a delivery; `X-Shopify-Event-Id` correlates deliveries from one merchant action. Ordering is not guaranteed.	Distinguish delivery deduplication from event correlation, queue work, and run reconciliation jobs.
Twilio Event Streams	At-least-once delivery can retry for up to four hours with exponential backoff and jitter.	Duplicates have the same event `id`; events can arrive out of order. Twilio says to return `200` when safely discarding a duplicate.	Store the event ID, use timestamps for reconciliation, and follow Twilio's status-code behavior for retry or discard.
Paddle	Live destinations can retry 60 times over three days; the server should return `200` within five seconds.	`event_id` identifies the event, while `notification_id` identifies a destination delivery. Delivery order is not guaranteed.	Deduplicate by event in the correct scope, preserve notifications as evidence, and compare `occurred_at` or current state.
GitHub	A response after ten seconds is treated as a failure. GitHub does not automatically redeliver failed deliveries.	`X-GitHub-Delivery` is unique per event and remains the same for a requested redelivery.	Queue promptly, monitor failed deliveries, and implement manual or scheduled redelivery recovery.
HubSpot	Webhook failures can be retried up to ten times across the next 24 hours; a response taking more than five seconds can trigger retry.	Notifications are not guaranteed to arrive in a specific order or batch. Current APIs also expose journal and snapshot options for recovery designs.	Preserve account and subscription context, tolerate batches, and reconcile from documented historical or current-state sources.
Queue layer	Cloudflare Queues documents at-least-once delivery, configurable retries, explicit acknowledgement, and dead-letter queues.	A queue can redeliver one message or an unacknowledged batch after consumer failure.	The worker still needs event and effect idempotency even after the HTTP endpoint accepted the webhook.

Stage 1-2

Verify the exact request before trusting its event identity

Capture the raw request body in the form required by the provider's signature algorithm. A framework that parses JSON, changes whitespace, converts encoding, or reconstructs the body before verification can make a valid request fail. Keep the signature verification path small, deterministic, and tested with the provider's official examples.

Signature validity and duplicate detection answer different questions. A valid signature says the request matches the secret and signed content under the provider's method. It does not say the event is new. A previously valid request can also be replayed, so apply the provider's timestamp or freshness control where available and still enforce event uniqueness. Conversely, a familiar event ID inside an invalid request does not make the request safe.

Check	Record safely	Do not expose	Failure action
Endpoint and method	Route version, method, host, and destination ID	Secrets in query strings	Reject or route to a controlled unknown-endpoint log
Signature	Algorithm, secret version, pass/fail, and failure class	Secret, complete signature material, or private payload	Reject before event claim or business processing
Freshness	Signed timestamp and policy result where supported	A globally reusable raw request	Reject stale or unverifiable requests under the provider contract
Payload contract	Content type, byte count, schema version, event type, and validation result	Payment details, credentials, unnecessary personal data, or full logs	Reject, quarantine, or alert according to permanent versus transient cause
Subscription	Provider account, tenant, destination, and enabled event types	Cross-customer payload access	Hold events that cannot be assigned to the expected tenant safely

Stage 3

Use an event ledger that can explain receipt, processing, and effects

A durable ledger is more useful than a list of processed IDs. It should answer which endpoint accepted the request, whether verification passed, which stable event was claimed, which payload version was seen, how often delivery repeated, which worker processed it, which business effects ran, and whether recovery remains open.

Field group	Recommended values	Why it matters
Source scope	Provider, environment, account or tenant, endpoint or subscription	Prevents an identifier from one customer or environment suppressing another
Event identity	Event ID, delivery ID, event type, object ID, API or schema version	Separates the provider occurrence from destination delivery evidence
Time	Occurred, provider-created, first received, latest received, accepted, completed	Supports delay, ordering, ageing, and service-level analysis
Integrity	Verification result, request hash, payload hash, safe size, secret version	Identifies changed payloads or verification drift without publishing secrets
Processing state	Received, verified, claimed, queued, processing, completed, retry, blocked, dead-letter	Makes uncertain and partial states visible
Effects	Effect name and version, idempotency key, destination ID, result, attempt, error class	Allows one event to drive several independently repeat-safe changes
Recovery	Replay source, next attempt, owner, review reason, reconciliation status, resolution note	Turns failures into an operable queue instead of hidden log entries

Choose retention from the recovery window, provider replay window, audit need, privacy policy, storage risk, and legal obligations of the actual system. "Keep every event forever" is not a reliability policy. Store the minimum necessary payload or a protected reference, define expiry, and preserve the stable deduplication or effect record long enough to cover realistic automatic and manual replay.

Stage 4

Use a unique constraint or atomic insert, not check-then-create

A handler that first asks "have I seen this event?" and then inserts a row has a race. Two deliveries can read "no" before either writes, so both workers process the event. The claim must be one atomic database operation or transaction protected by a uniqueness rule such as provider, environment, tenant, endpoint, and event ID.

The exact key depends on the provider. Shopify distinguishes a delivery ID from an event ID. Paddle distinguishes notification from event. GitHub retains its delivery identifier on redelivery. Stripe notes that separate Event objects can sometimes represent a duplicate business occurrence, requiring an additional domain-level effect key. Record both the provider identifiers and your selected business key rather than forcing every platform into one field.

Claim result	Interpretation	Safe action
New event inserted	This receiver owns first processing for the scoped event	Persist or enqueue, record accepted state, then acknowledge
Existing event, same payload hash	Expected retry or duplicate delivery	Increment delivery evidence, return the documented success response, do not repeat completed effects
Existing event, changed hash	Possible version, redaction, provider behavior, corruption, or spoofing issue	Quarantine for review; do not silently replace the accepted event
Existing event, failed effect	Delivery is duplicate but recovery may still be open	Acknowledge receipt and let the controlled worker retry only the unfinished effect
Separate event, same intended effect	Possible semantic duplicate or legitimate state update	Apply the domain-level effect key and state rules before changing the target

Stage 5

Acknowledge quickly, but only after the event is durably accepted

Vendor guidance commonly says to return success before complex downstream processing. That does not mean returning success before any reliable record exists. The balanced sequence is: receive raw bytes, verify the request, validate the minimum envelope, atomically claim the event, write it to a durable ledger or queue, and then return the provider's required success response. Complex CRM, payment, email, access, reporting, or enrichment work continues asynchronously.

Returning success first creates a loss window if the process crashes before persistence. Completing all business work first creates a retry window if the provider times out after the effect but before it reads the response. Durable acceptance between those two risks keeps the HTTP path short without pretending that memory, a background promise, or an in-process array is a queue.

A recognized duplicate should normally be acknowledged as successfully received according to the provider's contract. Twilio explicitly documents returning 200 when discarding a duplicate. Repeatedly returning an error for a safe duplicate can create more retries without changing the final outcome.

Stage 6

Give every state-changing effect its own idempotency key

Event deduplication prevents the same event from starting twice, but it does not completely protect a multi-step worker. The worker can update the CRM and crash before marking the event complete. On retry, it needs to know that the CRM effect already succeeded while the notification did not. Track each effect separately.

Effect	Unsafe operation	Repeat-safe pattern	Evidence
CRM contact or deal	Create a new record on every worker run	Resolve identity, upsert allowed state, and store the returned CRM ID against the effect key	Lookup result, operation, destination ID, prior and final state
Payment or refund	Send another charge or refund request after an uncertain timeout	Use the provider's idempotency control where supported and query the existing operation before repeating	Provider idempotency key, request ID, status, amount, currency, and result ID
Email or message	Send once per attempt	Reserve a message effect key before send and reconcile uncertain provider responses	Template/version, recipient reference, provider message ID, and final status
Access or entitlement	Append another grant, role, or expiry extension	Set the desired entitlement state from authoritative payment or subscription state	Source status, desired state, prior state, final state, and transition reason
Counter or balance	Increment for every delivery	Record one contribution keyed to the event or recompute from source records	Contribution key, previous value, new value, and source record
Analytics	Emit another conversion on every retry	Use a stable event ID where the analytics destination supports it and keep operational retries separate	Measurement event name, event ID, timestamp, and destination response

A useful internal key can combine the scoped source event, effect name, and effect version, for example provider:tenant:event-id:crm-upsert:v1. Version the effect when its business meaning changes. Do not include secrets or raw personal data in the key.

Stage 7

Use domain state, not arrival order, for out-of-order events

A later delivery is not necessarily a later business event. Stripe, Shopify, Twilio, Paddle, and HubSpot all document order limitations in the sources below. A simple "last request wins" rule can reopen a canceled subscription, replace a newer CRM value with an older one, or grant access after a refund.

Choose the strongest ordering evidence available for the selected provider and resource: monotonic version or sequence, provider event time, object update time, allowed state transition, or a fresh API read of authoritative current state. Timestamps are useful but not universally sufficient. Events can share a timestamp, clocks and formats differ, and an older event may still contain a necessary side effect. Record the rule for each event family.

Incoming condition	Example	Decision	Evidence
Newer resource version	Subscription version 8 after stored version 7	Apply the allowed transition and store version 8	Version, prior state, transition, and final state
Older resource version	Delayed version 6 after version 8	Mark stale; do not overwrite current state	Stale reason and retained event record
No version, trustworthy event time	Update occurred before the latest accepted update	Apply the documented timestamp rule or fetch current state	Occurred time, stored watermark, and API result if used
Forbidden transition	Active after terminal deletion without a restore event	Hold for reconciliation or fetch the authoritative resource	Transition rule, event type, and current provider state
Independent effect	Invoice payment and customer-profile update arrive in either order	Process each idempotent effect without forcing an unrelated total order	Separate effect keys and resource states

Response contract

Choose status codes from documented sender behavior

There is no universal webhook response-code table. Twilio documents different retry or discard behavior for 429, other 4xx, and 5xx on Event Streams webhook sinks. HubSpot documents retries for webhook 4xx and 5xx. GitHub does not automatically redeliver failures. Paddle expects 200. Stripe retries unsuccessful delivery. Implement the selected provider's contract rather than assuming that every 400 is permanent or every 429 is honored.

Receiver state	General response intent	Required caution
Verified and durably accepted	Return the provider's documented success response promptly	Success confirms acceptance, not completion of every asynchronous effect
Verified recognized duplicate	Return documented success after recording the duplicate safely	Do not run completed effects again
Invalid signature or wrong endpoint	Reject under the provider's security guidance	Do not disclose which secret, account, or payload field failed
Temporary inability to accept durably	Use the provider's retryable response behavior	Confirm that the selected code actually causes retry for this provider
Permanent invalid payload	Use the provider's permanent-failure or discard behavior, or quarantine after durable acceptance	A wrong code can create an endless retry or silently discard a recoverable event
Unknown outcome after partial work	Do not guess; reconcile the event and effect state	A blind retry can duplicate the business result

Recovery

Replay through the same path after the failure is understood

A replay is a recovery tool, not a separate bypass. Fix or contain the root cause, identify the affected event window, make the receiver and each unfinished effect repeat-safe, and then send the original event through the normal verification, claim, state, effect, and evidence path. The claim record should recognize the event while the effect ledger identifies what still needs work.

Provider replay behavior differs. GitHub allows recent deliveries to be redelivered and retains the original delivery header. Paddle creates a new notification linked to the original event. Stripe manual resend can overlap with automatic retry. Shopify recommends reconciliation because delivery alone should not be the only source of truth. Record who initiated a replay, from which provider control or script, which event window it covers, and what result changed.

Failure state	Retry or replay?	Required check first	Final evidence
No durable event record	Provider redelivery or reconciliation import may be needed	Confirm the source event exists and the receiver is now reachable	Recovered source event and first durable claim
Event accepted, no effect started	Retry the worker	Confirm event verification and current business validity	Effect key, attempt, result, and completion
Effect timed out with unknown result	Reconcile before retry	Query destination by idempotency key, request ID, or expected resource state	Existing result or one controlled new operation
One of several effects failed	Retry only the unfinished effect	Confirm completed effect records and dependencies	Independent effect statuses and final event state
Stale or forbidden event	Do not blindly replay	Fetch authoritative current state and apply transition policy	Stale, superseded, reconciled, or approved-manual decision
Poison event	Move to controlled dead-letter review after bounded retries	Classify permanent payload, code, permission, or business-rule failure	Owner, safe evidence, correction, replay decision, and closure

Acceptance testing

Test the uncertain moments that create duplicate effects

A successful sample event tests only the easiest path. Use non-private synthetic values and force the timing and failure cases that a production system will eventually see. Keep one correlation line from provider event through receiver, queue, worker, destination, and reconciliation record.

Test	How to simulate safely	Pass condition
Exact duplicate	Send the same valid event twice with the provider's test or redelivery control	Two delivery records, one event claim, and one intended effect
Concurrent duplicate	Deliver the same event to two workers at nearly the same time	The uniqueness rule allows one claim without a race-created duplicate
Timeout after effect	Complete a controlled effect, then interrupt result recording	Recovery finds the existing effect and does not repeat it
Out-of-order pair	Send a newer state before an older state	The final resource follows version, transition, or authoritative-state rules
Invalid signature	Alter a test payload or use the wrong test secret	The request is rejected before claim and no private diagnostics leak
Stale replay	Use a valid but expired test timestamp where the provider supports recency checks	The request follows the freshness policy and cannot repeat effects
Queue redelivery	Fail a worker before acknowledgement or inside a batch	Repeated queue delivery remains safe and only the unfinished effect continues
Dead-letter recovery	Force a bounded permanent test failure, correct it, and replay one event	The event is owned, corrected, replayed once, reconciled, and closed

Browser-local worksheet

Complete the 32-check webhook reliability review

The checkboxes and progress state stay in this browser through local storage. They are not submitted to eArif.com and they do not certify a live system. Reset clears the saved state. Print can be used for a privacy-safe review record.

Do not place webhook secrets, API keys, authorization headers, payment data, private payloads, customer records, unredacted logs, or production credentials in this worksheet.

Implementation route

Choose the route that matches the first unproven layer

Current evidence	Best next route	Why
A live endpoint has unknown signature, payload, response, timeout, retry, or replay behavior	API and Webhook Audit	The failure point and evidence need diagnosis before a rebuild or retry change.
The integration platform is not selected and the decision is between Zapier, Make, and n8n	Compare Zapier, Make, and n8n	Choose the runtime and operating model before diagnosing one platform's retry, replay, hosting, or recovery behavior.
Zapier is the known runtime and webhook runs fail, replay, or create repeated actions	Zapier Webhook Troubleshooting	The platform-specific run history, trigger, payload, replay, and action behavior should be reviewed together.
One form and CRM path is defined but still needs implementation and acceptance	Website Form to CRM Setup	The commercial owner covers the known form, CRM write, fields, routing, notification, QA, and handoff.
Several writers create duplicate contacts or conflicting identity decisions	CRM Automation Audit	The central problem is CRM identity and ownership rather than one delivery mechanism.
A connected workflow exists and needs final trigger, mapping, result, error, and recovery QA	Integration QA Checklist	The first unknown is broad launch acceptance, not only webhook delivery semantics.
Forms, CRM, payments, access, messaging, reporting, and business owners disagree	Start with Systems Audit	The source of truth and first failed handoff must be proven across the full system.

Review Integration and Workflow Services for adjacent fixed-scope routes, Proof before treating a public guide as project evidence, Privacy before sharing payload or customer context, Learning Cave for more implementation guides, and the AI Search Profile when a search or research agent needs the correct source for Arif's role and service boundaries.

Primary sources reviewed July 29, 2026

Official documentation used for the reliability model

Retry schedules, identifiers, timeouts, replay windows, response handling, signatures, limits, and interfaces can change. Recheck the current official page, selected account, environment, endpoint configuration, and one controlled provider test before production work.

IETF RFC 9110: Idempotent Methods for the standards definition of an idempotent HTTP method and the conditions around automatic retry.
Stripe: Receive events in a webhook endpoint for duplicate events, asynchronous handling, fast 2xx responses, signature recency, automatic and manual retries, and event ordering.
Shopify: About webhooks for event ordering, duplicate-delivery guidance, timestamps, and reconciliation jobs.
Shopify: Verify webhook deliveries for HMAC verification, webhook and event identifiers, duplicate handling, response timing, queuing, and retry behavior.
Shopify: Troubleshoot webhooks for delivery metrics, retry failures, timeouts, delays, removed subscriptions, and downtime recovery.
Twilio: Event delivery retries and event duplication for at-least-once delivery, exponential backoff with jitter, duplicate event IDs, out-of-order events, response-code behavior, and successful duplicate acknowledgement.
Paddle: How webhooks work for event versus notification identity, at-least-once delivery, event IDs, occurrence time, delivery order, and notification state.
Paddle: Handle webhook delivery for HTTPS, response timing, asynchronous work, live and sandbox retry behavior, order handling, signature verification, and replay.
GitHub: Best practices for using webhooks for event minimization, secrets, HTTPS, response timing, queues, event-type checks, redelivery, and delivery identity.
GitHub: Handling failed webhook deliveries for timeout behavior, failure inspection, scheduled recovery, and the absence of automatic redelivery.
GitHub: Redelivering webhooks for current manual and API redelivery controls and delivery availability.
HubSpot: Error handling for webhook connection, timeout, status-code, randomized retry, and retry-window behavior.
HubSpot: Webhooks API guide for current webhook settings, HTTPS endpoints, concurrency, subscriptions, batches, ordering, signatures, and retries.
HubSpot: Webhooks journal and management APIs for chronological event recovery, offsets, snapshots, filters, rate limits, and transient-error backoff.
Cloudflare Queues: Delivery guarantees for at-least-once delivery, duplicate processing risk, unique IDs, database constraints, and downstream idempotency keys.
Cloudflare Queues: Batching, retries and delays for explicit acknowledgement, batch redelivery, retry limits, delays, and non-idempotent work.
Cloudflare Queues: Dead Letter Queues for routing messages after bounded consumer retries and operating a recovery queue.
Zapier: How deduplication works for stable primary keys, seen-item handling, ordering requirements, and updated-item identity in Zapier integration triggers.

Frequently asked questions

Webhook retry and idempotency questions

What does idempotency mean for a webhook?

It means that receiving or processing the same source event repeatedly does not repeat the intended business effect. A POST webhook is not automatically idempotent. The receiver creates repeat-safe behavior with a scoped event key, atomic claim, durable acceptance, per-effect idempotency keys, destination state checks, and recorded outcomes.

Should a webhook return 200 for a duplicate event?

Return the selected provider's documented success response after the duplicate has been verified and recognized safely. Twilio explicitly says to return 200 when discarding a duplicate. Other providers may accept a broader 2xx range or require a specific response. Recording the duplicate and returning success usually prevents needless redelivery, but the current provider contract is the authority.

Is storing the provider event ID enough to prevent duplicates?

It is necessary in many integrations but not always sufficient. Scope the ID by provider, environment, tenant, and endpoint. Preserve delivery IDs separately. Protect the claim with an atomic uniqueness rule. Also use per-effect keys because a worker can fail after one downstream result. Stripe additionally documents cases where separate Event objects can represent a duplicate business occurrence, requiring object and event-type logic.

How long should webhook event IDs be stored?

Choose retention from the provider's automatic and manual replay windows, queue retention, outage-recovery window, business audit need, privacy policy, storage risk, and legal obligations. Keep the minimum evidence needed to prevent a realistic replay and reconcile effects. Define expiry explicitly rather than storing full private payloads forever.

How should out-of-order webhook events be handled?

Do not treat arrival time as business order. Use a provider sequence or resource version when available, otherwise apply documented occurrence or update times, allowed state transitions, or a fresh API read of the authoritative resource. Record stale and forbidden transitions instead of silently overwriting newer state.

How can I test webhook idempotency before launch?

Use clearly labeled non-private synthetic data. Send the same valid event twice, race two deliveries concurrently, deliver a newer state before an older one, force a worker retry, interrupt processing after one controlled effect, test an invalid signature and stale replay, and recover one dead-letter event. Pass only when every delivery is visible and each intended business effect occurs once.

Webhook Retries, Duplicate Events, and Idempotency: A Practical Guide