Your Webhook Handler Will Run Twice
Webhook providers retry. That is not a bug, it is the contract. The bug is on the receiving side when the same event changes business state twice.
If your webhook handler is not idempotent, every retry is a duplicate invoice, a duplicate email, or a duplicate customer.
Webhook providers retry. Timeouts, 5xx responses, network partitions, and deploy windows all cause the same event to arrive more than once. That is not a bug in their system. It is the contract of delivering messages over an unreliable network.
The bug is on the receiving side when the same event is processed twice and nobody notices until a customer sees two charges.
The failure is quiet
A duplicate webhook usually does not throw. The handler runs, the database accepts the insert, the email sends, and the logs show success twice. Nothing pages anyone.
The problem surfaces later, in finance or in support, and by then the evidence is buried under weeks of normal traffic.
A practical checklist
- Store the provider event ID and reject duplicates before doing any work.
- Make the business write and the event record part of the same transaction, or use an outbox.
- Return 2xx quickly, then process asynchronously.
- Make the handler safe to run twice, not just guarded against running twice.
- Alert on duplicate event IDs, not only on failed requests.
- Log the provider, the event ID, and the idempotency key for every attempt.
- Handle out-of-order delivery, because retries do not preserve ordering.
The distinction in the fourth item is the one teams get wrong. A unique constraint stops the second write, but it does not stop the side effects that happen before the constraint fires. Idempotency has to cover the whole handler, including the email, the queue publish, and the audit record.
Use the event ID the provider gives you
Most providers send an event identifier and a signature. Use them. Derive an idempotency key, store it with the processed result, and return the same response for a repeat delivery.
If the provider does not send an ID, hash the payload plus the resource version. It is a weaker key, but it is far better than nothing.
Test it before production does
Send the same webhook twice in staging and check the business state, not just the HTTP response. Then send two different events concurrently for the same record and check for lost updates.
The hard part is not receiving the webhook. It is proving that receiving it twice did not change the business state twice.
How do you handle a duplicate webhook today?
This was published first as Dispatches from the Workshop.
Subscribe on LinkedIn