Payments · 2026-11-12 · 6 min read

Payment succeeded, access denied: tracing a webhook failure end to end

A customer pays. The provider's webhook arrives more than once, or not at all. A queue job runs out of retries. The business sees a paid customer locked out, and support has no fast way to tell why.

Mohit Tanwani

Mohit Tanwani
Founder, Tech Mitrise

The short version

Payment providers deliver webhooks at-least-once, not exactly-once, so your handler will receive duplicates and will occasionally receive nothing until a retry. If the handler that grants access isn't idempotent, or the entitlement job's retry budget is too small, a customer ends up paid and locked out. The fix is a persisted delivery ledger, an idempotent consumer, and bounded retries with visible failure state — not a bigger timeout.

Why this isn't a rare edge case

A payment provider's webhook is not a promise that your server received it exactly once. Stripe says this directly in its own docs: webhooks are delivered at-least-once, and your endpoint has to handle duplicates and out-of-order delivery on its own. Most payment providers — Stripe, Paddle, Razorpay, Chargebee — make the same guarantee, because the alternative (exactly-once delivery over an unreliable network) isn't something a webhook can actually offer.

That guarantee is fine, as long as the code on the receiving end treats it as what it is: a feed that can repeat itself, arrive late, or arrive while your own queue worker is mid-restart. Three failure patterns show up over and over in systems we've inherited:

  1. The handler isn't idempotent. A payment.succeeded event grants access, extends a subscription, or increments a credit balance directly in the webhook handler. The provider retries the same event (a timeout, a 5xx, a deploy mid-request) and the customer gets charged-and-granted twice, or the grant logic runs against state it already changed, and throws.
  2. The entitlement job exhausts its retries silently. The webhook handler itself responds fast and queues the actual "give this customer access" work. If that queued job fails — a lock timeout, a transient DB error, a bug — and the retry budget is small (Laravel's default is 1 attempt unless you configure more), the job dies, the queue's failed-jobs table gets a row nobody is watching, and the customer stays paid and locked out until support escalates it.
  3. There's no durable record of what was received. Without a table that records "we got event evt_19ab... at this time and here's what we did with it," there's no way to answer "did we already process this?" except by re-deriving it from side effects — which is exactly the kind of logic that breaks under retries.

None of this requires a rare provider outage. It requires normal internet behaviour: a slow response, a deploy that restarts a worker mid-job, a provider that retries on a 2-second timeout. It happens on ordinary days, not just bad ones.

Not sure where your system stands?

A 20-minute call with the founder will tell you whether this applies to you and what to do first.

How to check your own system

You can get a real answer to "could this happen to us" in under an hour, without touching production:

  1. Read your webhook handler and ask: what happens if this exact request arrives twice in a row? If the answer involves double-granting, double-charging, or an exception on the second call, that's the idempotency gap.
  2. Check your queue's retry configuration for the job that grants access or updates entitlement, not just the webhook route itself. In Laravel, that's the job class's $tries and $backoff properties (or the retryUntil() method) — the webhook controller responding 200 OK fast tells you nothing about whether the actual grant succeeded.
  3. Look at your failed_jobs table (or provider's dashboard) for rows tied to payment or entitlement jobs. If you don't have alerting on that table, you likely have customers in this state right now and don't know it.
  4. Confirm you have a persisted idempotency key — a unique constraint on the provider's event ID, checked before any side effect runs, not just logged after.
# Laravel: a quick, read-only check for stuck entitlement jobs
php artisan queue:failed
php artisan tinker --execute="echo App\Models\FailedJob::where('queue', 'entitlements')->count();"

If you want to see this failure mode traced end to end — payment succeeding, a webhook delivered multiple times, a queued job exhausting bounded retries, and a safe, transaction-locked reconciliation that fixes it without double-granting — the Tech Mitrise Rescue Lab is a public, open-source Laravel project built to demonstrate exactly this. Every organisation, payment, and figure in it is explicitly synthetic lab evidence, not a real client's data; the project says so itself. It's real, runnable code (make demo brings up the whole stack locally), not a slide deck: a persisted delivery ledger, an idempotent consumer, three bounded job attempts with [1, 5, 15]-second backoff, and a reconciliation step that's verified safe to run twice. See github.com/TechMitrise/saas-revenue-integrity-rescue-lab.

What to do first

If you've found a real gap using the checks above, the fix is rarely a rewrite. In order of what actually closes the risk fastest:

  1. Add a webhook_deliveries (or similarly named) table keyed on the provider's event ID, with a unique constraint, written before any side effect runs. A duplicate delivery becomes a no-op, not a repeat charge.
  2. Raise the retry budget and backoff on the job that actually grants access, and make its failure visible — a Slack/email alert on that specific queue's failed-jobs count, not a generic "queue is backed up" dashboard.
  3. Make the reconciliation step idempotent and safe to re-run: given the same final state twice, running it again should change nothing. That's what lets you fix a stuck customer without worrying about making it worse.
  4. Add one regression test per failure mode: duplicate delivery, exhausted retries, and the reconciliation no-op. These are cheap to write and they're exactly the three ways this fails in practice.

None of this needs a new payment provider, a new queue system, or a rewrite of the billing flow. It needs the existing webhook handler and queue job to assume delivery will repeat, because it will.

Mohit Tanwani

Written by Mohit Tanwani

Founder of Tech Mitrise. He does the rescue and upgrade work described here. 10+ years building and rescuing production Laravel, Yii and PHP systems, including revenue and entitlement paths.

About the founder

Sources: Stripe's webhook documentation (stripe.com/docs/webhooks), Laravel's queue documentation (laravel.com/docs/queues), and the Tech Mitrise Rescue Lab, a public synthetic demonstration (see the Rescue Lab callout below). Last checked 12 Nov 2026.

Keep reading

ReferenceWhat actually breaks when a Laravel app passes end of supportHow we helpBook a free 20-minute triage