Testing this properly means simulating failure modes. Set up a test endpoint that only accepts the first call, then rejects everything else to test your retry handling. Use a webhook replay tool. Stripe gives you webhook logs so you can manually re-trigger them. Use that. Verify that re-triggering the same webhook three times in a row doesn't change your database. It should be idempotent.
The race condition is real. Imagine this sequence. Webhook 1 arrives for transaction ABC. Your handler checks the database. Event not found. It starts processing. Meanwhile, webhook 1 retries. Webhook 2 arrives checking the database while processing is still happening. Event still isn't in the database as "processed." Both webhooks process the same transaction. You've now minted two balance entries. Congratulations, you've broken your ledger.
The best approach here is stop pretending webhooks are settlement events. Webhooks are notifications that something occurred. Settlement is a separate concern. Your handler should update some transient state. A transaction claiming to be this address with this amount has appeared. Separate logic checks actual chain state and marks things as settled once they're actually final. Only at that point do you release goods or credits.
The standard pattern everyone uses looks like this in your handler. You receive the webhook. You check if you've already processed this event ID. If yes, return 200 OK and do nothing. If no, process it and store the event ID so you don't do it again. That's idempotency. It's boring. It's also the thing that separates "our system works" from "we lost someone's ETH transfer."
The final thing. Document your webhook handler. Write down exactly what "success" means for your endpoint. What status codes indicate success versus transient failure versus permanent failure. Most teams define this implicitly and it's different in every environment. Then someone runs the handler on a different server and suddenly the error handling breaks because the exception types are different. Use explicit error codes. Document the contract.
The testing problem is where most teams really messed up. You write your webhook handler locally. You test it by calling it once manually with curl. It works. You ship it. In production, the retry logic you never tested catches you. Most teams have done this. Someone gets paged at 2 AM because the webhook test didn't include "what happens if this endpoint gets called three times in parallel."
Now here's the part nobody talks about. Your payment processor is also not guaranteed to fire webhooks in order. Event 1 fires at timestamp T. Event 2 fires at timestamp T+500ms. But your webhook endpoint might process them backwards. Number 2 arrives first. Number 1 arrives late. If you're using creation timestamps to determine order, you're already wrong. Use monotonic sequence numbers if you can. Check them in your handler.
Also here's something I Googled recently and couldn't find a good answer to. What happens if your webhook endpoint and your payment processor's retry system drift out of sync on what "success" means? The processor thinks a webhook succeeded because the endpoint returned 200. Your endpoint returned 200 but the actual work failed. You need a reconciliation mechanism. Every few hours, query the payment processor's API and compare their view of settled transactions against your database. Anything missing gets reprocessed. This is unglamorous but it's what keeps real money systems honest.
The silent failure mode is the worst part. Your webhook endpoint returns 200 OK. Your payment processor thinks it's happy. But inside your handler, something failed in the actual business logic. Maybe you caught an exception and logged it. Maybe you just returned early because a database was down. The processor doesn't know. It thinks it was successful. It doesn't retry. Your customer's payment is processed on the payment side but never recorded on your side. The money is gone. They never got the goods. Your support team fields seventeen angry emails.
This is why you need observability inside the webhook handler. Not just "did we return 200" but "did we actually process the transaction successfully." Log the event ID. Log the result. Track which webhooks succeeded and which ones are sitting in a dead letter queue. If you're using any kind of job queue system (which you should be), put webhook processing on the queue. Let the queue handle retries. Your webhook endpoint becomes a dumb parser that enqueues the work and returns 200 immediately. The actual processing happens asynchronously. If it fails, the queue retries it automatically. This is how every production system worth money does it.
The real gotcha for blockchain payment stuff specifically is settlement timing. On a traditional payment system, the webhook fires after settlement is final. Money moved. The notification is confirmatory. Blockchain is different. You might get a webhook when a transaction enters the mempool (not mined yet). Then you get another when it's confirmed once. Then at twelve confirmations. Then at finality. Each webhook is technically a different event. But your merchant handler needs to understand that the transaction isn't actually settled until you say it is. A customer sees their balance increase after webhook 1. Then the transaction gets dropped due to gas spike. Now what? Their balance was wrong. The webhook was lying.
The retry strategy matters more than anyone admits. Most payment systems retry with exponential backoff. Stripe goes something like this. Immediate retry if it fails. Then 5 minutes later. Then 30 minutes. Then 2 hours. Then 5 hours. Then finally they give up and you're in webhook hell. But here's what breaks people. If your endpoint is actually broken (returning 500s, not just timing out), they'll keep hitting it for days. Which means when you finally fix the code at 6 AM Tuesday morning, suddenly seventeen webhook events flood in at once and you need idempotency to catch duplicates. This is genuinely what happened to us with a customer doing high-frequency token swaps. The handler had a bug. We fixed it. Seven retry waves hit us in sixty seconds. Without event deduplication, we would've minted fake balance records.