Idempotency and Retry Lab
Lose payment responses, replay idempotent requests, compare backoff with full jitter during an outage, and move exhausted jobs to a dead-letter queue.
Category: DevOps
What You Will Learn
- Create one UUID v4 idempotency key per intended operation and reuse it only for retries
- Replay a saved response instead of creating a duplicate charge after an ambiguous network failure
- Handle payload mismatches and concurrent keyed requests with distinct HTTP responses
- Choose exponential backoff or full jitter to reduce retry spikes during an outage
- Send exhausted jobs to a dead-letter queue with their key and final error
Topics covered: idempotency, retries, payments, backoff, dead-letter queue, interactive
// simulator
Idempotency and Retry Lab
Lose payment responses, replay idempotent requests, compare backoff with full jitter during an outage, and move exhausted jobs to a dead-letter queue.
Replay step: Request
Goal: Lose a response without an idempotency key and watch the customer pay twice. Press Run duplicate checkout: the server charges, the response is lost, and the client retries without a key.
Ready to model one checkout.
Run an experiment
Each action adds its observed outcome to the shared API model.
Checkout sequence
One row per observed request or response event.
- Run a checkout to see its requests.
Outage arrival chart
A simplified model: 200 clients call during a 3 s outage. Each bar is 100 ms.
Run no backoff, then full jitter to compare peaks.
Ready to model one checkout.
About the Idempotency and Retry Lab
What you'll practise
- Separating a lost response from a failed payment operation
- Reusing one idempotency key for one intended checkout and its retries
- Reading replay, payload-mismatch, and in-progress outcomes
- Comparing immediate retries with exponential backoff and full jitter
- Putting an exhausted job in a dead-letter queue instead of retrying forever
How it works
- The checkout model sends POST /charges to a payments API with a fixed example amount.
- The API records the request fingerprint and the first response for each keyed operation.
- Its outage chart is a simplified deterministic model of client arrivals, not a production measurement.
- The dead-letter queue records the final error and key, and does not retry its jobs automatically.
Sources
- IETF draft, The Idempotency-Key HTTP Header Field: key scope, payload fingerprints, conflicts, and expiration guidance.
- Stripe documentation, Idempotent requests: storing the first result, including failures, and removing keys after at least a day.
- AWS Architecture Blog, Exponential Backoff And Jitter: capped exponential delays and randomized full jitter during contention.
Overview
A network failure after a payment request is ambiguous: the client may not have a response, while the server may already have created the charge. This lab makes that ambiguity visible. First, you lose a response without a key and retry, which creates two charges. Then you repeat the checkout with one UUID v4 idempotency key for the intended operation. The API saves the first completed response and returns that same saved result to the retry, so the ledger has one charge. You also reuse a key with a different amount and see a 422 response, then send two identical keyed requests at once and see a 409 while the first request is processing. For retries, the lab compares immediate retrying, capped exponential backoff, and full jitter in a deterministic outage model with 200 clients. Full jitter randomizes each capped wait, reducing the arrival peak compared with no backoff. Finally, a job that remains unavailable through five attempts moves to a dead-letter queue with its idempotency key and final error. The model follows the IETF idempotency-key draft, Stripe idempotent-request behavior, and AWS guidance on exponential backoff and jitter.
Try this
- Press "Run duplicate checkout" and inspect the ledger after the lost response causes a second charge.
- Press "Run keyed checkout" and compare its one charge with the saved response replay in the idempotency store.
- Press "Run no backoff", then "Run full jitter", and compare the peak requests in the outage arrival chart.
Frequently asked questions
- Why does an idempotency key stop a duplicate charge?
- The server associates the key with one request fingerprint and saves the first result. A later request with that key and fingerprint returns the saved status and body instead of executing the charge again.
- What happens when the same key has a different payload?
- The lab returns 422 Unprocessable Content because one key cannot name two different operations. Stripe describes the equivalent case as an idempotency error, so the client should create a new key only for a genuinely new intended operation.
- Which failures should a client retry?
- Retry network errors, 409 conflicts, 429 rate limits, and 5xx responses, using a bounded delay. Do not retry 400 or 422, because those describe a request that needs correction rather than a transient failure.
- What does a dead-letter queue do here?
- After five unsuccessful attempts, the job is retained with its idempotency key and final error for inspection or deliberate replay. The queue does not automatically send it again, which prevents an endless retry loop.
Try next
// simulator
GraphQL N+1 and DataLoader Lab
Trace nested GraphQL resolvers, count N+1 SQL queries, batch users and comments with DataLoader, and discover why a loader cache must stay inside one request.
// simulator
Preview Environment Simulator
See how a pull request becomes a temporary copy of your app, complete with a private URL, safe test data, review checks, and automatic cleanup.
// game
Heroku-Style Name Generator
Generate names the way Heroku, Docker, Kubernetes, and petname do, see how the word lists combine, and learn when the birthday problem makes random names collide.