Idempotency: The Skill That Quietly Separates Senior Engineers

Here is a bug that has cost real companies millions of dollars, and it is only two words long: run twice.

A user taps "Pay" and the button seems to hang, so they tap it again. A mobile client's request times out at 30 seconds, so your SDK retries it. The original request actually succeeded; the response just got lost on the way back. A message queue redelivers an event because the consumer crashed one millisecond before acknowledging it. An AI agent gets a timeout from your API, shrugs, and calls the same tool a second time.

In every one of these stories, the same operation runs more than once. And in every one, the question that decides whether you ship a reliable product or a support-ticket generator is the same: what happens the second time?

The property that answers that question is called idempotency. It is not a framework, a library, or a language feature. It is a way of thinking about operations that, once you internalize it, changes how you design nearly everything. Most tutorials mention it in one sentence and move on. This piece does not. By the end you will be able to reason about idempotency across the entire stack: the browser, your API, your database, your message bus, and the autonomous agents now calling your endpoints. You will also know exactly how to explain it in an interview in a way that makes a hiring manager lean forward.

1. The real definition

The word comes from mathematics. An operation f is idempotent if applying it twice gives the same result as applying it once:

f(f(x)) = f(x)

Multiplying by 1 is idempotent. So is taking the absolute value: abs(abs(-5)) == abs(-5). So is setting a value: x = 5 leaves x at 5 no matter how many times you run it. Adding, by contrast, is not idempotent. x = x + 5 gives a different answer every time.

Now translate that from math to systems. An operation is idempotent if performing it multiple times has the same observable effect on the system as performing it once. Note the careful wording. Idempotency is not about getting the same response back. It is about leaving the world in the same state.

Once you hold that definition, the discipline becomes a single relentless question you ask of every write operation you design: "If this runs twice, does the world look the same as if it ran once?" If yes, you are done. If no, you have work to do, and the rest of this article is that work.

2. The HTTP methods lie you were taught

Most developers can recite that GET, PUT, and DELETE are idempotent while POST is not. Fewer understand what that claim actually means, and fewer still realize it is a promise you are responsible for keeping, not something the protocol enforces.

The HTTP specification distinguishes two properties. Safe methods (GET, HEAD, OPTIONS) are not supposed to modify state at all. Idempotent methods (GET, HEAD, OPTIONS, PUT, DELETE) can modify state, but doing so repeatedly must have the same effect as doing it once.

Here is the part the spec cannot do for you: nothing stops you from writing a PUT handler that appends a row to a table on every call. The moment you do, your "idempotent" endpoint is not, and the clients, proxies, and retry libraries that trust the HTTP contract will happily corrupt your data on your behalf. Idempotency is a semantic guarantee you implement, not a syntactic one you get for free.

The practical design rule that falls out of this: PUT should express "make the resource equal to this" (an absolute state), while POST expresses "do this thing" (a relative action). The first is naturally idempotent. The second is where all the danger lives, and where the rest of our techniques apply.

3. Frontend: the double-submit, and how to actually kill it

The most visible idempotency failure is the oldest one: a user submits a form twice. The naive fix is to disable the button on click. That helps with the impatient double-click, but it does nothing for the case that actually corrupts data: the request that succeeds on the server but fails to deliver its response, prompting a retry from a fresh page load or a flaky network layer.

Disabling the button is UX. Real idempotency has to reach the server. The correct frontend move is to generate a stable idempotency key at the moment the user's intent is formed, when the form mounts rather than when it submits, and send that same key on every retry of that logical action.

// The key represents the user's INTENT, created once per form instance.
// It survives re-renders and is reused across network retries.
function CheckoutForm() {
  const idempotencyKey = useRef(crypto.randomUUID()).current;
 
  async function submit(payload) {
    return fetch("/api/charges", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey, // same key on every retry
      },
      body: JSON.stringify(payload),
    });
  }
  // ...retry submit() as aggressively as you like; the server dedupes.
}

Notice the inversion of responsibility. The frontend's job is not to prevent duplicates, because networks make that impossible. Its job is to make duplicates recognizable so the backend can absorb them. That single header is the handshake between "the client will retry" and "the server can tell it is a retry." Now let's build the other half.

4. Backend: idempotency keys, done correctly

This is the pattern that powers Stripe, and it is worth studying because the naive version has three race conditions that only show up under load, which is exactly when you cannot afford them. Stripe's own approach is to save the status code and body of the first request for a given key and replay that stored result for any repeat, comparing parameters to reject misuse. Their engineering write-up is the canonical reference.

Start with storage. You need a table that records each key, a fingerprint of the request, and the eventual result:

CREATE TABLE idempotency_keys (
  key            TEXT PRIMARY KEY,
  request_hash   TEXT        NOT NULL,   -- fingerprint of the payload
  status_code    INT,                    -- NULL until the work completes
  response_body  JSONB,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

Now the handler. The trick to the whole thing is claiming the key atomically before doing any work, using the database's own uniqueness guarantee as the lock. An INSERT ... ON CONFLICT DO NOTHING either succeeds (you are the first, do the work) or affects zero rows (someone got here first, replay their result). No separate "check then insert," which is the classic race.

async function handleCharge(req, res) {
  const key = req.header("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });
 
  const requestHash = sha256(canonicalize(req.body));
 
  // (1) Atomically try to CLAIM the key. This is the lock.
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, request_hash)
     VALUES ($1, $2)
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, requestHash]
  );
 
  if (claimed.rowCount === 0) {
    // (2) We've seen this key before. Fetch the prior record.
    const prior = (await db.query(
      `SELECT request_hash, status_code, response_body
       FROM idempotency_keys WHERE key = $1`, [key])).rows[0];
 
    // (2a) Same key, DIFFERENT payload = client bug. Refuse.
    if (prior.request_hash !== requestHash) {
      return res.status(422).json({
        error: "Idempotency-Key reused with different parameters",
      });
    }
    // (2b) First request is still in flight. Tell client to wait.
    if (prior.status_code == null) {
      return res.status(409).json({ error: "Request in progress, retry shortly" });
    }
    // (2c) We have a stored result. Replay it byte-for-byte.
    return res.status(prior.status_code).json(prior.response_body);
  }
 
  // (3) We own the key. Perform the side effect EXACTLY ONCE.
  const charge = await paymentGateway.charge(req.body);
 
  // (4) Persist the result so future retries can replay it.
  await db.query(
    `UPDATE idempotency_keys
     SET status_code = $2, response_body = $3 WHERE key = $1`,
    [key, 200, charge]
  );
 
  return res.status(200).json(charge);
}

Four details separate this from the toy version you will find in most blog posts, and each one is worth pointing out in an interview:

• The request fingerprint (2a). Storing a hash of the payload lets you detect a client that reuses a key for a different operation, a real bug you want to surface loudly rather than silently paper over. Stripe does exactly this.
• The in-flight state (2b). The window between claiming the key and writing the result is real. If a retry lands during it, you must not run the work again and you must not return a stale result. You return 409 and let the client back off. This is the race that bites people in production.
• Result replay (2c). You return the original response, including the original status. A retry of a request that failed with 402 should get 402 again, not a fresh attempt that might now succeed and surprise everyone.
• Scope and expiry. Keys should be scoped to an account and expired on a schedule (Stripe uses a 24-hour window on its v1 API and up to 30 days on v2). Keys are not permanent; they cover the retry window, not all of history.

5. Natural idempotency: designing operations that cannot double

Idempotency keys are a way to bolt safety onto an operation that is not inherently safe. The senior move is to reach for them less often, by designing operations that are naturally idempotent, so retries are harmless without any bookkeeping at all. This is where the mathematical definition earns its keep. Prefer operations that set over operations that change-by.

Absolute writes over relative writes

-- NOT idempotent: every retry moves the balance further.
UPDATE accounts SET balance = balance + 100 WHERE id = 7;
 
-- Idempotent: converges to the same state no matter how many times it runs.
UPDATE accounts SET balance = 500 WHERE id = 7;

Guarded state transitions

When you must move through states (a relative change), make the transition conditional on the current state. The WHERE clause turns a dangerous update into a self-guarding one: the second execution matches zero rows and does nothing.

-- Ship the order only if it's currently paid.
-- Run it twice and the second call is a harmless no-op.
UPDATE orders SET status = 'shipped'
WHERE id = 42 AND status = 'paid';

Upserts instead of blind inserts

-- A retry can't create a duplicate row; it just re-settles the same one.
INSERT INTO subscriptions (user_id, plan)
VALUES (7, 'pro')
ON CONFLICT (user_id) DO UPDATE SET plan = EXCLUDED.plan;

Set semantics over counters

In Redis, INCR likes is not idempotent; a redelivered event inflates the count. SADD likes user:7 is idempotent, because adding the same member to a set twice leaves the set unchanged, and you get the count for free with SCARD. Choosing the right data structure can make an entire class of double-processing bugs impossible.

The best idempotency key is the one you never had to write, because you chose an operation that could not double in the first place.

6. Distributed systems: why "exactly-once" is a marketing term

Everything above assumed a single request path. Distributed systems add message brokers, queues, and event streams, and with them a hard truth worth stating plainly in any senior interview: exactly-once delivery is impossible. Not hard. Impossible, in the formal sense, because of the Two Generals Problem: you cannot guarantee that a message was both delivered and its acknowledgment received, since either can be lost, and no finite number of acknowledgments-of-acknowledgments closes the gap.

So systems offer one of three delivery guarantees:

What every "exactly-once" system on the market actually does is combine at-least-once delivery with deduplication. Kafka, for instance, gives its producers a unique ID and a per-partition sequence number so the broker can discard a resent batch it has already seen, and it layers transactions on top for atomic multi-partition writes. That is dedup plus atomicity, not magic. The broker promises never to lose your message, which means it will sometimes deliver it twice, and something downstream has to absorb the duplicate. The industry name for this honest version is effectively-once. Idempotency is not a nice-to-have here. It is the load-bearing wall.

The two patterns that implement it

The inbox (consumer-side dedup). Every message carries a unique ID. Before processing, the consumer records that ID in a "processed messages" table inside the same transaction as its business write. A duplicate hits the primary-key constraint and is skipped.

async function consume(message) {
  await db.transaction(async (tx) => {
    const inserted = await tx.query(
      `INSERT INTO processed_messages (message_id) VALUES ($1)
       ON CONFLICT DO NOTHING RETURNING message_id`,
      [message.id]
    );
    if (inserted.rowCount === 0) return; // already processed, skip
 
    await applyBusinessLogic(tx, message); // same tx: both commit or neither
  });
}

The outbox (producer-side atomicity). The mirror-image problem: a service updates its database and publishes an event, two systems with no shared transaction. If it writes the DB then crashes before publishing, state and events diverge (the "dual-write problem"). The outbox pattern solves it by writing the event into an outbox table in the same transaction as the business change; a separate relay process reads that table and publishes to the broker. The event is published if and only if the business transaction committed. Pair an outbox producer with an inbox consumer and you have a pipeline that is duplicate-safe end to end, the real, honest "exactly-once."

7. The 2026 frontier: idempotency for AI agents

Here is the part almost nobody is writing about yet, and the reason this old idea suddenly matters more than it did a year ago.

For decades, the entity retrying your API was a piece of code a human wrote, a retry library with a fixed policy you could read. Increasingly, it is an LLM-driven agent that decides on its own to call your tool again because a response was slow, a schema validation tripped, or the model simply was not sure the first call worked. The retry behavior is genuinely different in kind: a conventional client retries on a well-defined error, whereas an agent can re-issue a call it has no reliable way of knowing already succeeded. Analyses of production agent traces have found that a large share of retries are redundant, re-running work that already completed, precisely because the agent cannot observe the side effect it caused. Unlike a retry library, the agent will not read your docs about how to be careful. (I have written more about this in verifying AI-generated code and how agents lose context over time.)

The good news: everything you just learned applies directly. The emerging best practice for agent tools is to make write and commit operations enforce idempotency through a caller-supplied key computed from the call's parameters, returning the cached result if the same key reappears within a window. That is the Stripe pattern, wearing a new hat. Here is a decorator that turns any tool into a retry-safe one:

import hashlib, json
 
def idempotent_tool(store, ttl=86400):
    """Wrap an agent tool so identical calls execute at most once per TTL."""
    def decorator(fn):
        def wrapper(**kwargs):
            # Derive the key from the operation's meaningful inputs.
            key = hashlib.sha256(
                json.dumps({"tool": fn.__name__, "args": kwargs}, sort_keys=True)
                .encode()
            ).hexdigest()
 
            if (cached := store.get(key)) is not None:
                return cached            # retry: replay, don't re-run
 
            result = fn(**kwargs)          # the real side effect, once
            store.set(key, result, ttl=ttl)
            return result
        return wrapper
    return decorator
 
@idempotent_tool(store=redis_store)
def issue_refund(order_id: str, amount_cents: int):
    return payments.refund(order_id, amount_cents)   # safe to retry now

A subtlety worth naming: deriving the key from the arguments means an agent that calls issue_refund("order_99", 500) twice is deduplicated, which is what you want. But if the agent legitimately needs to refund the same order twice, the arguments alone cannot distinguish intent from retry, so for genuinely repeatable actions you let the caller pass an explicit operation ID. Knowing which of those two you are dealing with is the design judgment that separates someone who has read about idempotency from someone who has shipped it.

8. The engineer's idempotency checklist

Print this. Run every write path you design through it.

• Can this operation be naturally idempotent? Prefer absolute writes, guarded transitions, upserts, and set semantics before reaching for keys.
• If not, is there an idempotency key? Client-generated, high-entropy (V4 UUID), sent on every retry of the same logical action, never containing sensitive data.
• Is the key claimed atomically? Use a unique constraint, not check-then-act. No race window.
• Do you handle the in-flight case? A retry arriving mid-processing gets a 409, not a second execution.
• Do you detect key reuse with a different payload? Fingerprint the request; reject mismatches loudly.
• Do retries replay the original result? Same status, same body, including original failures.
• Are keys scoped and expired? Per account, with a defined retention window.
• Across services, is delivery at-least-once plus idempotent consumers? Inbox for dedup, outbox for atomic publish. Never trust "exactly-once" as a primitive.
• Are your AI-agent tools retry-safe by construction? Assume any write tool exposed to an agent will be called more than once, and design so the duplicate is harmless.

9. How to talk about this, and get hired

Idempotency is one of a handful of topics that instantly reveals engineering seniority, because it cannot be faked. Juniors have heard the word. Seniors have been paged at 3 a.m. by the absence of it. If a system-design interview involves payments, queues, webhooks, or anything an agent might call, and in 2026 that is most of them, you can create a visible moment by raising it before you are asked.

A line that lands: "Before we design the happy path, let's decide what happens when this runs twice, because in a distributed system, it will. I would make the write idempotent so retries are free, either naturally with a guarded state transition, or with a client-supplied idempotency key claimed atomically on a unique constraint." That single sentence signals that you think in failure modes, you know the vocabulary precisely, and you have clearly done this in production. It reframes you from someone who writes features to someone who ships systems that survive contact with reality.

That is the whole game. Idempotency is not a trivia answer; it is a lens. Once you start asking "what happens the second time?" of every operation you build, you write software that quietly refuses to break in the ways that page everyone else. Your users never see the duplicate charge that did not happen. And the engineers who inherit your code notice, immediately, that a senior wrote it.

Further reading: Stripe: Designing robust and predictable APIs with idempotency · Stripe API reference: idempotent requests · The Two Generals Problem.