Idempotency Keys: The Boring Feature That Saved Us

This post started as a note to my future self. Every few months I relearn the same lessons about idempotency, so this time I'm writing them down.

The first symptom was subtle. Nothing failed outright, but idempotency kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Where we started

The setup was ordinary: Kafka at the edges, Redis doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

When we finally measured, the numbers were worse than anyone guessed. Around 45% of the time spent on a typical change went into working around idempotency rather than building anything.

Trying something different

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Kafka already gives you out of the box.

Stripped of the project-specific bits, it looks like this:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Pair on the first few changes so the knowledge doesn't live in one head.
Make the default the safe choice, not just the documented one.
Measure before and after, even if it's just a rough number in a spreadsheet.
Write the migration guide before writing the migration.

What it cost us

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use SQLite for everything. That's a fair question with a boring answer: we tried, and it made idempotency harder to reason about.

Wrapping up

If you take one thing away from this, let it be this: idempotency is a design decision, not a cleanup task. Make it early and make it on purpose.