Timeouts Everywhere: A Field Guide

This post started as a note to my future self. Every few months I relearn the same lessons about API versioning, so this time I'm writing them down.

We had good intentions. We even had a document describing how API versioning should work. The trouble was that the document and the code had quietly gone their separate ways.

The problem

Most of the complexity was accidental. Over the years we'd layered three different approaches to API versioning on top of each other, and each one made sense at the time.

The first symptom was subtle. Nothing failed outright, but API versioning kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Trying something different

Rather than a big-bang rewrite, we migrated one module at a time over about 12 weeks, keeping both paths alive behind a flag.

The core of it looks like this:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Keep the old path around until the new one has survived a real incident.
Pair on the first few changes so the knowledge doesn't live in one head.
Agree on what "done" means before the first pull request.

Trade-offs

We still haven't solved rate limiting. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use PostgreSQL for everything. That's a fair question with a boring answer: we tried, and it made API versioning harder to reason about.

What's next

That's it for now. Next time I'll dig into caching, which turned out to be closely related.