Designing APIs You Won't Have to Version

A colleague asked me last week why our team cares so much about background jobs. I gave a rambling answer, so here is the longer, more organized version.

The setup was ordinary: gRPC at the edges, Kafka doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

What was going wrong

The first symptom was subtle. Nothing failed outright, but background jobs kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Most of the complexity was accidental. Over the years we'd layered three different approaches to background jobs on top of each other, and each one made sense at the time.

Trying something different

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what gRPC already gives you out of the box.

The core of it looks like this:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Keep the old path around until the new one has survived a real incident.
Agree on what "done" means before the first pull request.
Make the default the safe choice, not just the documented one.

Where this breaks down

The biggest cost was social, not technical. Changing how people think about background jobs takes much longer than changing the code.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Takeaways

If you take one thing away from this, let it be this: background jobs is a design decision, not a cleanup task. Make it early and make it on purpose.