The Case for Boring Background Jobs

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to background jobs.

We had good intentions. We even had a document describing how background jobs should work. The trouble was that the document and the code had quietly gone their separate ways.

How we got here

Most of the complexity was accidental. Over the years we'd layered three different approaches to background jobs on top of each other, and each one made sense at the time.

The first symptom was subtle. Nothing failed outright, but background jobs kept drifting in the wrong direction, and every fix seemed to make the next one harder.

What we changed

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Here's a trimmed-down version of what we ended up with:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

Agree on what "done" means before the first pull request.
Make the default the safe choice, not just the documented one.
Write the migration guide before writing the migration.

What it cost us

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use PostgreSQL for everything. That's a fair question with a boring answer: we tried, and it made background jobs harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Closing thoughts

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.