Postgres Is Probably Enough for Your Job Queue

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to observability.

When we finally measured, the numbers were worse than anyone guessed. Around 30% of the time spent on a typical change went into working around observability rather than building anything.

The problem

We had good intentions. We even had a document describing how observability should work. The trouble was that the document and the code had quietly gone their separate ways.

The setup was ordinary: SQLite at the edges, PostgreSQL doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

The fix

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what SQLite already gives you out of the box.

Here's a trimmed-down version of what we ended up with:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Measure before and after, even if it's just a rough number in a spreadsheet.
Make the default the safe choice, not just the documented one.
Delete dead code aggressively; it's the cheapest refactor there is.
Pair on the first few changes so the knowledge doesn't live in one head.

Caveats

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Node.js for everything. That's a fair question with a boring answer: we tried, and it made observability harder to reason about.

The biggest cost was social, not technical. Changing how people think about observability takes much longer than changing the code.

Wrapping up

That's it for now. Next time I'll dig into idempotency, which turned out to be closely related.