Structured Logging Is the Cheapest Observability You'll Buy

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to observability.

The first symptom was subtle. Nothing failed outright, but observability kept drifting in the wrong direction, and every fix seemed to make the next one harder.

What was going wrong

When we finally measured, the numbers were worse than anyone guessed. Around 36% of the time spent on a typical change went into working around observability rather than building anything.

The setup was ordinary: gRPC at the edges, Node.js doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

What we changed

The key insight was to treat observability as part of the interface rather than an implementation detail. Once we did that, most of the other decisions followed naturally.

Stripped of the project-specific bits, it looks like this:

UPDATE jobs
SET status = 'running', started_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload;

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Keep the old path around until the new one has survived a real incident.
Make the default the safe choice, not just the documented one.
Delete dead code aggressively; it's the cheapest refactor there is.
Pair on the first few changes so the knowledge doesn't live in one head.

What it cost us

The biggest cost was social, not technical. Changing how people think about observability takes much longer than changing the code.

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use SQLite for everything. That's a fair question with a boring answer: we tried, and it made observability harder to reason about.

Takeaways

If you take one thing away from this, let it be this: observability is a design decision, not a cleanup task. Make it early and make it on purpose.