What I Learned Writing My First gRPC Service

There is no shortage of articles about gRPC. Most of them cover the happy path. This one is about what happened after the happy path ended.

When we finally measured, the numbers were worse than anyone guessed. Around 20% of the time spent on a typical change went into working around idempotency rather than building anything.

What was going wrong

Most of the complexity was accidental. Over the years we'd layered three different approaches to idempotency on top of each other, and each one made sense at the time.

The first symptom was subtle. Nothing failed outright, but idempotency kept drifting in the wrong direction, and every fix seemed to make the next one harder.

The fix

Rather than a big-bang rewrite, we migrated one module at a time over about 5 weeks, keeping both paths alive behind a flag.

Here's a trimmed-down version of what we ended up with:

func withTimeout(next http.Handler, d time.Duration) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx, cancel := context.WithTimeout(r.Context(), d)
		defer cancel()
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

• Make the default the safe choice, not just the documented one.
• Measure before and after, even if it's just a rough number in a spreadsheet.
• Delete dead code aggressively; it's the cheapest refactor there is.
• Write the migration guide before writing the migration.

Caveats

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use PostgreSQL for everything. That's a fair question with a boring answer: we tried, and it made idempotency harder to reason about.

Wrapping up

That's it for now. Next time I'll dig into API versioning, which turned out to be closely related.