gao.ninja logogao.ninja

Panics, Request IDs and the Context That Flows the Wrong Way

Go's error handling is explicit: you return an error and the caller deals with it. That works beautifully until you meet the other half of the language, panic: a nil dereference, an unchecked type assertion, an index out of range. An unrecovered panic in a goroutine takes the process down. In an HTTP handler, net/http recovers for you, but all you get is a dropped connection and a log line nobody can tie to a customer complaint.

The Go services on a platform I worked on shared two generations of web libraries that handled this differently. Reading them next to each other is a good way to see what a decent panic story looks like, and where it can quietly go wrong.

Library one: recover once, in the wrapper

The older library lets you write handlers as plain typed functions. The framework decodes the request body, calls your function, and encodes the result. Because it owns the call, it owns the recovery:

// catch panics and gracefully respond
defer func() {
	err := recover()
	if err == nil {
		return
	}

	// set the stacktrace here so we only get stacktraces on panics
	ctx = ctx.WithValue("stack", debug.Stack())

	switch e := err.(type) {
	case error:
		g.http500Handler(ctx, e)(w, r)
	case string:
		g.http500Handler(ctx, errors.New(e))(w, r)
	default:
		panic(fmt.Sprintf("unhandled recover value of type %T: %v", err, err))
	}
}()

Three details are worth stealing:

The stack is captured only on a panic. debug.Stack() isn't free, and for ordinary returned errors the call site already tells you what happened. A panic is by definition unexpected, so that's when you want the stack.

Panics and returned errors converge. A handler that returns an error and a handler that panics both land in the same 500 handler. Logging, metrics and response shape live in one place.

The default branch re-panics. recover() returns any. If something panics with a value that is neither an error nor a string, the wrapper refuses to guess and crashes loudly. I think that's right: a quiet 500 for a truly strange panic value hides a bug you'd rather see.

The same wrapper also makes logs joinable with traces. When a request has an active APM span, it copies the IDs into the log context under the keys the APM product expects:

const traceID = "dd.trace_id"

if span, ok := tracer.SpanFromContext(ctx); ok {
	ctx = ctx.WithValues(map[string]interface{}{
		traceID:      span.Context().TraceID(),
		"dd.span_id": span.Context().SpanID(),
	})
}

One thing I would change: the default 500 handler in this library puts err.Error() into a details field in the response body. Convenient in development, risky in production, because an internal error message (a SQL fragment, a hostname) becomes part of your API surface. The library lets you replace the handler, and the safer default is a generic message plus an identifier.

Library two: a contract, a request ID, and a safety net

The newer router library takes a different angle. Routes are declared in a DSL (method, path, request schema, and the responses each route is allowed to return), and every route is wrapped in a chain of middleware: request-ID injection, route injection, request validation, then response validation.

The request-ID piece is small:

func (m *RequestIDInjector) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	requestID := uuid.NewString()
	m.wrappedHandler.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), ContextRequestID, requestID)))
}

The response-validation middleware is the interesting one. It runs the handler against an in-memory writer, then checks the status code and body against the schema declared for that route. If the handler returned something the contract doesn't allow, the caller never sees it:

w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(`{"error":"internal server error","reference":"` + requestID + `"}`))

That reference is the most valuable field in the response. When a partner's engineer or a support agent says "I got an error", the useful reply is what's the reference? One UUID finds the log lines for that request. And enforcing the declared contract at runtime means a handler can't quietly drift away from the API spec you published: the drift shows up as a 500 in testing rather than as a surprised client in production.

On top of that sits a router-level recovery middleware as a safety net, with an option to replace it. It logs the panic along with the request ID and trace IDs, then returns the same JSON shape.

The bug-shaped detail: context flows down, not up

Here is where I'd slow down in a code review. Look at the order of operations again.

The recovery middleware is registered on the router, so it wraps everything. The request-ID injector is applied per route, inside it, and it creates a new request via r.WithContext(...). Context values travel down the call chain through the request you pass along. They never travel back up to the caller's copy of the request.

So when a panic unwinds to the outer recovery, the r it holds is the original request, which doesn't contain the request ID:

Nested middleware showing the request ID added inside the recovery layer, so the recovery layer only sees the original requestThe injector passes a new request inward, and the outer recovery never sees it

requestID, ok := r.Context().Value(middleware.ContextRequestID).(string)
if ok {
	entry = entry.WithField("requestID", requestID)
}
// ... later
w.Write([]byte(`{"error":"internal server error","reference":"` + requestID + `"}`))

As far as I can tell from reading it (I didn't run it), on that path ok is false and requestID is the empty string, so the log line has no request ID and the response says "reference":"". The response-validation path, which lives inside the injector, does have the ID. Only the outermost safety net doesn't.

This is an easy mistake to make and an easy one to miss, because the happy path and the validation path both work. The fixes are cheap. The simplest is to generate the request ID in the outermost middleware, before recovery, so everything inside inherits it. If the order can't change, have the outer layer put a mutable holder in the context that the injector fills in, so the outer layer can read it back after the panic. Either way, test the unhappy path: a unit test that panics inside a handler and asserts the response contains a non-empty reference.

There's a second small gap: the router-level recovery logs the panic message but not the stack. The wrapper in the older library does capture it. Combining the two (stack from debug.Stack() plus request and trace IDs, in one log entry) is the version I'd want.

What I'd standardise

  1. One recovery layer per process, outermost, that owns the request ID, the trace IDs and the stack.
  2. A caller-visible reference and no internals in the body. Details go in the logs.
  3. Contract-checked responses where your framework can afford it.
  4. Recover in goroutines you spawn. A recover() in the handler cannot catch a panic in a goroutine your handler started. errgroup doesn't recover for you either, so start each goroutine with a deferred function that calls recover() (a bare defer recover() does nothing, because recover must be called from inside the deferred function).
  5. Alert on the rate of recovered panics. Recovering keeps the service up, but it can also keep it running in a corrupted state. A recovered panic is a bug report, so treat its frequency as a signal.

Wrapping up

A good 500 is one a caller can quote and an engineer can find in one search. Recover once, in the outermost layer, capture the stack there along with the request and trace IDs, and keep internals out of the response body. Above all, make sure the reference you hand the caller is created outside the layer that handles the failure, because context values only flow down.