Your /health Endpoint Is Lying to You
Here is a health handler from a production Go service on a payments-adjacent platform. I've trimmed the route metadata, but the logic is all there:
func (rt *Health) ServeHTTP(w http.ResponseWriter, r *http.Request) {
body := map[string]string{
"message": "ok",
}
// todo: consider adding simple check of the upstream if possible...
writeResponse(w, body, http.StatusOK)
}
It always returns 200 {"message":"ok"}. The author even left the comment that gives the game away: the check does not check anything.
That isn't a criticism of one engineer. This pattern is everywhere, because it solves the first problem (give the load balancer something to hit) and leaves the second problem (make that something meaningful) for later. The trouble is that "later" is when an outage is already in progress.
Who consumes a health endpoint?
On that platform there were at least three consumers of /health, and they want different things.
The first is Kubernetes probes. The kubelet uses liveness to decide whether to restart a container, and readiness to decide whether the pod should receive traffic. (The probe definitions came from a shared chart I couldn't see, so I can't tell you which path they hit, but /health is the obvious candidate.) The second is synthetic monitoring. External uptime tests hit the public /health URL of production services and assert on status code, response time, and a substring of the body (here, the string ok). Failures notify the on-call through the incident tool. The third is humans: someone curls it during an incident to answer "is it up?".
An endpoint that unconditionally returns ok gives all three the same dangerous answer: yes. If the database is unreachable, the pod stays "ready", the load balancer keeps sending traffic, the synthetic test stays green, and the first signal of the outage is customers.
Liveness and readiness are different questions
The classic mistake is to treat them as the same check.
Liveness asks: is this process wedged beyond recovery? It should be cheap and should not depend on downstream systems. If your liveness probe checks the database and the database goes down, Kubernetes will restart every healthy pod in your fleet at once, turning a database problem into a full outage.
Readiness asks: can this instance serve traffic right now? This one should check what the service needs to do its job: its database, its cache, a required upstream. Failing readiness takes the pod out of rotation without restarting it, which is exactly the right reaction to a dependency blip.
So an always-OK endpoint is, ironically, a reasonable liveness check. It is a bad readiness check. The mistake is serving both from one route.
Kubernetes also has a third probe, the startup probe, for services that take a while to boot. While it is configured and has not yet succeeded, the kubelet holds off liveness and readiness checks, so a slow start isn't mistaken for a wedged process.
Liveness restarts, readiness removes from rotation, and only readiness should look at dependencies
The library existed, adoption didn't
What I found interesting was that the platform already had a small shared library for the readiness half. It wraps a third-party health package and adds checks for Elasticsearch, so a service can register named dependencies and expose them on a status endpoint. The usage below follows its README (details simplified):
h, _ := health.New(health.WithComponent(health.Component{Name: "orders-api", Version: version}))
h.Register(health.Config{
Name: "mysql",
Timeout: 2 * time.Second,
SkipOnErr: false,
Check: healthMysql.New(healthMysql.Config{DSN: dsn}),
})
h.Register(health.Config{
Name: "search",
Check: healthElasticsearch.New(healthElasticsearch.Config{DSN: esURL}),
})
http.Handle("/status", h.Handler())
Among the repositories I explored, exactly one service used it for real, registering MySQL, Redis and Elasticsearch. Most others shipped the hand-written ok handler. The library wasn't the problem. The path of least resistance was. A copy-paste /health handler is five lines. A proper one needs a decision about which dependencies matter.
A pattern that tells the truth
If I were standardising this across a Go fleet, I'd ship two endpoints from the shared web-server library, wired by default, so a service gets them without writing a line.
// /livez: process is up. No dependencies. Cheap.
mux.HandleFunc("/livez", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
// /readyz: can I do useful work right now?
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
failed := map[string]string{}
for name, check := range checks { // registered by the service
if err := check(ctx); err != nil {
failed[name] = err.Error()
}
}
if len(failed) > 0 {
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{"status": "unavailable", "failed": failed})
return
}
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
})
A few design rules make this safer than it looks:
- Every check gets a short timeout. A readiness endpoint that hangs for 30 seconds because a dependency is slow is worse than one that fails fast.
- Separate "required" from "degraded". A missing optional upstream (say, a notification provider) shouldn't make the service unready. Report it, but don't fail on it.
- Don't leak internals publicly. The failure detail is for operators. If the endpoint is reachable from the internet, return the status code and nothing else, and put details in logs.
- Cache briefly. If probes and monitors hit it every few seconds across many replicas, a one-second cache stops your health check from becoming load on your database.
Make the monitors honest too
Fixing the handler is half the job. The external synthetic test also has to assert on something meaningful. Asserting body contains "ok" is satisfied by the old lie and by the new truth alike, which is fine, but it means the test only catches a failing check if the status code changes. Assert on the status code first (200), on latency second, and on the body last. And point the synthetic test at the readiness route, not at a route that can't fail.
Checklist
- Liveness: no downstream calls, cheap, restart-worthy failures only.
- Readiness: checks the dependencies this instance needs, each with its own timeout.
- Synthetic monitors target readiness through the public path, so they exercise the same ingress real users do.
- The shared web library provides both routes by default, so correctness is the easy path.
Wrapping up
A health check that cannot fail is not a health check. Liveness and readiness answer different questions, so give your services a cheap liveness route that never touches dependencies and an honest readiness route that does, with short timeouts. Point your synthetic monitors at the honest one, and ship both from the shared library so the correct setup is also the default.