K8s Liveness vs Readiness — Stop Restarting Healthy Pods

The rollout looks fine. Then pods flap: CrashLoopBackOff, restarts climbing, yet application logs show the process still booting JVM/classloading/migrating. Someone pointed the liveness probe at the same /health endpoint used for readiness, with a short initialDelaySeconds. Kubernetes did its job: it killed a process that was slow, not dead. You turned cold start into a restart storm.

Three probes, three jobs

Probe	Failure means	kubelet action
Startup	Container not finished starting	Disable liveness/readiness until success (or kill if failureThreshold exceeded)
Readiness	Not ready for traffic	Remove from Service endpoints
Liveness	Stuck / deadlocked	Restart container

Readiness failures are soft. Liveness failures are hard. Using liveness for “dependencies not ready yet” is how you punish the pod for Redis being briefly slow.

The anti-pattern

livenessProbe:
  httpGet:
    path: /health   # checks DB + Redis + self
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /health   # identical
  periodSeconds: 5

Deep health on liveness: if Postgres blips, kubelet restarts every pod, stampedes the DB on reconnect, and prolongs the outage. Deep health on readiness alone: pods drop from the Service until Postgres returns — usually what you want.

A safer split

startupProbe:
  httpGet:
    path: /livez
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /livez    # process up; no dependency checks
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz   # DB ping, migrations done, warm caches
  periodSeconds: 5

Application sketch:

app.get("/livez", (_req, res) => res.status(200).send("ok"));
app.get("/readyz", async (_req, res) => {
  try {
    await db.query("select 1");
    res.status(200).send("ready");
  } catch {
    res.status(503).send("not ready");
  }
});

Use startupProbe for slow boots (Spring, Prisma migrate-on-start, webpack in bad images). Until startup succeeds, liveness will not kill you mid-init. Size failureThreshold * periodSeconds to your p99 cold start plus margin — not to your best-case laptop boot.

Tuning numbers that matter

• initialDelaySeconds — legacy crutch; prefer startupProbe so delay is adaptive.
• timeoutSeconds — if the handler waits on a lock, short timeouts flap readiness; long timeouts hold kubelet worker capacity.
• failureThreshold * periodSeconds — total patience before action. Liveness patience should be longer than a GC pause, shorter than “forever deadlocked.”
• successThreshold — readiness can require 2 successes before rejoin to avoid flapping under partial dependency recovery.
• Probe storm — ten pods × aggressive periodSeconds × heavy /readyz can DoS your own database. Cache dependency checks for a second or use a shared pool.

Exec and gRPC probes

HTTP is not mandatory. exec probes that run curl inside the container add process overhead and shell quirks; prefer a tiny native HTTP handler when you can. gRPC health checks are excellent when your service already speaks gRPC — still keep the liveness check shallow. Never exec a migration or a full test suite as a probe.

Debugging restart loops

kubectl describe pod <pod>   # Last State, probe failures
kubectl get events --field-selector involvedObject.name=<pod>
kubectl logs <pod> --previous

Look for Liveness probe failed with app logs still printing “listening on :8080” a few seconds later — delay/threshold too aggressive. Look for restarts aligned with dependency outages — liveness too deep. If the container exits with a real OOM or panic, probes are a distraction; fix the crash, then re-check probe config.

During incidents, temporarily lengthening liveness thresholds can stop the bleeding while you ship a proper /livez — but treat that as a hotfix, not the steady state.

Policy for the team

1. Liveness = “this process should be killed.”
2. Readiness = “this process should not get traffic.”
3. Startup = “this process is still allowed to boot.”
4. Never gate liveness on external dependencies.
5. Load-test probe endpoints; a /readyz that opens a new DB connection per check can itself exhaust the pool.
6. Document the expected cold-start budget next to the Helm values so the next chart copy-paste does not set initialDelaySeconds: 5 again.

Healthy pods do not need restarts. Stop asking liveness to answer readiness questions — and give slow boots a startup probe instead of a death sentence.