ReplicaSets and why a stuck rollout is legible, DaemonSets and the update strategy that silently never rolls, Jobs and CronJobs that pile up on their own schedule, and the liveness probe that turns a dependency blip into an outage.
You declare what you want; a chain of controllers argues reality towards it. Knowing which link owns what is most of debugging.
The probes at the bottom are not part of starting the pod — they decide, continuously, whether it gets traffic and whether it gets killed. That is why they cause outages disproportionate to their size.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
What each controller guarantees, and the field in each that causes the incident.
You write a Deployment; the deployment controller creates a
ReplicaSet per revision and scales them against each other. The Deployment is
the rollout strategy. The ReplicaSet is the thing that actually keeps N pods alive.
That two-layer split is why kubectl get rs is one of the most informative
commands during a bad deploy. A rollout that is stuck shows it plainly: the new ReplicaSet has
desired=3 current=1 ready=0 while the old one still has 3 ready. The Deployment just says
"progressing".
revisionHistoryLimit (default 10) decides how many, and setting it to 0 means you
cannot rollout undo at all.spec.selector on a live
Deployment and the API rejects it — you delete and recreate, which is a real outage if you did
not plan it.A DaemonSet runs one pod per matching node, and adds one automatically when a
node joins. It is the shape for anything that is about the node: log collectors, CNI
agents, node exporters, storage drivers.
Two behaviours differ from every other workload:
node.kubernetes.io/not-ready, unreachable, disk-pressure
and others automatically — because a node agent that evacuates itself the moment the node is
unhealthy is useless precisely when you need it.kubectl drain. Drain skips DaemonSet pods, which
is why --ignore-daemonsets exists and why you pass it every time.updateStrategy: OnDelete means the DaemonSet will never roll a
new image — it waits for you to delete each pod by hand. It is a legitimate choice for a
storage driver you want to reboot deliberately, and a silent "our agent has been on the old
version for eight months" for everything else.
A Job runs pods until a target number succeed. A
CronJob creates Jobs on a schedule. Everything that goes wrong with them comes from
four fields.
| Field | Means | Gets you when |
|---|---|---|
completions | How many successes end the Job | Unset = 1, so a "batch" silently processes once |
parallelism | How many pods at once | Unset = 1; your 6-hour job could have been 20 minutes |
backoffLimit | Retries before Failed | Default 6, with exponential backoff to 6 minutes — a fast-failing job takes ~20 min to give up |
activeDeadlineSeconds | Wall-clock cap | Unset = a hung job runs forever, holding its resources |
The default is Allow. If a run takes longer than the interval, the next one
starts anyway — and on a five-minute schedule with a run that has started taking eight minutes,
you accumulate overlapping jobs until something saturates. Forbid skips the new run;
Replace kills the old one. Both are usually more correct than the default.
Also: startingDeadlineSeconds. If the controller is down past that window the
run is skipped, not queued — and if it is unset and the controller was down a long
time, the CronJob can fire every missed run at once on recovery.
Three probes, three jobs, and conflating them is the most common self-inflicted Kubernetes outage there is:
| Probe | On failure | Question it answers |
|---|---|---|
| startupProbe | Keeps the others waiting | Has it finished booting? |
| readinessProbe | Removed from Endpoints | Should it get traffic right now? |
| livenessProbe | Container is killed | Is it wedged beyond recovery? |
Point liveness at a /health that checks the database, and the moment the database
has a bad thirty seconds every replica fails liveness and gets killed — simultaneously. You have
converted a recoverable dependency blip into a full restart storm, and the restarts add load to
the thing that was already struggling.
The rule that avoids it: liveness checks only what a restart can fix. A deadlocked event loop, yes. A database you do not own, never — that belongs in readiness, where failing simply takes the pod out of rotation until the dependency returns.
A long initialDelaySeconds on liveness delays detection for the whole life of the
pod. A startupProbe with a generous failureThreshold gives a slow JVM
five minutes to boot and then hands over to a tight liveness probe.
Many replicas restarting within the same few seconds, all with Liveness probe failed and exit 137, while the application logs show nothing wrong — that is a dependency wobble being amplified by liveness. Move the dependency check to readiness before you tune any timeouts.
Nearly every workload symptom is one of four states, and the state tells you which object to look at. Guessing from the Deployment alone wastes the first ten minutes.
| Pod state | What it means | Look at |
|---|---|---|
Pending | Never scheduled — no node fits, or quota refused it | kubectl describe pod, the Events tail names the failed predicate |
ContainerCreating > 2 min | Image pull, volume attach or CNI | Events; then the node's kubelet log |
CrashLoopBackOff | It starts and exits | kubectl logs -p — the PREVIOUS container is the one that failed |
Running but 0/1 | Readiness failing — no traffic reaches it | describe for the probe message; the Service has no endpoint |
Running, restarts climbing | Liveness killing it, or OOM | lastState.terminated.reason — Error vs OOMKilled |
Completed but rerun | A Job that succeeded and the CronJob fired again | concurrencyPolicy and the job history |
kubectl logs on a crash-looping pod shows the container that is starting now and usually prints nothing useful. kubectl logs -p shows the one that already died, which is the one holding the stack trace.
| Command | What it answers |
|---|---|
kubectl get rs -l app=<x> | Which revision is stuck, old versus new |
kubectl rollout status deploy/<x> --timeout=5m | Block until it lands or fails |
kubectl rollout history deploy/<x> | Revisions available to roll back to |
kubectl rollout undo deploy/<x> --to-revision=N | Go back to a specific one |
kubectl logs -p <pod> | The container that actually crashed |
kubectl get pod <p> -o jsonpath='{.status.containerStatuses[0].lastState}' | Exit code and reason — Error vs OOMKilled |
kubectl get ds -A | Desired versus ready, per node agent |
kubectl get jobs -A --sort-by=.metadata.creationTimestamp | CronJob pile-ups |
kubectl get cronjob -A | Schedules, last run and suspension |
kubectl describe pod <p> | Scheduling, probe and image-pull failures |
kubectl get events -A --sort-by=.lastTimestamp | What just happened — expires in an hour |
kubectl debug <pod> -it --image=busybox --target=<c> | A shell beside a container with no shell |