The HPA algorithm in one line and the missing resource request that silently disables it, VPA as a measuring tool before it is an actuator, why the two fight on CPU, and the single pod that pins a node against every scale-down.
Count, size and capacity. They interact, and two of them actively conflict.
The constraint row is what makes autoscaling fail quietly: a PodDisruptionBudget or a node-group maximum stops the whole chain without anything reporting an error.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
What each one moves, and the one thing that stops each from working.
The Horizontal Pod Autoscaler adjusts replica count. The core calculation is one line:
desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue))
At 4 replicas averaging 80% CPU against a 50% target: ceil(4 × 1.6) = 7. There
is no mystery in the arithmetic — the surprises are all in the inputs and the damping.
requests.cpu has no denominator, the
HPA reports <unknown>, and it never scales. This is the single most common
cause.kubectl top
failing and the HPA showing <unknown> together point here.For a queue consumer, the honest metric is queue depth, not CPU: a worker blocked on I/O sits at 5% CPU with a backlog of 50,000. That is what custom metrics (via the Prometheus adapter) and external metrics (via KEDA) exist for, and it is usually the difference between autoscaling that works and autoscaling that is decorative.
The Vertical Pod Autoscaler adjusts requests and limits rather than replica count. It has three parts, and they are separable — which matters more than the docs make obvious:
In updateMode: "Off" you get only the recommender: no evictions, just a
recommendation you can read. That is the mode most clusters should start in — it turns
"what should this request be?" from guesswork into a measured number, with no runtime risk.
Both react to CPU. VPA raises the request; raising the request lowers CPU-as-a-percentage-of-request; the HPA sees lower utilisation and scales in; fewer pods means more load each; VPA raises requests again. The documented rule is simple: do not run both on CPU or memory for the same workload. HPA on a custom metric with VPA on CPU is fine.
The Cluster Autoscaler adds nodes when pods are Pending for want of capacity, and
removes nodes that have been underused for a while. Scale-up is straightforward. Scale-down is
where the money leaks, because a single pod can pin a whole node indefinitely.
cluster-autoscaler.kubernetes.io/safe-to-evict: "false".The autoscaler logs its reason for every node it declined to remove, which turns "why are we paying for twelve nodes at 20% utilisation" into a five-minute answer rather than a theory.
Karpenter takes a different approach — instead of scaling fixed node groups it provisions the instance shape that fits the pending pods, and consolidates aggressively. On AWS it is usually the better answer now; the mental model shifts from "which group do I grow" to "what shape does this workload need".
A single-replica Deployment with minAvailable: 1 can never be evicted, so its node can never be drained — not by the autoscaler, and not by an upgrade either. Two replicas, or maxUnavailable: 1 instead, costs less than the node you are pinning.
Autoscaling failures are quiet: nothing errors, the thing just does not scale. Each of the three has one dominant cause, so check that first rather than reading configuration.
| Symptom | Most likely cause | Check |
|---|---|---|
HPA target <unknown> | No resource requests, or no metrics-server | kubectl top pods; then the container's resources.requests |
| HPA at max, still slow | Bottleneck is downstream, not replicas | Database connections, a queue, a rate limit |
| Scales up, never down | 300s stabilization, or one busy replica in the average | kubectl describe hpa — the conditions explain each decision |
| Replicas flapping | scaleUp and scaleDown both aggressive | Set a behavior block |
| Pods Pending, no new nodes | Node group at max, or no shape fits | Cluster Autoscaler logs; the group's max size |
| Nodes never removed | A pod pinning each one | cluster-autoscaler-status ConfigMap |
| VPA and HPA both acting | Both on CPU — they fight by design | Move HPA to a custom metric, or VPA to Off |
kubectl describe hpa carries conditions — AbleToScale,
ScalingActive, ScalingLimited — each with a message stating exactly
what the controller decided and why. It is the autoscaling equivalent of the scheduler's Events
line, and equally underused.
| Command | What it answers |
|---|---|
kubectl get hpa -A | Targets, current versus desired, everywhere |
kubectl describe hpa <x> | The controller's own reasoning, in conditions |
kubectl top pods / nodes | Whether metrics-server works at all |
kubectl set resources deploy/<x> --requests=cpu=200m | Give the HPA a denominator |
kubectl describe vpa <x> | Recommended requests, measured not guessed |
kubectl logs -n kube-system deploy/cluster-autoscaler | Why a node was not removed |
kubectl get cm -n kube-system cluster-autoscaler-status -o yaml | Node group state at a glance |
kubectl get pdb -A | What blocks eviction, scale-down and upgrades |
kubectl get pods --field-selector status.phase=Pending -A | What should be triggering scale-up |
kubectl annotate pod <p> cluster-autoscaler.kubernetes.io/safe-to-evict=true | Unpin a node |