Service Mesh Fundamentals

A service mesh is two things: a data plane of proxies — one per pod, or one per node — that every request passes through, and a control plane that configures them. The application is not modified and usually does not know the proxy is there.

What you get, and the reason it is worth the weight: mTLS everywhere without touching application code, retries and timeouts and circuit breaking as policy rather than as library code in six languages, traffic splitting for canaries, and uniform golden-signal telemetry for every hop in the system.

The honest costs

• Latency. Two extra proxy hops per request. Single-digit milliseconds, but not zero, and it compounds across a deep call graph.
• Resources. A sidecar per pod. At a thousand pods that is a meaningful slice of the cluster spent on proxies.
• A new failure domain. The mesh can now break your traffic in ways that have nothing to do with your code, and debugging moves from "read the app log" to "read the proxy config".

When you do not need one

Ten services, one language, one team: a shared HTTP client library with retries and timeouts gets most of the benefit at none of the cost. The mesh wins when the count of services and languages is high enough that "put it in the library" stops being one change and becomes six.

Ambient / sidecar-less modes change this calculation. Istio's ambient mode moves mTLS and L4 telemetry to a per-node component and makes the L7 proxy opt-in per namespace, which removes the per-pod sidecar cost for workloads that only need encryption and metrics.

Istio, Gloo, Consul, Gateway API implementations and most of the ecosystem use Envoy as the data plane. Understanding four of its objects makes mesh debugging tractable, because every mesh CRD ultimately renders into them:

xDS is the protocol that streams these from the control plane to every proxy. When someone says "the config has not propagated", they mean a proxy's xDS state is stale — and that is directly observable rather than a matter of opinion.

The debugging move that skips all the guessing

Do not reason about what the CRDs should have produced. Ask the proxy what it actually has.

Kubernetes service discovery is already complete: CoreDNS resolves svc.ns.svc.cluster.local to a ClusterIP, and kube-proxy load-balances to the EndpointSlice. A mesh does not replace that — it intercepts after it.

What changes: the proxy has the full endpoint list rather than an iptables rule, so it can do things kube-proxy cannot — least-request instead of random, locality-aware routing that prefers same-zone endpoints and fails over to another zone only when the local ones are unhealthy, outlier detection that ejects an endpoint returning 5xx, and per-request retries with a budget.

Locality routing is usually the biggest single win

Cross-zone traffic costs money and latency in every cloud. With zone labels on the nodes and locality load balancing on, same-zone requests stay in-zone — often a double-digit percentage of inter-AZ transfer removed for one config change.

The decision is rarely about capability, because both do the core job. It is about whether you want the configurability and will staff for it, or want the smallest thing that provides mTLS, retries and golden signals.

Gateway API is where both are heading

The Kubernetes Gateway API is the successor to Ingress and now has a service-mesh profile (GAMMA). Both Istio and Linkerd implement it. For new configuration it is worth preferring Gateway API resources over vendor CRDs where they cover your case — the config outlives the mesh you picked.