K8s Observability Stack · Platform Ops

Deep-Dive

OBSERVABILITY REFERENCE

Click any pillar to explore concepts, queries, and production-tested guidance.

📊

How Prometheus Actually Collects Data

Prometheus scrapes — it doesn't receive pushes. Every target exposes a /metrics HTTP endpoint that gets polled on an interval.

Must Know

Must Know▾

ServiceMonitor (Prometheus Operator)

yaml

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

spec:

selector: {matchLabels: {app: billing-api}}

endpoints:

- port: metrics

interval: 30s

path: /metrics

Core Exporters

🔴node-exporter: host-level CPU, memory, disk, network — one per node

🔴kube-state-metrics: object state — pod counts, deployment status, PVC phase

🟠cAdvisor: built into kubelet — per-container resource usage

Metric Types

Counter — only goes up (request count, errors total)

Gauge — goes up or down (memory usage, queue depth)

Histogram — buckets of observations (request latency)

🔎

PromQL Patterns Worth Memorizing

A handful of query shapes cover most dashboards and alerts you'll ever write.

Important

Important▾

PromQL

# Request rate over 5 minutes

rate(http_requests_total[5m])

# p99 latency from a histogram

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Error ratio

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Pods not Ready right now

kube_pod_status_ready{condition="false"} == 1

📈

Long-Term Storage & Grafana

Prometheus itself is not built for years of retention — that's where Thanos/Mimir and Grafana's dashboarding layer come in.

Scaling

Recommended▾

Retention Problem

🔵Local Prometheus TSDB is fine for 15–30 days; longer needs remote storage

🔵Thanos or Mimir add object-storage-backed long-term retention + global query view across clusters

Grafana Practices

Use variables ($namespace, $pod) — one dashboard, many contexts

Version dashboards as JSON in git, provision via ConfigMap or the Grafana operator

Keep one "golden signals" dashboard per service: latency, traffic, errors, saturation

⚖️

Loki vs the ELK Stack

Loki indexes only labels and stores log content as compressed chunks. Elasticsearch full-text-indexes every field. That single design choice explains most of the cost/capability trade-off.

Must Know

Must Know▾

Trait	Loki	ELK (Elasticsearch)
Indexing	Labels only — cheap	Full-text on every field — expensive
Query language	LogQL	KQL / Lucene / DSL
Storage cost	Low (object storage, chunks)	High (inverted index overhead)
Best fit	Kubernetes-native, Grafana shops, cost-sensitive	Complex free-text search, security/SIEM use cases
Native to	Grafana ecosystem	Kibana

🚚

Shipping Logs Out of Pods

Neither stack reads pod stdout directly in production — a collector agent runs on every node and ships it.

Important

Important▾

Collector Options

🟠Promtail — Loki's own DaemonSet agent, tails /var/log/pods, attaches pod/namespace labels

🟠Fluent Bit — lightweight, ships to either Loki, Elasticsearch, or both

🔵Logstash — heavier, does parsing/enrichment/filtering in the pipeline itself

Structured logging (application side)

json

{"level":"error","service":"billing-api",

"trace_id":"a1b2c3","msg":"payment gateway timeout"}

🔍

LogQL Query Patterns

LogQL wraps a label selector — same syntax family as PromQL — around a text/metric pipeline.

Query

Recommended▾

LogQL

# All error lines from the billing-api pods

{app="billing-api"} |= "error"

# Parse JSON and filter on a field

{app="billing-api"} | json | level="error"

# Rate of error lines over time (metric from logs)

sum(rate({app="billing-api"} |= "error" [5m]))

🕸️

Spans, Trace Context & Propagation

A trace is a tree of spans. Each span represents one unit of work; context propagation is what stitches spans from different services into the same trace.

Must Know

Must Know▾

Core Terms

🔴Trace ID: one ID per end-to-end request

🔴Span: one operation — an HTTP call, a DB query — with start/end time

🟠Parent/child spans: build the call tree — CMP calling NGW shows up as a child span

Propagation header (W3C Trace Context)

http

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

# version - trace-id - parent-span-id - flags

🛰️

OpenTelemetry Collector

A vendor-neutral pipeline: receive traces (and metrics/logs) from apps, process them, export to Jaeger, Tempo, or any OTLP-compatible backend.

Important

Important▾

otel-collector-config.yaml

yaml

receivers: {otlp: {protocols: {grpc: {}, http: {}}}}

processors: {batch: {}, memory_limiter: {limit_mib: 512}}

exporters: {jaeger: {endpoint: "jaeger-collector:14250"}}

service: {pipelines: {traces: {receivers: [otlp], processors: [batch], exporters: [jaeger]}}}

🎚️

Sampling Strategy

Tracing every request at scale is expensive to store and query. Sampling decides which traces survive.

Cost Control

Recommended▾

Sampling Strategies

🔵Head-based: decide at trace start (e.g. 10% of traces) — cheap, simple, might miss rare errors

🔵Tail-based: decide after seeing the whole trace — always keep errors/slow traces, drop the boring fast ones

Rule of Thumb

Always sample 100% of error traces regardless of overall rate

Start with head-based at 5–10% for high-volume services, move to tail-based once volume justifies it

🚨

Alertmanager Routing

Prometheus fires alerts based on rules; Alertmanager decides who gets notified, how, and whether to group or silence them.

Must Know

Must Know▾

alert-rule.yaml + routing

yaml

- alert: HighErrorRate

expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05

for: 10m

labels: {severity: critical, team: billing-ops}

annotations: {summary: "Error rate above 5% for 10m"}

🔥

SLOs & Error Budget Burn Rate

A static threshold alert ("errors > 5%") tells you nothing about urgency. Burn-rate alerts ask: at this rate, how soon do we blow the monthly budget?

Important

Important▾

The Idea

🟠SLO: 99.9% availability = 43m of allowed downtime per month ("error budget")

🟠Fast burn (budget gone in hours) → page immediately

🔵Slow burn (budget gone in days) → ticket, not a page

Practical Guidance

Define SLOs on user-facing symptoms (latency, error rate) — not internal metrics like CPU

Use multi-window burn-rate rules (e.g. 1h + 5m) to catch both fast and slow burns

🔕

Reducing Alert Fatigue

The fastest way to lose an on-call rotation's trust is a pager that fires for things nobody needs to act on.

Operations

Recommended▾

Every alert needs a linked runbook — no page without a "here's what to do"

Group related alerts (group_by: [alertname, cluster]) instead of one page per pod

Review fired-but-not-actioned alerts monthly — delete or downgrade anything nobody acts on

🧭

Why Not Just Grafana?

Grafana visualizes whatever's in Prometheus/Loki/Jaeger. It doesn't know your org's billing pipelines, VIL-CMP deployment topology, or which team owns which alert.

Must Know

Must Know▾

What a Platform Layer Adds

🔴Domain context: maps raw metrics to business entities (CMP module, NGW gateway, billing run)

🟠Ownership routing: knows which team owns which service without hardcoding it into every alert rule

🔵A single API surface other internal tools can query instead of hitting Prometheus/Loki directly

Typical Flask Layer Shape

REST endpoints wrap PromQL/LogQL queries — frontend never talks to Prometheus directly

A background worker polls Alertmanager and persists incident history to a DB for trend reporting

Auth/RBAC sits in front — engineers only see the services their team owns

🔗

Cross-Signal Correlation

The highest-value feature of a unified platform: jumping from a metric spike straight to the logs and trace that explain it, with no manual timestamp matching.

Important

Important▾

Propagate the same trace_id into structured logs — makes "metric spike → logs → trace" a single click, not three tabs

Store an incident timeline: alert fired → first log match → resolution — this is what turns into postmortem data automatically

Expose a single /incidents/<id> view that pulls metrics graph + log window + trace waterfall together

🛠️

Build vs Buy Trade-off

A commercial APM (Datadog, New Relic) gives you this out of the box — at a price and with less control over data residency and custom domain modeling.

Decision

Recommended▾

Factor	Internal Platform (OpsCore-style)	Commercial APM
Cost model	Infra + engineering time	Per-host / per-GB licensing
Domain fit	Exact match to internal services & ownership	Generic, needs custom tagging
Data residency	Fully on-prem/internal	Vendor cloud, unless self-hosted tier
Time to value	Slower — you build it	Fast — mostly configuration

OBSERVABILITY
STACK

THE THREE PILLARS (+ 2)

OBSERVABILITY REFERENCE

COMMAND CHEATSHEET

OBSERVABILITY STACK

THE THREE PILLARS (+ 2)

OBSERVABILITY REFERENCE

COMMAND CHEATSHEET

OBSERVABILITY
STACK