Monitoring that is half switched off by default, Loki queries that return before they time out, the four security gates and which one refused you, and the upgrade that stops on one PodDisruptionBudget.
Two Prometheus instances behind one query endpoint — and the one that scrapes your applications does not exist until you enable it.
Thanos Querier is what the console and your dashboards talk to; it federates both instances. That is why platform metrics appear immediately and yours do not, from the same URL.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
The three things you are asked for after every incident, and the default that quietly prevents each one.
OpenShift ships a complete monitoring stack, but it is deliberately split in two:
openshift-monitoring) — scrapes the
cluster itself. Always on, not configurable by you, and it will not scrape your namespaces.openshift-user-workload-monitoring)
— scrapes your applications. Disabled by default.This is the single most common "our metrics don't work" cause on OpenShift. A team writes a
ServiceMonitor, applies it, sees no error — because a ServiceMonitor with nothing
watching it is a perfectly valid object — and the metrics never appear. Nothing is broken;
the second Prometheus simply does not exist yet.
spec.endpoints[].port is the Service's port name, not its number. An unnamed port cannot be selected at all, and a mismatched name matches nothing — silently, with the ServiceMonitor still showing as healthy. If the target list is empty, check the port name before anything else.
Alerts are PrometheusRule objects. Platform rules ship with the cluster; yours
live in your namespace and are picked up by user-workload Prometheus.
The design question is not "what threshold" but what makes a human get out of bed. A CPU-above-80% alert fires constantly and teaches people to ignore the pager. An SLO burn-rate alert fires when you are consuming the error budget fast enough to miss the objective — which is the same thing the user is experiencing.
Multi-window burn rate is the standard shape: a fast window catches sudden severe breakage, a slow window catches sustained mild breakage, and requiring both to be hot suppresses the one-minute blip that recovered on its own.
OpenShift logging moved from Elasticsearch/Kibana to Loki with the console as the UI. The practical differences matter:
application, infrastructure,
audit — with separate access. Audit logs are not collected by default.The collector is Vector, configured by a ClusterLogForwarder. That object is
also how you ship elsewhere — Splunk, Kafka, an external Loki — and you can forward and store
locally at the same time.
Four independent gates sit between a request and a running workload. Reading the refusal tells you which one, and they are not interchangeable:
| Gate | Answers | Refusal looks like |
|---|---|---|
| Authentication | Who are you? | Unauthorized, 401 |
| RBAC | May you perform this verb on this resource? | Forbidden: User "x" cannot get pods |
| SCC | May this POD have the privileges it asks for? | unable to validate against any security context constraint |
| NetworkPolicy | May this packet arrive? | Nothing. Timeout. |
The last row is the operationally important one: NetworkPolicy has no error message. Every other layer tells you it refused. A policy drop is indistinguishable from a dead backend, which is why it is worth proving or excluding early rather than late.
The API server writes audit events for every request. They are on the masters at /var/log/kube-apiserver/ and reachable with oc adm node-logs --role=master --path=kube-apiserver/audit.log. They are not forwarded to Loki unless the ClusterLogForwarder names the audit input — so the one log you need after a security question is usually the one nobody enabled.
What the cluster already knows before you start, and where the hours go.
The CVO applies manifests in a fixed order and stops at the first that will not go ready, so a stalled upgrade always names its blocker in the Progressing message.
Channels decide which versions are offered:
Before any upgrade, the cluster has already formed an opinion. The
Upgradeable condition on ClusterVersion is set to False
by any operator that knows the upgrade will hurt — most often because a deprecated API
is still in use and the next minor removes it. Upgrading anyway is how a workload
disappears mid-upgrade.
The control plane phase is fast and mostly invisible. The node phase is where the hours go: the MCO cordons a node, drains it, applies the new OS config, reboots it, uncordons, and moves to the next — one at a time per pool by default.
Draining is where upgrades stop, and the cause is nearly always one of two things:
minAvailable: 1 can never be evicted. The drain retries forever, politely,
and the upgrade sits at the same percentage for hours.Both are the cluster protecting availability exactly as instructed. The fix is the PDB or the workload, not forcing the drain.
At the default of 1, a 100-node pool is 100 sequential reboots — easily a working day. Raising maxUnavailable on the MachineConfigPool to 3 or 5 cuts that proportionally, provided your workloads have enough replicas and spread to survive losing that many nodes at once. Check PDBs and topology spread before you raise it, not after.
Three Red Hat operators cover the security questions that arrive as audit findings:
ComplianceCheckResult objects. Many findings ship with an
auto-remediation you can apply as a MachineConfig.Independently: image signature verification. A cluster that will pull any image from anywhere has no supply chain guarantee, and this is a cluster-wide policy, not a per-workload one.
Two different things get called "backup" and conflating them is how a DR test fails:
| etcd snapshot | Application backup (OADP/Velero) | |
|---|---|---|
| Contains | Every API object | Selected namespaces + PV data |
| Restores | The whole cluster, to that instant | Namespaces, into this or another cluster |
| Granularity | All or nothing | Per namespace, per label |
| PV contents | No | Yes, via snapshots or restic |
| Use for | Control-plane disaster | Namespace deleted, migration, real DR |
An etcd restore is disruptive by design: you stop the control plane, restore on one master, and rebuild the others from it. It is the right tool for "we lost quorum", and the wrong tool for "someone deleted the prod namespace" — for which OADP restores in minutes without touching anything else.
An etcd restore rolls every node and takes a cluster down for the duration. If it has never been rehearsed, the first rehearsal will be during an outage, under time pressure, by someone reading the docs for the first time. Schedule it on a cluster you can afford to break, and time it — the number you get is your real RTO.
Operations failures are mostly absences: a metric that never arrived, a log that was never collected, an alert that was silenced, an upgrade that stopped. Absences have no error message, so each one needs a positive check rather than a glance at a dashboard.
| Symptom | The absence behind it | Positive check |
|---|---|---|
| App metrics missing | User workload monitoring never enabled | oc -n openshift-user-workload-monitoring get pods |
| ServiceMonitor exists, no data | Port name mismatch — it matched nothing | Prometheus /api/v1/targets, not the ServiceMonitor |
| Alert never fired | An unexpired silence, or the rule is in the wrong namespace | amtool silence query; oc get prometheusrule -A |
| Logs stop at a date | LokiStack retention, or storage pressure dropping streams | oc get lokistack -o yaml; the ingester pod's logs |
| No audit trail for an incident | ClusterLogForwarder never included the audit input | oc adm node-logs --role=master --path=kube-apiserver/audit.log |
| Upgrade at the same % for hours | A drain blocked by a PDB or a bare pod | oc get mcp -o yaml — the Degraded message names the pod |
| Upgrade refuses to start | Upgradeable=False — a removed API is still in use | oc get apirequestcount names the caller |
| Node rebooted unexpectedly | MCO applying a MachineConfig, as designed | oc get mcp; the node's machineconfiguration annotations |
Prometheus keeps platform metrics for roughly 15 days and Loki for whatever retention you set. Both are shorter than most post-incident reviews take to schedule, so capture the query results during the incident rather than planning to look later.
Kubernetes Events have a default TTL of 60 minutes. They hold the scheduling failures, admission rejections, image pull errors and probe failures that explain the incident — and they are gone before most postmortems begin. Capturing them is the first command of an incident, not the last.
| Command | What it answers |
|---|---|
oc -n openshift-user-workload-monitoring get pods | Whether your metrics are being scraped at all |
oc get servicemonitor,podmonitor -A | What has asked to be scraped |
oc get prometheusrule -A | Every alert rule, platform and yours |
amtool silence query | The silence that stopped the page |
oc get lokistack -n openshift-logging -o yaml | Retention and storage sizing |
oc get clusterlogforwarder -A | Which log tenants are collected and where they go |
oc adm node-logs --role=master --path=kube-apiserver/audit.log | Who did what to the API |
oc auth can-i --list -n <ns> --as=<user> | Effective RBAC, resolved |
oc adm policy who-can <verb> <res> | The reverse RBAC question |
oc adm upgrade | Which versions are on offer right now |
oc get apirequestcount | Who is still calling a deprecated API |
oc get pdb -A | The budget that will block the next drain |
oc get compliancecheckresult -n openshift-compliance | Current compliance failures |
oc get backup -n openshift-adp | Whether the application backup actually ran |
oc get events -A --sort-by=.lastTimestamp | What just happened — expires in an hour |