Observability on OpenShift: Why the Built-In Stack Isn’t a Strategy

Most OpenShift clusters arrive with observability already switched on. Prometheus is running. Alertmanager is running. There are dashboards in the console. For the first six months, nobody questions any of it.

Then the cluster grows. Retention turns out to be three days when the compliance answer needed to be ninety. A cardinality spike quietly OOM-kills a Prometheus pod during an incident, so the one window you needed to look at is the one window you don’t have. Nobody can say who owns the alerts, so the on-call rota starts muting channels.

None of that is an OpenShift defect. It’s the difference between monitoring shipped with a platform and an observability architecture designed for a specific estate. This post covers the shape of that difference — the components, the decisions, and where teams most often get hurt.

What actually ships in OpenShift 4.x

Worth being precise here, because “OpenShift has Prometheus” hides a lot of structure.

The Cluster Monitoring Operator owns the platform stack: a Prometheus pair, Thanos Querier, Alertmanager, node-exporter, kube-state-metrics and the console integration. It is opinionated by design. You configure it through a ConfigMap, not by editing its resources — anything you change directly gets reconciled away.

User workload monitoring is a separate, opt-in stack for your applications. It has its own Prometheus instance, its own Thanos component and its own retention and resource settings. This split is the single most misunderstood thing about OpenShift monitoring: platform and application telemetry are not one system, and tuning one does nothing for the other.

Beyond that, the current generation of observability capability arrives as operators:

• Cluster Observability Operator (COO) — the umbrella for newer monitoring capability, including the UI plugins that surface logs and traces in the console
• Loki Operator — LokiStack for log aggregation, object-storage backed
• Tempo Operator and OpenTelemetry Operator — distributed tracing and collection/pipeline management
• Network Observability Operator — eBPF-based flow telemetry, which answers a category of question metrics simply cannot

All of these install through OLM. That matters more than it sounds: a Subscription gives you a channel, an approval strategy and a documented upgrade path that survives cluster upgrades. Helm charts and raw manifests give you a snowflake that someone has to remember to reconcile by hand at 4.16.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: loki-operator
  namespace: openshift-operators-redhat
spec:
  channel: stable-x.y
  installPlanApproval: Manual
  name: loki-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

The channel and approval strategy you choose there are a governance decision, not a copy-paste. Manual approval on a production cluster and automatic on a sandbox is a reasonable default. Getting the pinning strategy consistent across an estate of clusters is where the real work lives.

The four decisions that determine what this costs you

Every OpenShift observability build we’ve done comes down to the same four questions. Answer them early and the rest is implementation. Answer them late and you’re doing a migration.

1. Where does the data live, and for how long?

Default retention is short and PVC-backed. Object storage — S3, or ODF if you’re on-prem — changes the economics entirely and is a prerequisite for LokiStack and Tempo anyway. The decision is not “do we use object storage”; it’s what your retention tiers are per signal, and whether metrics, logs and traces need the same answer. They almost never do.

2. How many time series are you actually creating?

Cardinality is the failure mode that ends careers. A single label carrying a pod UID, a request ID or a customer identifier can add hundreds of thousands of series without anyone noticing until Prometheus is consuming more memory than the workloads it’s monitoring. The fix is a governance layer — relabelling, limits, and a review point before new exporters land — not a bigger node.

3. Who is the tenant?

LokiStack has a real multi-tenancy model. So does user workload monitoring, via namespace scoping and RBAC. If you have multiple teams, multiple business units or any regulatory separation, tenancy needs designing at the start. Retrofitting it means re-ingesting history.

4. Where do alerts go, and who owns them?

Alertmanager routing is trivial to configure and very difficult to get right. Most estates we inherit have a technically functional routing tree that nobody trusts, because severity levels were never defined and every alert goes to every channel. Alert fatigue is an ownership problem wearing a configuration costume.

Where teams most commonly get hurt

A short list, in rough order of frequency:

• Editing the operator-managed resources directly. Changes vanish at the next reconcile and the team concludes the platform is broken.
• Treating the platform Prometheus as a general-purpose metrics store. It isn’t, and remote-writing everything out of it without a filtering strategy gets expensive fast.
• Logging everything at DEBUG “for now.” Log volume is a cost line and a retention constraint. Someone should own log levels as a policy.
• No dashboard ownership. Dashboards proliferate, drift, and get abandoned; during an incident nobody knows which one is current.
• No pre-upgrade observability check. Cluster upgrades change operator versions, CRD schemas and default behaviour. The time to find out is not during the upgrade window.
• Tracing installed but never instrumented. The operator is running, the collector is healthy, and no application is emitting spans. This is more common than you’d think.

A quick maturity check

Run through these against your own cluster. Any “no” is a gap worth costing.

1. Can you answer a question about cluster state from ninety days ago?
2. Do you know your current active series count, and what it was last month?
3. Is user workload monitoring enabled, and is its retention set deliberately?
4. Is every observability component installed through OLM with a pinned channel?
5. Does every alert route to a named owner, not a shared channel?
6. Can you correlate a metric spike to logs and traces without leaving the console?
7. Do you know what your observability stack costs per month?

Most teams score three or four. That’s normal. The gap between four and seven is usually two to three weeks of focused work — not a re-platform.

Where we come in

The public documentation will tell you what each component does. What it won’t tell you is what your retention tiers should be, which of your labels are about to cause a cardinality incident, how to structure tenancy for your team layout, or what the whole thing should cost at your scale. Those answers depend on your workloads, your compliance position and your team’s operating model — and they’re the difference between a stack that works and a stack that’s installed.

We run fixed-scope OpenShift observability engagements: an assessment of the current state, a target architecture, an OLM-based implementation plan with pinned versions, and a handover your team can actually operate. Fixed price, defined deliverables, no open-ended retainers.

Theta Cloud Consulting is an independent consultancy. We are not affiliated with, endorsed by, or sponsored by Red Hat, Inc. OpenShift is a trademark of Red Hat, Inc.