Theta Cloud Consulting Insights

Category: Theta Cloud Consulting

  • Observability on OpenShift: Why the Built-In Stack Isn’t a Strategy

    Most OpenShift clusters arrive with observability already switched on. Prometheus is running. Alertmanager is running. There are dashboards in the console. For the first six months, nobody questions any of it.

    Then the cluster grows. Retention turns out to be three days when the compliance answer needed to be ninety. A cardinality spike quietly OOM-kills a Prometheus pod during an incident, so the one window you needed to look at is the one window you don’t have. Nobody can say who owns the alerts, so the on-call rota starts muting channels.

    None of that is an OpenShift defect. It’s the difference between monitoring shipped with a platform and an observability architecture designed for a specific estate. This post covers the shape of that difference — the components, the decisions, and where teams most often get hurt.

    What actually ships in OpenShift 4.x

    Worth being precise here, because “OpenShift has Prometheus” hides a lot of structure.

    The Cluster Monitoring Operator owns the platform stack: a Prometheus pair, Thanos Querier, Alertmanager, node-exporter, kube-state-metrics and the console integration. It is opinionated by design. You configure it through a ConfigMap, not by editing its resources — anything you change directly gets reconciled away.

    User workload monitoring is a separate, opt-in stack for your applications. It has its own Prometheus instance, its own Thanos component and its own retention and resource settings. This split is the single most misunderstood thing about OpenShift monitoring: platform and application telemetry are not one system, and tuning one does nothing for the other.

    Beyond that, the current generation of observability capability arrives as operators:

    • Cluster Observability Operator (COO) — the umbrella for newer monitoring capability, including the UI plugins that surface logs and traces in the console
    • Loki Operator — LokiStack for log aggregation, object-storage backed
    • Tempo Operator and OpenTelemetry Operator — distributed tracing and collection/pipeline management
    • Network Observability Operator — eBPF-based flow telemetry, which answers a category of question metrics simply cannot

    All of these install through OLM. That matters more than it sounds: a Subscription gives you a channel, an approval strategy and a documented upgrade path that survives cluster upgrades. Helm charts and raw manifests give you a snowflake that someone has to remember to reconcile by hand at 4.16.

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: loki-operator
      namespace: openshift-operators-redhat
    spec:
      channel: stable-x.y
      installPlanApproval: Manual
      name: loki-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace

    The channel and approval strategy you choose there are a governance decision, not a copy-paste. Manual approval on a production cluster and automatic on a sandbox is a reasonable default. Getting the pinning strategy consistent across an estate of clusters is where the real work lives.

    The four decisions that determine what this costs you

    Every OpenShift observability build we’ve done comes down to the same four questions. Answer them early and the rest is implementation. Answer them late and you’re doing a migration.

    1. Where does the data live, and for how long?

    Default retention is short and PVC-backed. Object storage — S3, or ODF if you’re on-prem — changes the economics entirely and is a prerequisite for LokiStack and Tempo anyway. The decision is not “do we use object storage”; it’s what your retention tiers are per signal, and whether metrics, logs and traces need the same answer. They almost never do.

    2. How many time series are you actually creating?

    Cardinality is the failure mode that ends careers. A single label carrying a pod UID, a request ID or a customer identifier can add hundreds of thousands of series without anyone noticing until Prometheus is consuming more memory than the workloads it’s monitoring. The fix is a governance layer — relabelling, limits, and a review point before new exporters land — not a bigger node.

    3. Who is the tenant?

    LokiStack has a real multi-tenancy model. So does user workload monitoring, via namespace scoping and RBAC. If you have multiple teams, multiple business units or any regulatory separation, tenancy needs designing at the start. Retrofitting it means re-ingesting history.

    4. Where do alerts go, and who owns them?

    Alertmanager routing is trivial to configure and very difficult to get right. Most estates we inherit have a technically functional routing tree that nobody trusts, because severity levels were never defined and every alert goes to every channel. Alert fatigue is an ownership problem wearing a configuration costume.

    Where teams most commonly get hurt

    A short list, in rough order of frequency:

    • Editing the operator-managed resources directly. Changes vanish at the next reconcile and the team concludes the platform is broken.
    • Treating the platform Prometheus as a general-purpose metrics store. It isn’t, and remote-writing everything out of it without a filtering strategy gets expensive fast.
    • Logging everything at DEBUG “for now.” Log volume is a cost line and a retention constraint. Someone should own log levels as a policy.
    • No dashboard ownership. Dashboards proliferate, drift, and get abandoned; during an incident nobody knows which one is current.
    • No pre-upgrade observability check. Cluster upgrades change operator versions, CRD schemas and default behaviour. The time to find out is not during the upgrade window.
    • Tracing installed but never instrumented. The operator is running, the collector is healthy, and no application is emitting spans. This is more common than you’d think.

    A quick maturity check

    Run through these against your own cluster. Any “no” is a gap worth costing.

    1. Can you answer a question about cluster state from ninety days ago?
    2. Do you know your current active series count, and what it was last month?
    3. Is user workload monitoring enabled, and is its retention set deliberately?
    4. Is every observability component installed through OLM with a pinned channel?
    5. Does every alert route to a named owner, not a shared channel?
    6. Can you correlate a metric spike to logs and traces without leaving the console?
    7. Do you know what your observability stack costs per month?

    Most teams score three or four. That’s normal. The gap between four and seven is usually two to three weeks of focused work — not a re-platform.

    Where we come in

    The public documentation will tell you what each component does. What it won’t tell you is what your retention tiers should be, which of your labels are about to cause a cardinality incident, how to structure tenancy for your team layout, or what the whole thing should cost at your scale. Those answers depend on your workloads, your compliance position and your team’s operating model — and they’re the difference between a stack that works and a stack that’s installed.

    We run fixed-scope OpenShift observability engagements: an assessment of the current state, a target architecture, an OLM-based implementation plan with pinned versions, and a handover your team can actually operate. Fixed price, defined deliverables, no open-ended retainers.

    Did any of those seven questions get a “no”?

    Tell us about your cluster and we’ll come back with a scoped, fixed-price proposal. No sales sequence, no retainer pitch — a 30-minute call and a written scope.

    Request a scoped proposal

    Live hours 1–5pm CT · Fixed scope, fixed price · UK-based, US-facing

    Theta Cloud Consulting is an independent consultancy. We are not affiliated with, endorsed by, or sponsored by Red Hat, Inc. OpenShift is a trademark of Red Hat, Inc.

  • What We Check First When We Inherit Someone Else’s Cluster

    The first post on this blog, and a reasonable summary of how we work.

    Most of our engagements start the same way. A team has a Kubernetes platform that works — mostly — and nobody left on staff who built it. Deploys take longer than they used to. The AWS bill has drifted upward without an obvious cause. There’s an upgrade everyone knows is overdue and nobody wants to be the one to run.

    We do fixed-scope work on exactly that problem. So rather than open this blog with an introduction nobody would read, here’s the actual checklist we work through in the first few days of a cluster handover. It’s the same list whether the platform is EKS, OpenShift, or something assembled from kubeadm and optimism.

    1. How far behind is the control plane, really?

    Version skew is the single most common reason an “upgrade project” turns into a quarter of work. Start with the honest numbers:

    kubectl version -o json | jq '.serverVersion.gitVersion'
    kubectl get nodes -o custom-columns=\
    NAME:.metadata.name,\
    VERSION:.status.nodeInfo.kubeletVersion,\
    OS:.status.nodeInfo.osImage

    What we’re looking for isn’t just the version — it’s the spread. A control plane at 1.29 with nodes at 1.26 means the supported skew window has already been breached, and the upgrade path is now serial rather than parallel. Each minor version has to be walked through in order, and on EKS that’s a distinct maintenance window per hop.

    The second question is what breaks on the way. Deprecated API usage is the usual landmine:

    kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

    That metric tells you which deprecated APIs are still being called and by what. It’s considerably more reliable than grepping manifests, because it catches the controller you forgot was installed three years ago.

    2. Who can actually reach AWS, and how?

    On EKS, the failure mode we see most often is credentials that work but shouldn’t. Node instance profiles carrying permissions that individual workloads inherit by default — meaning any pod on that node can assume them.

    kubectl get sa -A -o json | jq -r '
      .items[]
      | select(.metadata.annotations."eks.amazonaws.com/role-arn")
      | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations."eks.amazonaws.com/role-arn")"'

    If that returns very little on a busy cluster, workloads are almost certainly falling back to the node role. Moving to IRSA — or EKS Pod Identity on newer clusters — is usually a week of careful work and one of the highest-value security changes available. It also makes the blast radius of a compromised pod something you can reason about, which matters when a customer security questionnaire lands.

    While we’re here, we check whether the Terraform that provisioned the cluster still matches reality:

    terraform plan -detailed-exitcode

    An exit code of 2 on a cluster nobody has “changed” is a useful conversation starter. Drift accumulates through console clicks made during incidents, and every one of them is a change that will silently disappear the next time someone runs an apply.

    3. On OpenShift: what’s pinned, and what’s drifting?

    OpenShift clusters have a different failure profile. The platform itself is well-managed; the trouble is usually in what’s been installed on top of it.

    oc get clusterversion
    oc get subscription -A -o custom-columns=\
    NS:.metadata.namespace,\
    NAME:.metadata.name,\
    CHANNEL:.spec.channel,\
    APPROVAL:.spec.installPlanApproval

    That last column is the one that matters. A Subscription set to Automatic approval will upgrade its operator whenever the channel publishes a new version — which means an operator can move underneath you during a cluster upgrade, at the worst possible moment. We generally move production Subscriptions to Manual and pin the channel, so operator upgrades become a decision rather than an event.

    We install through OLM wherever an operator exists, rather than Helm or raw manifests. It’s not dogma — it’s that OLM gives you a dependency graph, a defined upgrade path, and a cluster-wide view of what’s installed and where it came from. Helm gives you a tarball and good intentions. On day one they look equivalent. On day four hundred, when someone needs to know why a CRD version changed, they’re not remotely equivalent.

    4. What does the deploy pipeline actually do?

    Most CI/CD archaeology comes down to one question: can you tell, from a running pod, exactly which commit produced it?

    kubectl get deploy -A -o json | jq -r '
      .items[]
      | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.template.spec.containers[0].image)"' \
      | grep -E ':(latest|master|main)$'

    Anything that returns from that grep is a deployment you cannot roll back with confidence. Mutable tags mean the image running in production is whatever was last pushed under that name, and the relationship to source control is a matter of trust rather than record.

    The fix is unglamorous — immutable tags, digest pinning, and a deployment annotation carrying the commit SHA — and it takes a couple of days. It also converts “roll back the bad release” from an investigation into a command.

    5. Where is the money going?

    Cost work usually gets framed as a finance request, but the useful version of it is an engineering question: what is running that nobody asked for?

    The three findings that come up almost every time are requests set far above actual usage (so the cluster scales out to satisfy reservations nobody needs), storage volumes left behind by deleted workloads, and non-production clusters running around the clock for a team that works one timezone’s business hours.

    None of that is exotic. It’s just that nobody owns it, so it compounds quietly.

    What this looks like as a piece of work

    Everything above is a few days of investigation followed by a scoped set of changes. That’s deliberately how we sell it: a defined piece of work with a stated outcome and an end date, not an open-ended retainer that becomes part of your run rate.

    Typical first engagements look like a cluster upgrade path with the deprecated-API work done up front, an IRSA migration, an OpenShift day-2 review covering operator pinning and upgrade readiness, or a CI/CD rebuild that makes deployments traceable and reversible.

    If any of the commands above returned something you’d rather they hadn’t, that’s usually a reasonable place to start a conversation.

    Get in touch

    Theta Cloud Consulting is a trading name of Theta Consulting Services Ltd. We are not affiliated with Red Hat, and OpenShift is a trademark of Red Hat, Inc.