Theta Cloud Consulting Insights

Blog

One Pane of Glass for a Hybrid Trading Estate: OpenShift, Grafana LGTM and Alloy The problem: three worlds, three toolsets
Most trading and banking infrastructure isn’t one clean estate — it’s three at once:
- Containerised applications running on OpenShift or Kubernetes
- On-prem data centre infrastructure — network switches, routers and firewalls quietly generating syslog
- Windows estate — domain controllers, app servers, and legacy services still producing Windows Event Logs
Traditionally that means three different tools, three different query languages, and an incident bridge call where nobody can see the whole picture at once. When an order-gateway service throws an exception, the question is rarely “what did the app log say” in isolation — it’s “did this correlate with a network event, an ACL change, or something on the Windows side.” If your logging estate is fragmented, that correlation happens by phone call instead of by query.

The architecture: Grafana LGTM + Alloy on OpenShift

For a recent lab build, I stood up the Grafana LGTM stack (Loki for logs, Grafana for visualisation, and the wider Mimir/Tempo family for metrics and traces) on OpenShift, with Grafana Alloy as the single, vendor-neutral collector feeding it all.

The shape of it:
- OpenShift hosts Grafana and Loki, exposed via an OpenShift Route, and everything is installed and lifecycle-managed through OLM/OperatorHub rather than hand-rolled manifests — consistent with how I approach day-2 operations on OpenShift generally.
- Alloy is deployed as the collection layer across the estate, tagging every stream with consistent labels — job, estate, host, os — so a single Loki instance can hold containerised app logs, on-prem syslog, and Windows Event Logs side by side without them turning into an undifferentiated soup.
- Label design does the heavy lifting: an estate=onprem-dc1 label separates on-prem sources from cloud/cluster-native ones, while job=tradingapp, job=syslog and job=windowsevent let you pivot between application, network, and Windows telemetry inside the same Explore view — no swivel-chairing between three consoles.
Explore, filtered to /var/log/tradingapp/order-gateway.log via the Loki-trading datasource

Filtering the same Loki instance by estate=onprem-dc1 to isolate on-prem sources

What it looks like in practice

In the walkthrough below, a simulated order-gateway trading application is logging structured INFO/WARN/ERROR events (including a Java IllegalStateException on a settlement window closing) alongside network syslog and Windows Event Log — all landing in the same Loki instance with consistent labels.

Log-level volume (error/warning/info) for the on-prem estate, with the order-gateway settlement exception visible in the stream

Pivoting to job=tradingapp to see just the trading application’s logs

Trading app INFO/WARN order events landing in Loki with consistent estate/host/job labels

Network syslog from the on-prem estate — interface state changes, BGP neighbour flaps, and ACL denies (%SEC-6-IPACCESSLOGP) — arrives through the same pipeline with the job=syslog label, and Windows Event Log entries land under job=windowsevent, sitting in the exact same query surface as everything else.

Windows Event Log entries collected via Alloy under job=windowsevent

Network syslog isolated with job=syslog, ready to correlate against app-level errors

The syslog stream: interface state changes and BGP neighbour events alongside an ACL deny

That means when the order-gateway starts throwing settlement exceptions, the very next query is “did anything change on the network or the Windows estate in the same window” — and it’s a label filter away, not a different tool and a different login.

Why this matters for regulated, hybrid environments
- Faster root cause, fewer bridge calls. One query surface across app, network and Windows telemetry cuts the “let me check with the network team” loop out of an incident.
- Consistent retention and access control. Centralising through Loki/Alloy means one place to apply retention policy and access boundaries, rather than three siloed log stores with three different rules.
- Cloud-agnostic collection. Alloy isn’t tied to a single vendor’s format, which matters when your estate genuinely is hybrid — cloud-native services next to twenty-year-old on-prem infrastructure that isn’t going anywhere soon.
- OpenShift-native lifecycle. Running the observability stack itself through OLM keeps upgrades and version pinning predictable — the same day-2 discipline I’d apply to any other OpenShift workload.
- Built for noisy, latency-sensitive estates. Trading and market-data environments generate high log volume with low tolerance for blind spots — this pattern is designed to scale with that, not fall over under it.
Where this fits

This is a reference pattern, not a fixed product — the label taxonomy, retention tuning, and collector placement all get shaped around the actual estate: how many on-prem sites, how much Windows footprint, what’s already emitting syslog versus what needs an agent. If you’re running a hybrid OpenShift/on-prem/Windows estate and logging is currently three tools stitched together with hope, this is exactly the kind of engagement I take on.

Recognise your estate in any of this?

Tell us about your estate and we’ll come back with a scoped, fixed-price proposal. No sales sequence, no retainer pitch — a 30-minute call and a written scope.

Request a scoped proposal

Live hours 1–5pm CT · Fixed scope, fixed price · UK-based, US-facing

Theta Cloud Consulting is an independent consultancy. We are not affiliated with, endorsed by, or sponsored by Red Hat, Inc. OpenShift is a trademark of Red Hat, Inc.
August 12, 2026
Observability on OpenShift: Why the Built-In Stack Isn’t a Strategy
Most OpenShift clusters arrive with observability already switched on. Prometheus is running. Alertmanager is running. There are dashboards in the console. For the first six months, nobody questions any of it.

Then the cluster grows. Retention turns out to be three days when the compliance answer needed to be ninety. A cardinality spike quietly OOM-kills a Prometheus pod during an incident, so the one window you needed to look at is the one window you don’t have. Nobody can say who owns the alerts, so the on-call rota starts muting channels.

None of that is an OpenShift defect. It’s the difference between monitoring shipped with a platform and an observability architecture designed for a specific estate. This post covers the shape of that difference — the components, the decisions, and where teams most often get hurt.

What actually ships in OpenShift 4.x

Worth being precise here, because “OpenShift has Prometheus” hides a lot of structure.

The Cluster Monitoring Operator owns the platform stack: a Prometheus pair, Thanos Querier, Alertmanager, node-exporter, kube-state-metrics and the console integration. It is opinionated by design. You configure it through a ConfigMap, not by editing its resources — anything you change directly gets reconciled away.

User workload monitoring is a separate, opt-in stack for your applications. It has its own Prometheus instance, its own Thanos component and its own retention and resource settings. This split is the single most misunderstood thing about OpenShift monitoring: platform and application telemetry are not one system, and tuning one does nothing for the other.

Beyond that, the current generation of observability capability arrives as operators:
- Cluster Observability Operator (COO) — the umbrella for newer monitoring capability, including the UI plugins that surface logs and traces in the console
- Loki Operator — LokiStack for log aggregation, object-storage backed
- Tempo Operator and OpenTelemetry Operator — distributed tracing and collection/pipeline management
- Network Observability Operator — eBPF-based flow telemetry, which answers a category of question metrics simply cannot
All of these install through OLM. That matters more than it sounds: a Subscription gives you a channel, an approval strategy and a documented upgrade path that survives cluster upgrades. Helm charts and raw manifests give you a snowflake that someone has to remember to reconcile by hand at 4.16.
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat spec: channel: stable-x.y installPlanApproval: Manual name: loki-operator source: redhat-operators sourceNamespace: openshift-marketplace
The channel and approval strategy you choose there are a governance decision, not a copy-paste. Manual approval on a production cluster and automatic on a sandbox is a reasonable default. Getting the pinning strategy consistent across an estate of clusters is where the real work lives.

The four decisions that determine what this costs you

Every OpenShift observability build we’ve done comes down to the same four questions. Answer them early and the rest is implementation. Answer them late and you’re doing a migration.

1. Where does the data live, and for how long?

Default retention is short and PVC-backed. Object storage — S3, or ODF if you’re on-prem — changes the economics entirely and is a prerequisite for LokiStack and Tempo anyway. The decision is not “do we use object storage”; it’s what your retention tiers are per signal, and whether metrics, logs and traces need the same answer. They almost never do.

2. How many time series are you actually creating?

Cardinality is the failure mode that ends careers. A single label carrying a pod UID, a request ID or a customer identifier can add hundreds of thousands of series without anyone noticing until Prometheus is consuming more memory than the workloads it’s monitoring. The fix is a governance layer — relabelling, limits, and a review point before new exporters land — not a bigger node.

3. Who is the tenant?

LokiStack has a real multi-tenancy model. So does user workload monitoring, via namespace scoping and RBAC. If you have multiple teams, multiple business units or any regulatory separation, tenancy needs designing at the start. Retrofitting it means re-ingesting history.

4. Where do alerts go, and who owns them?

Alertmanager routing is trivial to configure and very difficult to get right. Most estates we inherit have a technically functional routing tree that nobody trusts, because severity levels were never defined and every alert goes to every channel. Alert fatigue is an ownership problem wearing a configuration costume.

Where teams most commonly get hurt

A short list, in rough order of frequency:
- Editing the operator-managed resources directly. Changes vanish at the next reconcile and the team concludes the platform is broken.
- Treating the platform Prometheus as a general-purpose metrics store. It isn’t, and remote-writing everything out of it without a filtering strategy gets expensive fast.
- Logging everything at DEBUG “for now.” Log volume is a cost line and a retention constraint. Someone should own log levels as a policy.
- No dashboard ownership. Dashboards proliferate, drift, and get abandoned; during an incident nobody knows which one is current.
- No pre-upgrade observability check. Cluster upgrades change operator versions, CRD schemas and default behaviour. The time to find out is not during the upgrade window.
- Tracing installed but never instrumented. The operator is running, the collector is healthy, and no application is emitting spans. This is more common than you’d think.
A quick maturity check

Run through these against your own cluster. Any “no” is a gap worth costing.
1. Can you answer a question about cluster state from ninety days ago?
2. Do you know your current active series count, and what it was last month?
3. Is user workload monitoring enabled, and is its retention set deliberately?
4. Is every observability component installed through OLM with a pinned channel?
5. Does every alert route to a named owner, not a shared channel?
6. Can you correlate a metric spike to logs and traces without leaving the console?
7. Do you know what your observability stack costs per month?
Most teams score three or four. That’s normal. The gap between four and seven is usually two to three weeks of focused work — not a re-platform.

Where we come in

The public documentation will tell you what each component does. What it won’t tell you is what your retention tiers should be, which of your labels are about to cause a cardinality incident, how to structure tenancy for your team layout, or what the whole thing should cost at your scale. Those answers depend on your workloads, your compliance position and your team’s operating model — and they’re the difference between a stack that works and a stack that’s installed.

We run fixed-scope OpenShift observability engagements: an assessment of the current state, a target architecture, an OLM-based implementation plan with pinned versions, and a handover your team can actually operate. Fixed price, defined deliverables, no open-ended retainers.

Did any of those seven questions get a “no”?

Tell us about your cluster and we’ll come back with a scoped, fixed-price proposal. No sales sequence, no retainer pitch — a 30-minute call and a written scope.

Request a scoped proposal

Live hours 1–5pm CT · Fixed scope, fixed price · UK-based, US-facing

Theta Cloud Consulting is an independent consultancy. We are not affiliated with, endorsed by, or sponsored by Red Hat, Inc. OpenShift is a trademark of Red Hat, Inc.
August 8, 2026
What We Check First When We Inherit Someone Else’s Cluster
The first post on this blog, and a reasonable summary of how we work.

Most of our engagements start the same way. A team has a Kubernetes platform that works — mostly — and nobody left on staff who built it. Deploys take longer than they used to. The AWS bill has drifted upward without an obvious cause. There’s an upgrade everyone knows is overdue and nobody wants to be the one to run.

We do fixed-scope work on exactly that problem. So rather than open this blog with an introduction nobody would read, here’s the actual checklist we work through in the first few days of a cluster handover. It’s the same list whether the platform is EKS, OpenShift, or something assembled from kubeadm and optimism.

1. How far behind is the control plane, really?

Version skew is the single most common reason an “upgrade project” turns into a quarter of work. Start with the honest numbers:
kubectl version -o json | jq '.serverVersion.gitVersion' kubectl get nodes -o custom-columns=\ NAME:.metadata.name,\ VERSION:.status.nodeInfo.kubeletVersion,\ OS:.status.nodeInfo.osImage
What we’re looking for isn’t just the version — it’s the spread. A control plane at 1.29 with nodes at 1.26 means the supported skew window has already been breached, and the upgrade path is now serial rather than parallel. Each minor version has to be walked through in order, and on EKS that’s a distinct maintenance window per hop.

The second question is what breaks on the way. Deprecated API usage is the usual landmine:
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
That metric tells you which deprecated APIs are still being called and by what. It’s considerably more reliable than grepping manifests, because it catches the controller you forgot was installed three years ago.

2. Who can actually reach AWS, and how?

On EKS, the failure mode we see most often is credentials that work but shouldn’t. Node instance profiles carrying permissions that individual workloads inherit by default — meaning any pod on that node can assume them.
kubectl get sa -A -o json | jq -r ' .items[] | select(.metadata.annotations."eks.amazonaws.com/role-arn") | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations."eks.amazonaws.com/role-arn")"'
If that returns very little on a busy cluster, workloads are almost certainly falling back to the node role. Moving to IRSA — or EKS Pod Identity on newer clusters — is usually a week of careful work and one of the highest-value security changes available. It also makes the blast radius of a compromised pod something you can reason about, which matters when a customer security questionnaire lands.

While we’re here, we check whether the Terraform that provisioned the cluster still matches reality:
terraform plan -detailed-exitcode
An exit code of 2 on a cluster nobody has “changed” is a useful conversation starter. Drift accumulates through console clicks made during incidents, and every one of them is a change that will silently disappear the next time someone runs an apply.

3. On OpenShift: what’s pinned, and what’s drifting?

OpenShift clusters have a different failure profile. The platform itself is well-managed; the trouble is usually in what’s been installed on top of it.
oc get clusterversion oc get subscription -A -o custom-columns=\ NS:.metadata.namespace,\ NAME:.metadata.name,\ CHANNEL:.spec.channel,\ APPROVAL:.spec.installPlanApproval
That last column is the one that matters. A Subscription set to Automatic approval will upgrade its operator whenever the channel publishes a new version — which means an operator can move underneath you during a cluster upgrade, at the worst possible moment. We generally move production Subscriptions to Manual and pin the channel, so operator upgrades become a decision rather than an event.

We install through OLM wherever an operator exists, rather than Helm or raw manifests. It’s not dogma — it’s that OLM gives you a dependency graph, a defined upgrade path, and a cluster-wide view of what’s installed and where it came from. Helm gives you a tarball and good intentions. On day one they look equivalent. On day four hundred, when someone needs to know why a CRD version changed, they’re not remotely equivalent.

4. What does the deploy pipeline actually do?

Most CI/CD archaeology comes down to one question: can you tell, from a running pod, exactly which commit produced it?
kubectl get deploy -A -o json | jq -r ' .items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.template.spec.containers[0].image)"' \ | grep -E ':(latest|master|main)$'
Anything that returns from that grep is a deployment you cannot roll back with confidence. Mutable tags mean the image running in production is whatever was last pushed under that name, and the relationship to source control is a matter of trust rather than record.

The fix is unglamorous — immutable tags, digest pinning, and a deployment annotation carrying the commit SHA — and it takes a couple of days. It also converts “roll back the bad release” from an investigation into a command.

5. Where is the money going?

Cost work usually gets framed as a finance request, but the useful version of it is an engineering question: what is running that nobody asked for?

The three findings that come up almost every time are requests set far above actual usage (so the cluster scales out to satisfy reservations nobody needs), storage volumes left behind by deleted workloads, and non-production clusters running around the clock for a team that works one timezone’s business hours.

None of that is exotic. It’s just that nobody owns it, so it compounds quietly.

What this looks like as a piece of work

Everything above is a few days of investigation followed by a scoped set of changes. That’s deliberately how we sell it: a defined piece of work with a stated outcome and an end date, not an open-ended retainer that becomes part of your run rate.

Typical first engagements look like a cluster upgrade path with the deprecated-API work done up front, an IRSA migration, an OpenShift day-2 review covering operator pinning and upgrade readiness, or a CI/CD rebuild that makes deployments traceable and reversible.

If any of the commands above returned something you’d rather they hadn’t, that’s usually a reasonable place to start a conversation.

Get in touch

Theta Cloud Consulting is a trading name of Theta Consulting Services Ltd. We are not affiliated with Red Hat, and OpenShift is a trademark of Red Hat, Inc.
August 6, 2026