Observability on NKPPart 1: the concepts

Summary: monitoring tells you whether something is broken; observability tells you why. This part brings together the concepts I use with customers (pillars, methods, SLOs and cost) and shows what NKP 2.19 already ships. In the lab, the workload cluster had metrics only. Turning on logs from the catalog took four apps and about 30 minutes, including a complete Ceph storage cluster. And the Alertmanager, on since day one, was firing alerts to no one. This is part 1 of 3.

Why this series

After the NKP upgrade to 2.19, the next question from customers was almost always the same: “so how do I see what is happening inside the cluster?”. The short answer is that NKP already brings a lot. The long answer involves concepts, choices and a few caveats, and it became this series.

Everything was tested in the Clear IT AI lab: one management cluster and one workload cluster on NKP 2.19, running on Nutanix AHV.

Monitoring × observability

The two terms are often used as synonyms, but they answer different questions.

Monitoring is a subset of observability. You still need the alerts; the difference is having enough data to investigate what the alert does not explain.

The pillars

In Kubernetes, events work in practice as a fourth pillar: an OOMKilled or a FailedScheduling explains a lot before any dashboard does. The real value shows up when the signals are linked. In Part 3, one click on a failed trace opens the logs of that very request.

What to measure: three methods

Measuring everything is expensive and confusing. These methods help you pick the useful minimum.

A rule of thumb: RED for applications, USE for nodes and storage. The dashboards NKP ships cover USE well; RED depends on instrumenting the applications, which is the subject of Part 3.

SLI, SLO and error budget

This is where observability becomes a business conversation.

With an SLO in place, the alert stops being “CPU went up” and becomes “we are burning the error budget too fast”. That cuts useless alerts and gives a clear criterion for choosing between shipping new features and investing in stability.

OpenTelemetry: the open standard

OpenTelemetry is a CNCF project that standardizes how applications produce metrics, logs and traces and how that data is transported. The core idea: instrument once and send to any tool, open source or commercial.

In practice, it removes lock-in from the instrumentation layer. Changing backends becomes a configuration change in the Collector, not a code rewrite. NKP 2.19 ships the OpenTelemetry Operator in the catalog, and it is the topic of Part 3.

The tools on the market

A neutral view of what is out there. There is no best one; there is the one that fits each company’s team, budget and retention rules.

The criteria I use to choose:

Many companies end up with a mixed model: the platform’s open stack for day-to-day work and a commercial tool for the most critical applications.

The cost of observability

Observability is not free, even with open source tools. The cost comes from three places:

• Metric cardinality: every combination of labels becomes a series. A label holding the user ID can multiply the series by millions.
• Log volume and retention: debug logging left on in production fills storage fast.
• Cluster resources: every component of the stack uses CPU, memory and disk.

In the lab, the management cluster’s Prometheus alone used about 5.9 GiB of memory, with 10-day retention on a 100 GiB volume. And by the end of this series, after turning on logs, Insights and OpenTelemetry in the workload cluster, the cluster’s reserved CPU (requests) went from 27.1% to 72.5%. Observability has to be part of sizing from the start.

What NKP already ships turned on

Before enabling anything, I ran a read-only inventory on both clusters. The difference between them was the first finding.

The workload cluster comes with metrics but no logs. For whoever runs applications there, the most-used signal in daily work was missing.

The workload Grafana already showed the resource dashboards working, with a detail worth flagging to any customer: the CPU limits of the kube-system namespace added up to 161% of what the cluster has.

In the catalog, the Observability category has two apps, both off by default. They are the topics of parts 2 and 3.

In the lab: turning on logs in the workload cluster

Goal: get the logs of every pod in the workload cluster into Grafana, using only the NKP catalog.

This is the log path in NKP:

There are four apps in the catalog, each depending on another:

The shortcut that did not work

I started with Grafana Logging and accepted the dependencies the interface showed. When I enabled Loki, NKP warned that two were missing: Rook Ceph Cluster and Logging Operator. The warning lets you continue anyway, and I did, to see what would happen.

The app showed as Enabled in the interface. The pods did not come up:

grafana-loki-v3-ingester-zone-a-0   0/1   CreateContainerConfigError
...
Warning  Failed  kubelet  Error: secret "dkp-loki" not found

The dkp-loki Secret holds the credentials for the bucket Loki writes to. That bucket is created by Rook Ceph Cluster, which did not exist yet. Note a confusing detail: Rook Ceph and Rook Ceph Cluster are different apps. The first is only the operator; the second creates the storage cluster.

In the right order

I enabled the missing dependencies. The Logging Operator came up in about a minute: one operator, 6 Fluent Bit pods (one per node) and one Fluentd. Rook Ceph Cluster took longer, because it builds an entire storage cluster in sequence: monitors, managers, disks and the S3 gateway.

The Ceph cluster created by the app uses volumes from the cluster’s default storage and delivered 160 GiB raw, with 4 data disks across the 3 workers, 2 S3 gateways and 3+1 erasure coding for bucket data. It created two buckets: Loki’s and one for Velero.

The logs that arrived before Loki

In Grafana Logging, kube-system logs showed up from 00:20, about 12 minutes before Loki was ready. The explanation was in the Fluentd log:

[warn]: ... failed to flush the buffer. retry_times=0 ... error="502 Bad Gateway"
[warn]: ... failed to flush the buffer. retry_times=1 ...

Fluentd tried to deliver, got a 502 from the Loki gateway, kept the logs in an on-disk buffer and kept retrying. When Loki came up, it delivered what it had stored. Nothing was lost.

The alerts that warned no one

Alertmanager comes turned on in both clusters. In the workload cluster, 5 alerts were firing:

KubeProxyDown is a false alarm because the cluster has no kube-proxy: Cilium does that job.

$ kubectl -n kube-system get ds kube-proxy
Error from server (NotFound): daemonsets.apps "kube-proxy" not found
$ kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.kube-proxy-replacement}'
true

And the most important part was in the default configuration:

route:
  receiver: "null"
receivers:
- name: "null"

The only configured destination is "null". Alerts fire and go to no one: no email, no Slack, no Teams. Together with a false critical alert firing for two days, that is the scenario that teaches a team to ignore alerts.

What about AI?

The same logic applies to AI workloads, with their own signals:

OpenTelemetry already has semantic conventions specific to generative AI, to record model, tokens and latency in a standard way.

A maturity model

To place where each customer stands:

Parts 2 and 3 take the lab from level 2 to level 3.

What I learned

1. The workload cluster comes with metrics only. Logs and traces have to be enabled, and that has a cost.
2. The dependency warning lets you continue, and the result is an “Enabled” app that does not work. Check the pods after every Enable.
3. Rook Ceph and Rook Ceph Cluster are different apps. The first is the operator; the second is the storage.
4. The Fluentd buffer holds the logs when the destination goes down. That resilience comes out of the box.
5. An alert with no destination protects nothing, and a false critical alert teaches people to ignore alerts.
6. Observability uses resources. It goes into sizing, not after it.

Basic observability checklist on NKP

• Inventory of what is already on in each cluster
• Logs enabled on workload clusters, with dependencies in order
• Pods checked after every Enable, not just the app card
• At least one real destination configured in Alertmanager
• Watchdog wired to an external “dead man’s switch” service
• Rules that do not apply to the environment tuned or silenced
• Metric and log retention defined by business rules
• Observability stack consumption included in cluster sizing

References

• Site Reliability Engineering, chapter 6: Monitoring Distributed Systems (Google)
• The USE Method (Brendan Gregg)
• OpenTelemetry
• Grafana Loki
• Logging Operator (Kube Logging)
• Rook
• Nutanix Kubernetes Platform Guide 2.19 (Nutanix portal)