Observability on NKPPart 2: NKP Insights

Summary: NKP Insights is the catalog app that analyzes the cluster and points out availability, configuration, best-practice and security points of attention, each with a cause and a solution. On the management cluster it came up in one minute, because the dependencies were already there. On the workload cluster it came up in about two minutes, with CloudNativePG enabled first and the Ceph storage from Part 1. In 10 minutes, it showed the Prometheus alerts with context and a lifecycle, inside Kommander, where the administrator already works. In 11 hours, it found 204 items, including security audits and the CIS Benchmark. This is part 2 of 3.

Where we left off

In Part 1, we turned on logs in the workload cluster and saw that Alertmanager, on since day one, comes with the default "null" destination, waiting for each environment’s real one, and that a critical rule, KubeProxyDown, does not apply to a cluster running Cilium.

The question for this part: can an app from the platform itself turn those alerts into something someone reads and acts on?

What NKP Insights is

It is a Nutanix analytics tool that runs inside NKP. It continuously examines the cluster and produces insights: findings with severity, type, probable cause and suggested solution, shown in Kommander itself.

The note on the details page deserves attention: with the default configuration, Insights writes to an S3 bucket and expects the Rook Ceph and Rook Ceph Cluster apps enabled, with enough space. The alternative is to point it at Nutanix Objects or another S3 store.

How it is built

The Part 1 inventory already showed a detail worth knowing: the central part of Insights (nkp-insights-management) runs on the management cluster since installation, even with the app off. Enabling the app adds the analysis engine to each cluster.

Enabling it on management: one minute

Goal: have Insights analyzing the management cluster.

In the management workspace, the Enable screen lists two dependencies, Prometheus Monitoring and CloudNativePG, and does not ask you to pick a cluster, because there is only one. Both dependencies were already there, as was Ceph with HEALTH_OK.

$ kubectl -n kommander get pods | grep insights
nkp-insights-backend-deployment-...       1/1  Running    3m59s
nkp-insights-backend-forwarder-...        1/1  Running    4m14s
nkp-insights-management-auth-server-...   1/1  Running    30h
nkp-insights-management-mgmt-cm-...       1/1  Running    30h
nkp-insights-object-store-init-...        0/1  Completed  4m14s
nkp-insights-postgresql-1                 1/1  Running    4m36s
nkp-insights-resolution-cm-...            1/1  Running    4m14s

Note the ages: the central part was 30 hours old; the rest, 4 minutes. The only warning event was a forwarder health check in the first second, normal during startup.

Enabling it on the workload cluster: dependencies first

Goal: the same, on the workload cluster, applying the best practice from Part 1: dependencies before the app.

Here CloudNativePG did not exist. I enabled it first (it sits in the General category of the catalog) and only then Insights. In the workload workspace, the Enable screen asks you to pick the cluster:

In about two minutes, the CloudNativePG operator, the Insights database and the analysis engine were up:

$ kubectl get clusters.postgresql.cnpg.io -A
NAME                      INSTANCES   READY   STATUS
nkp-insights-postgresql   1           1       Cluster in healthy state

object-store-init used the Ceph cluster we created in Part 1 for logs. Without it, Insights would wait for the bucket, like Loki in Part 1.

What it found in the first 10 minutes

On management, four findings. On workload, three.

Two things stand out. First, these are the same points Alertmanager showed in Part 1, but without the helper alerts Watchdog and InfoInhibitor, and inside Kommander, where the administrator already works. Second, the same cause got different severities: throttling is Warning on management and Notice on workload, because the limit was exceeded more on one than the other.

A finding from the inside

What sets Insights apart from a list of alerts is the detail page. Here is the one for the Fluentd throttling:

The finding’s Headlamp link opens the Fluentd StatefulSet directly. From warning to object in one click.

The lifecycle: Open, Muted and Resolved

The nkpclusters quota finding is intentional: NKP limits each workspace to one cluster. It is the textbook case for Mute.

Once muted, it leaves the open list but stays tracked and can come back with Unmute. A finding marked Resolved only reappears if the condition returns. That is a kind of control Alertmanager alone does not offer: there, either the alert is firing or someone created a time-boxed silence.

After 11 hours: 204 findings

The next day, the workload cluster had 204 open findings: 9 Critical, 50 Warning and 145 Notice.

The last item deserves attention. During the Part 3 lab, the OpenTelemetry Collector went into a crashloop because of a setting missing from the configuration I wrote. Insights recorded the finding about two minutes later, before I investigated.

How much it uses

Measured with kubectl top, right after startup:

backend is the biggest memory consumer. For sizing, add the dependencies: CloudNativePG and, if it does not exist yet, the Ceph cluster from Part 1.

Insights × Alertmanager × Headlamp

They complement each other: Insights organizes the daily review for whoever opens Kommander, and Alertmanager, with a real destination configured, is what notifies people outside it. In the lab, I did not find an external notification option in Insights.

Privacy, access and licensing

• Where the data lives: in the cluster itself, in PostgreSQL and the Ceph bucket. I did not analyze outbound traffic; for regulated environments, confirm with Nutanix.
• Who sees it: the Insights page lives inside each Kommander workspace and filters by cluster and project, so it follows the NKP workspace model and permissions.
• Licensing: the lab runs NKP Ultimate. Confirm with Nutanix which edition includes Insights before putting it in a proposal.

What I learned

1. The central part of Insights comes installed, even with the app off. Enabling it adds the analysis engine to each cluster.
2. On the workload cluster, dependencies come first: Ceph, CloudNativePG and only then Insights.
3. Within minutes, Insights shows the alerts inside Kommander, with cause, solution and a link to the object.
4. Mute and Resolved give findings a lifecycle, complementing the alert list.
5. Volume is not urgency: filter by severity and type, and start with the 9 Critical items before the 145 Notice ones.
6. Insights and Alertmanager work together. Insights organizes the review inside Kommander; Alertmanager, with a real destination, notifies people outside it.

NKP Insights checklist

• S3 storage available (Rook Ceph Cluster, Nutanix Objects or another S3)
• CloudNativePG enabled in the workspace
• Insights enabled in each workspace, picking the clusters
• Insights pods checked (backend, forwarder, resolution-cm, postgresql)
• Critical and Warning findings reviewed on day one
• Intentional findings marked Muted, with a justification
• Periodic review of Security and Best Practices findings
• NKP edition confirmed with Nutanix

What I haven’t tested yet

• NKP Insights notifications outside Kommander. In the lab, I followed the findings on the Insights page and did not verify external delivery.
• Insights outbound traffic. I did not run a network analysis; for regulated environments, that is the next step.
• Other NKP editions. The lab runs NKP Ultimate; I have not tested Insights on another edition.

References

Nutanix

• Nutanix Kubernetes Platform Guide 2.19, NKP Insights section (Nutanix portal)

Projects and community

• Polaris (Fairwinds)
• CIS Kubernetes Benchmark
• CloudNativePG