Upgrading NKP 2.18 → 2.19 in practicePart 2: the upgrade

Summary: the window took about 1h42 and ended with both clusters on NKP 2.19, Kubernetes 1.36.2 and Rocky Linux 9.8. The management cluster went up with the CLI in two steps, the workload cluster went up through the new UI in 21 minutes, and two incidents showed up along the way: one that had been hiding since the cluster was created, and a warning that cleared on its own. This is part 2 of a three-part series.

Where we left off

In part 1, I went through phases 0 to 3: assessment, creating the workload cluster while still on 2.18, downloads and backup. With everything green, the window started from here:

How I watched the window

Three terminals stayed open the whole time, and the entire window was recorded:

1. A loop against podinfo, the test application from part 1, which runs on the workload cluster: one request per second, printing the time and the HTTP code.
2. The upgrade commands, inside a tmux session on the bastion. If the SSH connection drops, the upgrade keeps running.
3. A watch on the nodes or the Cluster API machines, to see the replacement happen.

# Terminal 1 (laptop): the thermometer
while true; do printf "%s " "$(date +%T)"; curl -s -m 2 -o /dev/null -w "%{http_code}\n" http://<podinfo-ip>/ || echo FAIL; sleep 1; done

# Terminal 2 (bastion): everything inside tmux
ssh -t user@<bastion> 'tmux new -A -s upgrade'

Phase 4.1: Kommander

Goal: upgrade the NKP management layer (Kommander and the platform applications) to 2.19.

Coming from 2.18, this step is done with the CLI. Before starting, I checked the CLI version and node health, and sent the output to a log:

export KUBECONFIG=~/.kube/mgmt-admin.conf
nkp version
kubectl get nodes
nkp upgrade kommander 2>&1 | tee ~/phase4-1-kommander-$(date +%Y%m%d-%H%M).log

The command runs four checks before touching anything:

✓ Checking for deprecated GitRepository-based catalog apps
✓ Ensuring that new app dependencies introduced by the upgrade are enabled
✓ Ensuring upgrading conditions are met
✓ Checking docker.io/nutanix credentials are present for the fleet
• Upgrading ManagementPlane nkp-platform...

The last one is the Nutanix registry credential, the one I thought was configured and wasn’t (part 1, Phase 3). This time, it passed.

After the checks, the upgrade has two steps, and both stay silent for several minutes:

The criterion to move on

Once the command finished, I checked whether any platform application had been left unhealthy:

kubectl get helmreleases -A --no-headers | grep -v "True" || echo "ALL APPS READY"

The answer was ALL APPS READY. Only then did I move to the next phase.

Phase 4.2: the management cluster

Goal: take the management cluster itself to Kubernetes 1.36.2 and Rocky Linux 9.8 without taking anything down.

The command needs the cluster’s name and namespace in Cluster API, plus the new image. Instead of typing the first two, I read them straight from the NKP resource:

export MANAGEMENT_CLUSTER_NAME=$(kubectl get -n kommander nkpcluster \
  -l 'kommander.d2iq.io/host=true' -o jsonpath='{.items[0].status.capiClusterRef.name}')
export MANAGEMENT_CLUSTER_NAMESPACE=$(kubectl get -n kommander nkpcluster \
  -l 'kommander.d2iq.io/host=true' -o jsonpath='{.items[0].status.capiClusterRef.namespace}')
export VM_IMAGE_NAME=nkp-rocky-9.8-release-cis-1.36.2-<build>.qcow2

echo "Cluster: $MANAGEMENT_CLUSTER_NAME | Namespace: $MANAGEMENT_CLUSTER_NAMESPACE | Image: $VM_IMAGE_NAME"
nkp upgrade cluster nutanix \
  --cluster-name ${MANAGEMENT_CLUSTER_NAME} \
  --namespace ${MANAGEMENT_CLUSTER_NAMESPACE} \
  --vm-image ${VM_IMAGE_NAME}

The echo before the upgrade is on purpose: if a variable comes out empty, there is time to cancel. And the image name must be exactly what is in Prism Central (hence the don’t-rename rule from part 1).

✓ Waiting for ClusterClasses to be ready
✓ Running preflight checks
nkpcluster.clusters.nkp.nutanix.com/<management> upgraded
• Upgrading the cluster...

The rolling upgrade, node by node

From here on, Cluster API does the work. For each node, it:

1. creates a new VM with the new image;
2. waits for the new node to become Ready in Kubernetes;
3. cordons the old node (SchedulingDisabled) and drains its pods;
4. only then removes the old VM.

For a few minutes, the cluster has one extra node (9 instead of 8). It never has less capacity than before. In a terminal running watch -n 15 kubectl get nodes, you can see the versions change line by line:

NAME                       STATUS                     ROLES           AGE   VERSION
<mgmt>-cp-new-1            Ready                      control-plane   28s   v1.36.2
<mgmt>-cp-old-1            Ready,SchedulingDisabled   control-plane   29d   v1.35.2
<mgmt>-cp-old-2            Ready                      control-plane   29d   v1.35.2
...

Phase 4.3: validating the management cluster

Before touching the workload cluster, I validated the management cluster and checked that the workload cluster was still untouched:

kubectl get nodes -o wide                       # 8 nodes Ready, v1.36.2, Rocky Linux 9.8
kubectl get helmreleases -A --no-headers | grep -v "True" || echo "ALL APPS READY"
kubectl -n kube-system get cm coredns -o yaml | grep -c "lab.example.local"   # internal zone preserved
kubectl --kubeconfig ~/.kube/services.conf get nodes                          # workload still on v1.35.2

All good, including a detail I was worried about: the internal DNS zone I had added to CoreDNS (the Phase 1 troubleshooting) survived the upgrade on both clusters.

After logging back in, the dashboard showed v2.19.0 and Kubernetes v1.36.2, plus two new menus under Administration: Metal Host Inventory and Upgrades. The second one is Phase 5.

Before Phase 5: the problem that had been hiding

The 2.19 UI only allows a workload cluster upgrade when all of its applications are healthy. And part 1 had left a loose end: in the Phase 3 pre-checks, the workload cluster’s OpenCost showed an install failure. Time to understand it.

opencost   False   Helm install failed ... timeout waiting for: [Deployment/<namespace>/opencost status: 'InProgress']

I pulled the thread:

1. OpenCost was in CrashLoopBackOff, with more than 200 restarts. The log was clear: Failed to create Prometheus data source ... connection refused. OpenCost itself was fine; Prometheus was the one not answering.
2. The Prometheus HelmRelease showed True, but the Prometheus pod never even started. The events showed why:

   Error: configmap "cluster-info-configmap" not found

3. That ConfigMap existed on the management cluster, but not on the workload cluster. It has a single field, CLUSTER_ID, and Kommander should have delivered it when the cluster was attached to the workspace.

My hypothesis: a side effect of the cluster’s first day, when Flux couldn’t reach Kommander for lack of DNS (part 1, Phase 1). Almost everything was delivered after the fix, but not this item.

The fix, in four steps:

# 1. Confirm the pattern on the management cluster: CLUSTER_ID = UID of the kube-system namespace
kubectl -n kommander get cm cluster-info-configmap -o jsonpath='{.data.CLUSTER_ID}'
kubectl get ns kube-system -o jsonpath='{.metadata.uid}'

# 2. Create the ConfigMap on the workload cluster, with its own kube-system UID
export WK=~/.kube/services.conf; export NS=<workspace-namespace>
UID_WK=$(kubectl --kubeconfig $WK get ns kube-system -o jsonpath='{.metadata.uid}')
kubectl --kubeconfig $WK -n $NS create configmap cluster-info-configmap --from-literal=CLUSTER_ID="$UID_WK"

# 3. Restart Prometheus and OpenCost
kubectl --kubeconfig $WK -n $NS delete pod prometheus-kube-prometheus-stack-prometheus-0
kubectl --kubeconfig $WK -n $NS rollout restart deploy/opencost

# 4. Make Flux try again (it gives up after running out of retries)
kubectl --kubeconfig $WK -n $NS patch helmrelease opencost --type merge -p '{"spec":{"suspend":true}}'
kubectl --kubeconfig $WK -n $NS patch helmrelease opencost --type merge -p '{"spec":{"suspend":false}}'
T=$(date +%s)
kubectl --kubeconfig $WK -n $NS annotate helmrelease opencost \
  reconcile.fluxcd.io/requestedAt="$T" reconcile.fluxcd.io/forceAt="$T" reconcile.fluxcd.io/resetAt="$T" --overwrite

In about 8 minutes: Prometheus 3/3 Running, OpenCost 2/2 Running with zero restarts, the HelmRelease True and, in Kommander, the banner “You have cluster(s) that are ready for an upgrade”.

Phase 5: the workload cluster from the UI

Goal: test the headline feature of 2.19, upgrading a workload cluster from the Kommander UI.

Under Administration → Upgrades, the screen shows both clusters side by side. The management cluster already showed Newest Version; the workload cluster showed Ready, with the path already worked out: from v2.18.0 to v2.19.0 and from v1.35.2 to v1.36.2.

The Upgrade link on the cluster’s row opens a two-step wizard.

Step 1, Select Version: the current version and the recommended one (NKP v2.19.0 & Kubernetes v1.36.2), with the release notes built into the screen.

Step 2, Upgrade Configuration: the current and new image for each group of nodes (control plane and each node pool). This is where I picked the same Rocky Linux 9.8 image used on the management cluster.

Note the Pre-Checks message: from the UI, the checks always run. Skipping one is only possible from the CLI, and only if you are sure the failure is a false positive.

The Cluster Upgrade Tracker

One click on Upgrade, and the Upgrading link on the cluster’s row opens the Cluster Upgrade Tracker, another new feature in 2.19. It shows three steps, in this order:

1. Validating Inputs
2. Upgrading Platform Applications
3. Upgrading Kubernetes and Cluster Infrastructure

The order is the opposite of what I did on the management cluster: here, the applications go first and Kubernetes after. Machines only start being replaced in the third step.

The yellow warning

At the start of the third step, the Tracker showed this:

Before replacing machines, Cluster API calls a hook, BeforeClusterUpgrade, answered by an NKP extension running on the management cluster. The call took too long.

I didn’t click anything. The step was still spinning, with no failure mark, and Cluster API retries on its own. A likely cause: that extension’s pods had just moved to new nodes during the management upgrade. In about a minute, the warning was gone and the Tracker showed KubeadmControlPlane upgrading to v1.36.2.

The node replacement, seen from below

The same steps as on the management cluster, now on the workload cluster’s six nodes. With three terminals, you can see what the UI summarizes:

• Machines in Cluster API: kubectl get machines -n <namespace> shows each new machine going from Provisioning to Running, and the old ones in Deleting.
• Nodes in Kubernetes: the new node shows NotReady for a minute or two (pod networking still coming up) and then Ready on v1.36.2.
• The control plane safeguard: the cluster conditions show “Scaling down 4 → 3 blocked, waiting for control plane components to become healthy”. NKP only removes an old control plane after the new one is healthy.

Done

On the Upgrades screen, both clusters on the same version:

The final validation on the workload cluster repeated Phase 4.3: six nodes Ready on v1.36.2 with Rocky Linux 9.8, all applications ready, the internal DNS zone preserved, and the cluster-info-configmap I had created was still there.

What about podinfo?

The thermometer answered 200 every time I captured it. During Phase 4, that was expected: podinfo runs on the workload cluster, and the management upgrade shouldn’t affect it. Phase 5 was the real test, because the nodes being replaced were its own. In the screenshots and the recording, it kept answering; with 3 replicas and a PodDisruptionBudget, draining one worker never takes all of them down. I didn’t run a formal second-by-second failure count, so I won’t claim “absolute zero downtime”: that is the next test.

Window summary

Window checklist

• tmux session on the bastion for long-running commands
• A thermometer running against a real application
• nkp upgrade kommander and HelmRelease validation before moving on
• A plan for the Kommander login that drops after Phase 4.1
• nkp upgrade cluster variables checked with an echo first
• Exact image name from Prism Central
• CPU headroom for the rolling upgrade’s extra node
• Customizations (CoreDNS) checked after the upgrade
• All workload applications healthy before using the UI
• Patience with yellow warnings in the Tracker

References

• NKP 2.19 — Product Updates (Nutanix Community)
• Nutanix Kubernetes Platform Guide 2.19 — Upgrade Nutanix Kubernetes Platform (Nutanix portal)