Upgrading NKP 2.18 → 2.19 in practicePart 1: the preparation

Summary: before the first upgrade command, I went through four preparation phases: assessment, creating a workload cluster, downloads, and backup with pre-checks. That is where almost every problem in the project showed up. This is part 1 of a three-part series.

The context

Nutanix released Nutanix Kubernetes Platform (NKP) 2.19 on October 20, 2026. Besides Kubernetes 1.36, the feature I cared about most was upgrading workload clusters from the UI, one cluster at a time.

The starting environment:

• Nutanix hyperconverged cluster with 3 AHV nodes (376 GiB of RAM each), Prism Central and AOS 7.6
• NKP 2.18 Ultimate with a single cluster, the management cluster: 3 control planes and 4 workers, Kubernetes 1.35.2, Rocky Linux 9.7
• Internet access (not air-gapped)

The phased plan and the series

I split the project into seven phases. Each post in the series covers one block:

Two rules guided everything:

• Management goes first. Coming from 2.18, Kommander and the management cluster are upgraded with the CLI. Only then can the UI upgrade workload clusters.
• There is no in-place downgrade. If something goes badly wrong, the way back is restoring the backup. That is why Phase 3 is not optional.

Phase 0: assessment and cleanup

Goal: know exactly what exists before touching anything, and fix whatever would get in the way of the upgrade.

Read-only assessment

I wrote a script that only reads information from the cluster and saves everything to an evidence file:

export KUBECONFIG=~/.kube/mgmt-admin.conf
kubectl get nodes -o wide
kubectl get clusters -A                       # version, ClusterClass, phase
kubectl get svc -A | grep LoadBalancer        # MetalLB IPs
kubectl get gitrepositories,ocirepositories -A  # catalogs
kubectl get helmreleases -A                   # failing apps
kubectl get pdb -A                            # PDBs that could block the drain
kubectl -n kube-system get cm coredns -o yaml # DNS customizations

On the first run, almost everything failed with You must be logged in to the server. The kubeconfig I was using went through Kommander and relied on a token that had expired. With the admin kubeconfig, which uses a certificate and points straight at the control plane VIP on port 6443, the script ran end to end.

The cluster was healthy. The points that matter for 2.19:

• ClusterClass nkp-nutanix-v2.18.0: the requirement for the simplified upgrade.
• All catalogs were OCI. 2.19 no longer supports GitRepository-based catalogs; the only one present was Kommander’s own internal repository, which is standard.
• PDBs allowed at least one disruption, so node drains should not get stuck.
• CoreDNS had extra zones. Keep this in mind: it comes back in Phase 1.

2.19 prerequisites

I cross-checked the release notes and the official Upgrade Guide against the environment:

Capacity: does it fit if a host goes down?

On a 3-node HCI cluster, the right question is whether everything fits with N+1, meaning below roughly 2/3 of total memory. My first estimate assumed 256 GiB hosts and said it would not fit. Prism showed 376 GiB per node:

A rolling upgrade brings up a new VM before removing the old one, so that headroom has to be in the math.

Cleanup: what depends on what you are about to delete?

To free capacity, I removed an idle bastion VM and a legacy Kubernetes cluster (RKE2). Two questions prevented trouble:

• Where is the SSH key for the NKP nodes? Before deleting the bastion, I tested from the other one: ssh -o BatchMode=yes konvoy@<worker-ip> hostname. The key was on the right bastion.
• What runs on the legacy cluster? After deleting it, Prism Central login through the identity provider stopped working: the IdP was running right there. Access continued through the local account, and Keycloak comes later. The management cluster’s CoreDNS also had a zone pointing at that cluster, which I removed.

Network: IPAM does not see the physical world

A physical device (an NVIDIA DGX Spark) had a static IP inside the Nutanix IPAM pool. IPAM only knows about AHV VMs and could hand that IP to a new node while creating a cluster or during the rolling upgrade. I shrank the pool and laid out the network plan (anonymized example):

Alerts: severity is not impact

Of the two critical Prism Central alerts, only one affected NKP: “Data Services IP is unreachable”, the iSCSI address the CSI driver uses for volumes. I tested it (nc -vz <dsip> 3260), it answered, and the alert was stale. The other one only affected Prism Self-Service and was left for later.

Phase 1: create the workload cluster (still on 2.18)

Goal: have a workload cluster on 2.18 to test the headline feature of 2.19, the UI upgrade. Without one, there would be nothing to upgrade.

I created a new workspace and, inside it, the services cluster from the Kommander UI (Clusters → Add Cluster → Create Cluster): 3 control planes, 3 workers, VIP and MetalLB range outside the IPAM pool. Three form details:

• The workspace comes pre-selected with a different one. Check it.
• The worker count defaults to 4.
• The private registry accepts the Docker Hub URL with a username and token, which avoids the anonymous pull limit.

In about 10 minutes the cluster was Active. I validated it with a test volume (a 1 GiB PVC created, written and deleted) and deployed podinfo with 3 replicas, a PodDisruptionBudget and a LoadBalancer. It was the upgrade thermometer: a curl loop hit it throughout the whole window.

Troubleshooting: the cluster born without DNS

Half an hour later, no platform application had been installed. NKP installs apps in three steps: the cluster joins Kommander, Flux is installed on it, and Flux pulls the configuration from Kommander. The third step was stuck:

NAMESPACE        NAME         READY   STATUS
kommander-flux   management   False   failed to checkout and determine revision:
  unable to clone 'https://nkp.lab.example.local/dkp/kommander/git-operator/repositories/kommander/kommander.git':
  dial tcp: lookup nkp.lab.example.local on 10.96.0.10:53: no such host

The cause: Flux needs to reach Kommander by its internal name, but the new cluster’s CoreDNS forwarded queries to public DNS, which does not know the domain. The management cluster only resolved that name because of the extra zones from Phase 0. The new cluster was born without them.

The fix: replicate the zone in the new cluster’s CoreDNS and ask Flux to retry:

export KUBECONFIG=~/.kube/services.conf
kubectl -n kube-system get cm coredns -o yaml > coredns-backup.yaml
kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
cat >> Corefile <<'B'
lab.example.local:53 {
    errors
    hosts {
        192.0.2.61 nkp.lab.example.local
    }
    cache 30
}
B
kubectl -n kube-system create cm coredns --from-file=Corefile=Corefile \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n kommander-flux annotate gitrepository management \
  reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite

Within minutes, every app in the workspace was Deployed. The permanent fix is a real internal DNS server in IPAM; without it, every new cluster will need the same patch.

And that patch did not fix everything: a side effect of that first day without DNS stayed hidden in the cluster and only surfaced the day of the workload upgrade. That story is in part 2.

Phase 2: downloads

Goal: have everything the upgrade will use, at the exact version.

nkp CLI 2.19

The CLI must be exactly the target version. On the portal, under Downloads → NKP, the item is NKP for Linux (v2.19.0, nkp_v2.19.0_linux_amd64.tar.gz, about 180 MB). Instead of downloading it to your laptop and copying it over, use the three-dot menu: Copy Download Link gives you a temporary link to download straight onto the bastion, and Copy SHA256 copies the hash to verify the file.

On the bastion, I downloaded it, checked the hash and installed it:

curl -fLo nkp_v2.19.0_linux_amd64.tar.gz "<link-copied-from-the-portal>"
echo "38c1198a4226cdfb99afe601b1bedfe4459df478c4a4bac59a461c1926032fe4  nkp_v2.19.0_linux_amd64.tar.gz" | sha256sum -c
tar xzf nkp_v2.19.0_linux_amd64.tar.gz
sudo mv nkp /usr/local/bin/
nkp version    # nkp, kommander and konvoy at v2.19.0

sha256sum -c must answer OK. The link copied from the portal is signed and expires within a few hours: don’t store or share it, just generate a new one when you need it.

Rocky Linux 9.8 node image

On the portal, the image is the 2.19 NKP Node OS Image (Rocky Linux 9.8, CIS Hardened) for AHV. I uploaded it to Prism Central as a new image, following two rules:

• Check the SHA256 published on the portal.
• Do not rename the image. Both the upgrade command and the UI reference it by its exact name (in my case, nkp-rocky-9.8-release-cis-1.36.2-…qcow2).

The same image works for the management cluster and the workload clusters.

Phase 3: backup and pre-checks

Goal: guarantee a way back and arrive at the window with every prerequisite green.

The pre-check script

A second script, also read-only, checked the items the 2.19 Upgrade Guide requires:

1. Health of both clusters (nodes, failing HelmReleases and pods)
2. The Prism Central user NKP uses
3. Nutanix registry credentials present
4. DNS timeouts in the last hour (ideally zero)
5. Export of manifests and CoreDNS from both clusters

What the pre-checks found

• The Nutanix registry credentials were missing. The clusters only had a regular Docker Hub credential, and I assumed that was enough. It is not. I cover this step in detail below, because it can stop the upgrade before the first step.
• CPU headroom on management. CPU requests were at 87%, even though actual usage was 11% to 21%. During the upgrade, each node is drained and its pods have to fit somewhere else. I added a worker (from 4 to 5) from the UI, and requests dropped to about 72%.
• The Prism Central user had the required permissions, and there were zero DNS timeouts.
• One failing application on the workload cluster (OpenCost, with an install timeout). I noted it and moved on: it did not block the management upgrade. It would show up again in part 2.

Nutanix registry credentials on every cluster

This was the most important item in Phase 3. The platform images live in Nutanix’s private Docker Hub repository (docker.io/nutanix), and access requires a token generated in the Nutanix portal. The 2.19 release notes are straightforward: management clusters without that token must provide one during the upgrade.

Why “I already have a credential” is not enough. Both clusters already had a registry secret, configured with a regular Docker Hub account to avoid the anonymous pull limit. That account reaches public Docker Hub, not Nutanix’s private repository. nkp upgrade kommander checks for this right at the start, for the whole fleet:

✓ Checking docker.io/nutanix credentials are present for the fleet

Without the right credential, it stops with docker.io/nutanix credentials are missing, before touching anything.

Step 1: generate the token in the portal. On portal.nutanix.com, go to Downloads → Nutanix Kubernetes Platform (NKP). The Download from Nutanix Docker Hub Private Repository box has the Generate Access Token button.

Manage Access Token shows the registry username, provided by Nutanix, and the generated tokens. Three portal rules worth knowing before you generate one:

• The token is visible to every Support Portal user in your organization. Treat it as a shared company credential and keep it in a password vault.
• There can be at most 2 tokens at a time. That lets you rotate the token without stopping anything: generate the new one, apply it to the clusters, and only then delete the old one.
• If your organization’s Support Portal access is deactivated, all tokens are revoked.

Step 2: test the username and token pair. It is worth checking before applying, with the command the portal itself shows:

docker login --username <portal-username>
# Password: paste the token

Step 3: apply it to every cluster. A single command, run against the management cluster, updates the credential on the clusters in every workspace:

read -rsp "Token: " NKP_TOKEN; echo     # keeps the token out of shell history
nkp update cluster nutanix-registry-credentials --all-namespaces \
  --nutanix-registry-username <portal-username> \
  --nutanix-registry-password "$NKP_TOKEN"
unset NKP_TOKEN

Step 4: the confirmation. The proof came at the start of Phase 4: the credentials pre-check in nkp upgrade kommander passed with ✓ for the whole fleet, and the upgrade moved on.

Layered backup

Since there is no downgrade, I made three layers of backup:

Two traps here:

• On a cluster with CloudNativePG, kubectl get backup resolves to the CNPG CRD, not Velero’s. My backup “looked” status-less until I used the full name, backups.velero.io.
• Velero’s default destination is object storage on the management cluster itself. A backup that lives on the cluster you are upgrading does not protect you from losing that cluster. That is why the etcd snapshot went off the cluster.

Summary: what each phase revealed

Not a single upgrade command has been run so far.

Downloadable checklist

• Read-only assessment with saved evidence
• Admin kubeconfig tested
• Prerequisites checked against the target version’s Upgrade Guide
• N+1 capacity with headroom for the rolling upgrade
• Dependencies mapped before any deletion
• VIPs, MetalLB and physical devices outside the IPAM pool
• Internal DNS resolving the Kommander endpoint on every cluster
• Target version CLI and image (SHA256 checked, original name)
• Nutanix registry credentials on every cluster
• CPU and memory headroom on management
• Layered backup, with one copy off the cluster

References

• NKP 2.19 — Product Updates (Nutanix Community)
• Nutanix Kubernetes Platform Guide 2.19 — Upgrade Nutanix Kubernetes Platform (Nutanix portal)