Nutanix Enterprise AI on NKPPart 2: Nutanix Files for Kubernetes

Summary: Nutanix Enterprise AI stores models on a ReadWriteMany (RWX) volume, and NKP’s default storage class is Nutanix Volumes, ReadWriteOnce; RWX comes from Nutanix Files. In this part I deploy Nutanix Files 5.4.0.1 from Prism Central (3 FSVMs, 5 TiB, 27 minutes), enable NFS v3 and v4, test the mount from a VM and connect Files to Kubernetes. A configuration detail worth knowing: the NKP 2.19 CSI runs in Prism Central mode, and dynamic Files provisioning needs its own secret with a REST API user from the file server. In the end, an RWX volume is ready in 6 seconds, and two pods on different nodes write to the same file. This is part 2 of 4.

Where we are

The series follows the enablement of Nutanix Enterprise AI (NAI) in a lab with NKP 2.19 and a Nutanix HCI cluster with no GPU.

Before installing NAI, I read the 2.8.1 User Guide and the two Nutanix Validated Designs that cover it. All three agree on one prerequisite NKP does not ship ready: shared NFS storage. This part solves it end to end, and it applies to any NKP application that needs RWX, not just NAI.

Why NAI needs RWX

A Kubernetes volume has an access mode. The two that matter here:

NAI downloads each model once and keeps it on a shared volume. When an endpoint gets a second replica, or when a pod is recreated on another node, it reads the weights from the same place, without downloading everything again. That is why the model volume must be RWX.

What the NAI 2.8.1 User Guide asks for in storage:

The architecture

Nutanix Files runs as a set of VMs on the HCI cluster itself, the FSVMs (File Server VMs). They serve NFS to clients and store data in a storage container on the cluster. On the Kubernetes side, the Nutanix CSI does two different things: through the Files REST API, it creates one share per requested volume; through NFS, it mounts that share on the node where the pod runs.

Planning: the decisions before the wizard

Size

The Nutanix NVDs use FSVMs with 12 vCPU and 64 GB in production. In a lab, the scarce resource is memory, so I went with the supported minimum.

The 3 FSVMs add up to 36 GB of RAM. I ran the cluster math first: the lab has 3 hosts, and I want everything to keep running if one host fails (N+1), which caps usage at about two thirds of total memory.

Network

The wizard asks for two networks. The lab has a single VLAN, managed by AHV IPAM, so both went there.

DNS and NTP

Files asks for DNS and NTP servers. The lab has no internal DNS, and I wanted to confirm what the gateway resolved before filling in the form:

$ dig @192.0.2.1 pe.lab.example.local
;; WARNING: .local is reserved for Multicast DNS
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN
;; AUTHORITY SECTION:
.   86400  IN  SOA  a.root-servers.net. nstld.verisign-grs.com. ...

NXDOMAIN with the internet root SOA: the gateway only forwards queries out and does not know the internal domain. The practical consequence: clients will mount Files by IP, not by name. It works well, and the CSI accepts an IP normally.

License

Files is licensed through Nutanix Unified Storage (NUS), metered in usable TiB. Before deploying, I checked what was applied to the cluster:

Deploying the file server from Prism Central

Goal: a file server with 3 FSVMs, 5 TiB and NFS, created through the Prism Central wizard (Files → File Servers → New File Server).

The starting point, with no file server:

1. Location: Prism Central or Prism Element

The first screen asks for the provider (Nutanix, AWS or Azure) and already shows a warning: some features are not supported on file servers deployed through Prism Central; to use them, deploy from Prism Element.

I went ahead with Prism Central. This warning matters for the protection choice (see the protection section).

2. Cluster and version

The wizard only lists versions that are compatible and available in LCM. There were 13; I picked the newest, 5.4.0.1.

3. Basics and sizing

Name (up to 15 characters, no dots), DNS domain, logical capacity and FSVM size.

4. Client and storage networks

Both screens show the IPAM range, how many IPs are free and how many the file server will use: 3 on client, 4 on storage.

5. Settings: DNS, NTP and two switches for Kubernetes

This is the most important screen, because nothing here can change after deployment.

The Deployment Profile field has three options:

And the two switches:

6. Summary

The deployment: 27 minutes

Prism Central created the 3 FSVMs in the first 90 seconds, then mounted the Files image and configured the file server cluster.

The file server dashboard opened Healthy, with no alerts and 5 TiB free:

Right after Succeeded, Files ran two more tasks: FS Deploy VG Create and Volume Group Manage. That is Containerized Workload at work, pre-creating the volume groups Kubernetes shares will use.

The file server network

The network screen shows the layout and the IPs IPAM handed out:

The fourth storage IP (the wizard’s “+1”) does not show in the UI table. It exists; look it up in the subnet IPAM if you need to record it.

The container Files creates

The wizard creates a storage container just for the file server, NTNX_files-lab_ctr. I checked its properties against the GPT-in-a-Box NVD:

Same as recommended, without touching anything. The 32 GiB already used right after deployment are the pre-created volume groups.

Enabling NFS

The file server starts with no protocol. Under Configuration → Authentication, I enabled NFS only:

First export and the manual test

Before involving Kubernetes, I tested Files on its own, with a 10 GiB export mounted from a Linux VM.

Standard or Distributed

Squash and sudo

On the permissions screen, the default is Root Squash:

With root squash, the client’s root becomes an anonymous user on the server. Since my test wrote with sudo, it would fail with Permission denied. For the test, I switched it to None:

Mounting from a VM

$ sudo apt install -y nfs-common
$ sudo mount -t nfs4 192.0.2.128:/teste-nfs /mnt/teste-nfs
$ echo "ok $(date)" | sudo tee /mnt/teste-nfs/teste.txt
ok Fri Oct 30 01:56:55 AM UTC 2026
$ df -h /mnt/teste-nfs
Filesystem               Size  Used Avail Use% Mounted on
192.0.2.132:/teste-nfs      0     0     0    - /mnt/teste-nfs
$ mount | grep teste-nfs
192.0.2.132:/teste-nfs on /mnt/teste-nfs type nfs4 (rw,vers=4.2,proto=tcp,sec=sys,addr=192.0.2.132)

Three things to notice:

1. I mounted through .128 and ended up on .132. On a Standard export, a single FSVM serves the whole export, and Files uses NFSv4 redirection to send the client to it. The client can use any client IP.
2. NFS 4.2, negotiated automatically.
3. df shows size 0. Files does not expose the export quota to the NFS client. It is not an error; good to know so it does not surprise you.

Kubernetes seeing Files

What NKP already has

NKP installs the Nutanix CSI on every cluster. The workload cluster inventory:

$ kubectl get sc
NAME                       PROVISIONER                              VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
dkp-object-store           <workspace>.ceph.rook.io/bucket          Immediate              false
nutanix-volume (default)   csi.nutanix.com                          WaitForFirstConsumer   true

$ kubectl get csidrivers
NAME              ATTACHREQUIRED   PODINFOONMOUNT   MODES
csi.nutanix.com   true             true             Persistent

$ kubectl -n ntnx-system get ds,deploy -o wide | grep csi
daemonset.apps/nutanix-csi-node         6 6 6 ...  docker.io/nutanix/ntnx-csi:3.9.0
deployment.apps/nutanix-csi-controller  2/2 ...    docker.io/nutanix/ntnx-csi:3.9.0

The driver is 3.9.0, the default storage class is Volumes (RWO), and there is a single credentials secret, ntnx-system/nutanix-csi-credentials.

The detail: the CSI runs in Prism Central mode

The CSI 3.x documentation shows dynamic Files with the driver talking to Prism Element, and a secret with a second key, files-key. Before copying the example, I checked who the NKP CSI talks to, without exposing any password:

# Only the secret key names
$ kubectl -n ntnx-system get secret nutanix-csi-credentials \
    -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'
key

# Only host and port of the key (the command cuts user and password)
$ kubectl -n ntnx-system get secret nutanix-csi-credentials \
    -o jsonpath='{.data.key}' | base64 -d | cut -d: -f1,2
192.0.2.88:9440

$ kubectl -n ntnx-system get deploy nutanix-csi-controller -o yaml \
    | grep -iE 'usepc|use_pc|management-plane|mixed'
    app.kubernetes.io/management-plane: PrismCentral
        - name: USE_PC
        - --use-mixed-mode=false

The address is Prism Central’s, and the controller has USE_PC. The NKP 2.19 CSI runs in PC mode.

1. REST API user on the file server

The files-key uses a user of the file server’s own API, not a Prism user. It is created in the file server console, under Configuration → Manage Roles → REST API access users:

2. Dedicated secret

The script reads the Prism Central key from the NKP secret and builds the new secret without printing anything:

read -rp  "File server REST API user: " FU
read -rsp "Password (hidden): " FP; echo
PCKEY=$(kubectl -n ntnx-system get secret nutanix-csi-credentials \
          -o jsonpath='{.data.key}' | base64 -d)
kubectl -n ntnx-system create secret generic nutanix-files-csi \
  --from-literal=key="$PCKEY" \
  --from-literal=files-key="192.0.2.128:$FU:$FP"
unset PCKEY FP

I used the client IP in files-key, not the name, because the lab nodes cannot resolve the internal domain.

3. The storage class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nai-nfs
provisioner: csi.nutanix.com
parameters:
  storageType: NutanixFiles
  dynamicProv: ENABLED
  nfsServerName: files-lab        # file server name in Prism
  nfsServer: 192.0.2.128          # client IP: mount and API
  squashType: "none"
  csi.storage.k8s.io/provisioner-secret-name: nutanix-files-csi
  csi.storage.k8s.io/provisioner-secret-namespace: ntnx-system
  csi.storage.k8s.io/node-publish-secret-name: nutanix-files-csi
  csi.storage.k8s.io/node-publish-secret-namespace: ntnx-system
  csi.storage.k8s.io/controller-expand-secret-name: nutanix-files-csi
  csi.storage.k8s.io/controller-expand-secret-namespace: ntnx-system
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate      # required by NAI

The RWX test: two nodes, one file

The test creates a 2 GiB RWX PVC and a deployment with 2 replicas, forced onto different nodes (podAntiAffinity). Each replica writes a line every 5 seconds to the same file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: rwx-teste }
spec:
  accessModes: [ReadWriteMany]
  storageClassName: nai-nfs
  resources: { requests: { storage: 2Gi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: rwx-escritor }
spec:
  replicas: 2
  selector: { matchLabels: { app: rwx-escritor } }
  template:
    metadata: { labels: { app: rwx-escritor } }
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: { matchLabels: { app: rwx-escritor } }
            topologyKey: kubernetes.io/hostname
      containers:
      - name: c
        image: busybox:1.36
        command: ["sh","-c","while true; do echo \"$(date -u +%H:%M:%S) $(hostname) no nó $NODE\" >> /dados/log.txt; sleep 5; done"]
        env: [{ name: NODE, valueFrom: { fieldRef: { fieldPath: spec.nodeName } } }]
        volumeMounts: [{ name: d, mountPath: /dados }]
      volumes: [{ name: d, persistentVolumeClaim: { claimName: rwx-teste } }]

The result (the log lines are in Portuguese, as in the lab script: “no nó” means “on node”):

$ kubectl -n files-teste get pvc rwx-teste
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rwx-teste   Bound    pvc-bc03c1b0-8456-4cda-a43c-3a5f70b446ea   2Gi        RWX            nai-nfs        6s

$ kubectl -n files-teste get pods -o wide
NAME                            READY   STATUS    NODE
rwx-escritor-68f798cf6d-v5s5s   1/1     Running   worker-a
rwx-escritor-68f798cf6d-wrlph   1/1     Running   worker-b

$ kubectl -n files-teste exec <pod> -- tail -6 /dados/log.txt
02:38:48 rwx-escritor-68f798cf6d-wrlph no nó worker-b
02:38:50 rwx-escritor-68f798cf6d-v5s5s no nó worker-a
02:38:53 rwx-escritor-68f798cf6d-wrlph no nó worker-b
02:38:55 rwx-escritor-68f798cf6d-v5s5s no nó worker-a
02:38:58 rwx-escritor-68f798cf6d-wrlph no nó worker-b
02:39:00 rwx-escritor-68f798cf6d-v5s5s no nó worker-a

lines per pod:
      5 rwx-escritor-68f798cf6d-v5s5s
      5 rwx-escritor-68f798cf6d-wrlph

$ kubectl -n files-teste exec <pod> -- mount | grep /dados
192.0.2.128:/pvc-bc03c1b0-... on /dados type nfs4 (rw,vers=4.2,proto=tcp,...)

The lines from both replicas interleave: two nodes writing to the same file. That is RWX.

What the CSI did under the hood

The controller log shows the full path, in about 2.6 seconds:

Using Files secrets available in PC Mode
GET  https://192.0.2.128:9440/api/files/v4.0/config/file-servers
POST https://192.0.2.128:9440/api/files/v4.0/config/file-servers/<uuid>/mount-targets
Task completed with status: SUCCEEDED, progress: 100%
Setting cfg.volumePath with value: pvc-bc03c1b0-8456-4cda-a43c-3a5f70b446ea
ControllerPublishVolume called ... nfsServer:192.0.2.128 nfsServerName:files-lab provNFSShare:ENABLED

Three informational messages worth knowing:

The lifecycle, seen from Files

With the test running, the volume shows up in the Files console as a share named after the PV:

I deleted the test namespace. With reclaimPolicy: Delete, the CSI removed the share on its own:

The NAI model export

NAI also asks for a separate NFSv4 export to import custom or manually downloaded models. I created nai-modelos (Portuguese for “nai-models”): Standard, NFS v3 and v4, Read-Write, squash None, no size limit.

“Max Size 5 TiB” means the export can use the whole file server. The Mount Path shows the DNS name; without internal DNS, use the client IP.

Protection: Protection Domain, Smart DR or SSR

The file server console offers two kinds of protection:

When configuring the Protection Domain, the screen shows it applies to file servers created from Prism Element:

In the lab, I decided not to protect the file server: models can be downloaded again, and the NAI data that matters (database, metrics, cache) lives on RWO volumes, on Nutanix Volumes. Protecting those volumes is a job for NDK and Velero.

Lab × Nutanix Validated Design

The numbers

What I learned

1. NKP’s RWX comes from Nutanix Files. The default storage class is Nutanix Volumes (RWO); for NAI, and for any app with a shared volume, add Files and a dedicated storage class.
2. The NKP 2.19 CSI runs in Prism Central mode. Dynamic Files works in that mode with a dedicated secret holding key (PC) and files-key (file server REST API user).
3. Containerized Workload and Direct Attach are the right options for Kubernetes, and they cannot change after deployment.
4. Enable NFSv4 before creating exports. The factory default is NFSv3 only.
5. Root squash and sudo do not mix in a manual test; choose the squash setting consciously in the storage class.
6. For a Protection Domain, deploy the file server from Prism Element. From Prism Central, protection is Smart DR.
7. Without internal DNS, mount by IP. Files redirects the client to the right FSVM.

Checklist: Nutanix Files for NKP

• NUS license applied to the cluster (NCI Ultimate includes 1 TiB)
• Memory for the FSVMs within the cluster’s N+1
• Subnet with IPAM and free IPs (FSVMs on client; FSVMs + 1 on storage)
• DNS and NTP reachable from the client network
• Decided: Prism Central or Prism Element (protection)
• Containerized Workload and Direct Attach enabled in the wizard
• Post-deployment tasks finished before the first share
• NFSv4 enabled
• REST API user created (and saved with the ✓)
• Dedicated secret with key and files-key, without touching the NKP secret
• RWX storage class with Immediate
• Test with 2 pods on different nodes, and the share gone after cleanup
• Separate export for custom models (if you will use NAI)

What I haven’t tested yet

• Nutanix Objects (S3), which the NVD uses in its design: I did not install it in the lab, and it is planned for later (~104 GB of RAM).
• File server protection with Smart DR, which needs a second file server. In the lab, I chose not to protect Files, because the models can be downloaded again.
• The NVD’s production sizing: 12 vCPU / 64 GB FSVMs and the client network on a separate subnet. In the lab, I stayed at the supported minimum and on a single VLAN, as a conscious deviation.

References

Nutanix

• Nutanix Validated Design: Nutanix Enterprise AI (NVD-2057)
• Nutanix CSI Operator: storage classes and secrets (opendocs.nutanix.com)
• Files and AOS version compatibility (Nutanix Community)
• Nutanix Enterprise AI Guide 2.8 (Nutanix portal), requirements and NKP deployment sections

Projects and community

• Kubernetes: Persistent Volumes, access modes

Independent

• Deploying Nutanix Enterprise AI on a non-Nutanix Kubernetes cluster (Jonas Hogman)