Nutanix Enterprise AI on NKPPart 4: CPU, GPU and MCP in a hybrid architecture

Summary: in Part 3, NAI answered its first token on CPU. Now it stops being an island. I put the same OpenAI client calling the NAI CPU endpoint and a 30B model on a GB10 GPU, changing only the model. I get +40% tokens/s on CPU with one environment variable. I bring the vLLM metrics into the NKP Grafana. I turn on the NAI Agentic Gateway: unified endpoint, token-based rate limiting and MCP, validated end to end with two configuration adjustments I show step by step. And I protect NAI with Velero (copy on Backblaze B2) and NDK (local snapshot), with a tested restore on both. This is part 4 of 4.

Where we are

The starting point: NAI 2.8.1 on a dedicated NKP cluster, a qwen3-06b-cpu endpoint (Qwen3 0.6B on vLLM, 8 vCPU) and, in the same lab, an ASUS Ascent GX10 (NVIDIA GB10 chip, the DGX Spark platform) serving Nemotron 30B through NIM behind a LiteLLM that the NKP applications already used.

The roadmap in one table

1. The same API for CPU and GPU

The design is simple: the LiteLLM that already served the applications becomes the single entry point. Each application has a LiteLLM key; LiteLLM has a NAI API key just for itself.

What changed on the GX10

Three things, with backup and rollback:

1. A new API key in NAI (litellm-gx10), only for the CPU endpoint. The test key stays separate.
2. LiteLLM learned to trust the internal CA. The container uses Python’s certifi, which doesn’t know the lab CA. I built a bundle (the certifi CAs + the internal CA) and pointed SSL_CERT_FILE and REQUESTS_CA_BUNDLE to it. While the NAI name isn’t in DNS, an extra_hosts entry does the job.
3. A new model in config.yaml:

  - model_name: qwen3-cpu-nai
    litellm_params:
      model: openai/qwen3-06b-cpu
      api_base: https://nai.lab.example.local/enterpriseai/v1
      api_key: os.environ/NAI_API_KEY
      extra_body:
        chat_template_kwargs:
          enable_thinking: false

The openai/ provider works because the NAI API is OpenAI-compatible. Only the LiteLLM container was recreated, and it was back in 11 seconds; NIM didn’t even notice.

The test: same code, two worlds

A Python script with the official OpenAI SDK, running in a pod in the NKP applications cluster. The only difference between the calls is the model:

client = OpenAI(base_url="https://gx10-llm.ai-gateway.svc/v1", api_key=os.environ["API_KEY"])
for model in ["nemotron-lightning", "qwen3-cpu-nai"]:
    stream = client.chat.completions.create(model=model, stream=True, ...)

The GPU is 13× faster with a model 50× larger. Nobody will use the 0.6B on CPU for chat; the point is different: the application code doesn’t change, and the LiteLLM usage log records the calls to both backends in the same place.

2. +40% on CPU with one variable

In Part 3 the CPU endpoint delivered 4.5 tokens/s, and the explanation was CPU throttling: the container sees the node’s 12 cores, vLLM opens 11 OpenMP threads, but the pod limit is 8 vCPU. The Grafana dashboard (section 3) showed the size of the effect: 60% to 77% of periods throttled during generation.

The adjustment is the VLLM_CPU_OMP_THREADS_BIND=0-7 variable, which pins the vLLM threads to 8 cores. In NAI 2.8.1, the endpoint’s advanced configuration offers Engine Arguments and KV Cache Aware Routing; an environment variable like this one goes on the InferenceService:

I applied the variable directly to the KServe InferenceService. This path is outside the NAI UI (in a production environment, proceed with caution); in the lab, NAI kept the change. The rollout needed one adjustment: the predictor uses RollingUpdate, which starts the new pod before removing the old one, and my lab’s inference node (12 vCPU) can’t hold two 8-vCPU pods.

+40% tokens/s and 30% less total time, without changing vCPU, memory or model.

3. vLLM metrics in the NKP Grafana

NAI shows requests and tokens per endpoint. To also see latency and the pod’s CPU usage, vLLM exposes all of that at /metrics on the predictor’s port 8000; you just point the NKP Prometheus there. It only collects monitors with the Kommander label:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: nai-vllm
  namespace: nai-admin
  labels:
    prometheus.kommander.d2iq.io/select: "true"
spec:
  selector:
    matchLabels:
      endpoint.iep.nai.nutanix.com/engine: vllm
  podMetricsEndpoints:
    - portNumber: 8000
      path: /metrics
      interval: 15s

Within 10 seconds the target was up and Prometheus had 349 vllm:* series. The dashboard goes in a ConfigMap with grafana_dashboard: "1", which the NKP Grafana sidecar loads on its own:

The panel that justified section 2 is predictor CPU and throttling (CFS): during generation, the pod used 5.5–6.3 vCPU and spent 60–77% of periods throttled.

4. The Agentic Gateway

NAI 2.8 brought the Agentic Gateway: unified endpoints (one name in front of several models, with load balancing or fallback), external providers, token-based rate limiting and MCP. Underneath is the Envoy AI Gateway. My lab license doesn’t include the Agentic Gateway (Part 3), but the menu is already in the console, and I used the lab to get to know the feature in depth.

The drawing above sums up what I tested: chat and MCP come in through the same door, and the gateway does three things in sequence before reaching the model or the MCP server. It is adapted from the Agent Gateway workflow Nutanix presented at the NAI 2.7 launch.

Unified endpoint

I created unified-endpoint-pr, type Chat, with the local CPU endpoint as the model and the test API key:

The UI generated an Envoy AI Gateway AIGatewayRoute that matches the x-ai-eg-model: unified-endpoint-pr header, rewrites the model name to the real endpoint’s and counts input, output and total tokens. The path is /enterpriseai/gateway/v1/chat/completions; the local endpoints’ /enterpriseai/v1 returns 404 for the unified endpoint name.

In the lab, I reduced the requests (not the limits) in the NAI GatewayConfig and EnvoyProxy. The new pod came up in 67 seconds, the old one left on its own, and the unified endpoint answered 200 in 5.4 s. In production, the recommended path is to size the gateway nodes for the default values. That’s about 5 vCPU and 10 GiB per replica that enter the math before any model.

Token-based rate limiting

Rate limiting is by tokens, not requests: global for the endpoint and, optionally, per API key (or per value of a header, when several applications share the key).

Underneath it becomes an Envoy BackendTrafficPolicy with two rules. The client rule identifies the key by the X-Nutanix-Api-Key-Id header, which NAI itself injects after authenticating, and the cost of each request is the total tokens in the response. The test burst, with 300 tokens/min for the key:

The counter deducts each response’s real tokens, and the 11th call is blocked at the gateway, in 13 ms, without reaching the model.

MCP: servers, connectors and tool control

The MCP part has three pieces:

• MCP Servers: local (NAI deploys them from a Docker image) or remote (a URL).
• MCP Connectors: group one or more servers and choose which tools each one exposes.
• MCP Client Keys: a key type of their own, separate from the chat API keys.

I used a public remote MCP server, Context7 (library documentation). The connector discovered both tools; enabling both requires typing “confirm”:

The connector becomes an MCPRoute at /enterpriseai/gateway/mcp/<connector>, with a toolSelector listing the allowed tools. That’s how the UI’s “Tool Control” becomes configuration.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: mcp-context7
  namespace: nai-admin-extensions
spec:
  type: Endpoints
  endpoints:
    - fqdn: { hostname: mcp.context7.com, port: 443 }
  tls:
    sni: mcp.context7.com
    wellKnownCACertificates: System
    ecdhCurves: [X25519, P-256, P-384, P-521]

With that, MCP through the Agentic Gateway worked end to end:

The prefix is what lets a connector group several servers without name clashes.

5. Protecting NAI: Velero and NDK

NAI keeps state in four RWO volumes in nai-system (the PostgreSQL database, ClickHouse, its keeper and Valkey) and the models in Files RWX volumes. The database runs as a single instance, so include these volumes in your backup routine. I used both tools from the NKP catalog, with different roles.

Velero with an off-lab copy (Backblaze B2)

NKP 2.19’s Velero is 1.18, with the AWS plugin, data mover (Kopia) and node-agent. The target was a Backblaze B2 bucket, S3-compatible and outside the lab. Two notes for B2: checksumAlgorithm: "" in the backup storage location (the new plugin sends a checksum B2 may reject) and the bucket lifecycle set to “keep only the last version”, or deleted backups become hidden versions and the space only grows.

When enabling Velero, the console warns that Rook Ceph Cluster is missing:

Two more details of Velero on NKP:

• The pre-install waits 5 minutes for Ceph. With an external target, Velero comes up on its own after that; it’s not an error.
• After editing an override in the console, check the Helm revision. In my test, I changed the schedule and the release stayed on revision 1; checking that the release moved to a new revision confirms the change reached the cluster.

On the first full backup, the limit was the lab’s upload link to B2 (0.2–0.3 MB/s): the 1.5 GB model would take hours. Decision: models out of the backup (they come back from Hugging Face in a minute), with a resourcePolicy that skips the Files storage class:

version: v1
volumePolicies:
  - conditions:
      storageClass: [nai-nfs]
    action:
      type: skip

Result: 510 objects and 4 volumes in 776 s, 0 errors. The volumes’ ~710 MB became ~313 MB in the bucket (Kopia compresses and deduplicates):

NDK: a local snapshot in seconds

Nutanix Data Services for Kubernetes 2.3 protects the application with snapshots from the Nutanix storage itself. Installation asks for an NDK-specific Docker Hub credential (the Nutanix portal generates a token), passed in the override:

NDK is light: 35m CPU and 256 MiB of requests. After the Enable, three steps:

1. StorageCluster: ties NDK to Prism Central and the AOS cluster, by the UUIDs of both. I got them from the Prism v3 API with the credential the CSI already had.
2. Application: the whole nai-system namespace. NDK asks for an explicit selector, and the namespace selector is part of a feature that’s off by default (the message is “multi namespace feature not enabled”). The simple way to take the whole namespace is to select everything that doesn’t have a nonexistent label:

apiVersion: dataservices.nutanix.com/v1alpha1
kind: Application
metadata:
  name: nai-system
  namespace: nai-system
spec:
  applicationSelector:
    resourceLabelSelectors:
      - labelSelector:
          matchExpressions:
            - { key: lab.example.local/exclude, operator: DoesNotExist }

3. ApplicationSnapshot: ready in 41 seconds, crash-consistent, with NAI’s deployments, statefulsets, secrets, configmaps and PVCs.

I didn’t restore the whole of NAI into another namespace: it would start a second copy of 18 workloads on a lab cluster with no headroom. I tested the same mechanism with a small application (1 volume with 50 random MB): snapshot in 37 s, restore in 21 s, identical hash. To restore into another namespace, NDK requires a ReferenceGrant in the source namespace.

Velero or NDK? Both

The numbers

What I learned

1. A gateway in front frees the application from choosing the hardware. CPU for the cheap work, GPU for the heavy work, the same code.
2. On CPU, pin the vLLM threads to the pod limit. That was 40% for free. And use Recreate for endpoints larger than half a node.
3. Measure on the server side. The vLLM PodMonitor and the throttling panel showed what no client-side number explained.
4. Reserve ~5 vCPU and ~10 GiB per Agentic Gateway replica before any model. And check that the new proxy pod is Running before the first unified endpoint test.
5. Token-based rate limiting works as expected, charging the response’s real tokens.
6. Validate the remote MCP server before creating the connector: the certificate curve and the MCP session are the two points to check.
7. Velero and NDK don’t compete. One takes the copy off-site; the other brings you back in seconds.
8. Restoring volumes with the data mover needs a consumer pod when the storage class is WaitForFirstConsumer.

Checklist: hybrid and protected NAI

• Dedicated NAI API key for the LLM gateway, separate from the test key
• Internal CA in the gateway’s CA bundle (SSL_CERT_FILE)
• Gateway keys enabled per model, per application
• VLLM_CPU_OMP_THREADS_BIND on the CPU endpoint and deploymentStrategy: Recreate
• Warm-up call after any endpoint restart
• vLLM PodMonitor with prometheus.kommander.d2iq.io/select: "true" and a Grafana dashboard
• ~5 vCPU and ~10 GiB of headroom per gateway replica before creating the first unified endpoint
• API keys associated with each unified endpoint; MCP client keys separate
• Remote MCP server tested (certificate curve, session) before becoming a connector
• Velero with an external target (no Rook Ceph); resourcePolicy for the models
• Bucket lifecycle set to “last version”
• Helm revision checked after each override Edit
• NDK with StorageCluster, the nai-system Application and a ProtectionPlan
• Restore tested on both

What I haven’t tested yet

• The GPU inside NKP itself. In this part, the GPU comes in through the GX10, behind LiteLLM; a GPU endpoint served by NAI on NKP is the lab’s next step.
• External providers in the Agentic Gateway. In the lab, the unified endpoint routed to the local CPU endpoint and MCP to Context7; external providers were outside this test.
• Restoring the whole of NAI into another namespace with NDK. It would start a second copy of 18 workloads on a lab cluster with no headroom; I tested the same mechanism with a small application.

References

Nutanix

• Nutanix Enterprise AI Guide 2.8.1 (Nutanix portal): endpoints, API keys, Agentic Gateway (unified endpoints, MCP servers and connectors, rate limiting)
• Nutanix Data Services for Kubernetes 2.3 (Nutanix portal)

Projects and community

• Envoy AI Gateway: documentation
• Envoy Gateway: Backend and TLS settings
• LiteLLM: OpenAI-compatible providers
• vLLM: installation and CPU variables
• KServe: InferenceService
• Velero: data movement and resource policies
• Model Context Protocol: specification