2026 Kimi K3 vLLM Startup Failure: Upgrade or Switch?

nvidia-smi shows an r575 driver, while the Kimi K3 container reports a cu130 build and exits during engine initialization.

Fastest fix: if the failure is confirmed as cu130-versus-r575 incompatibility and you have a maintenance window, upgrade to r580 or newer with a tested rollback plan. If the shared cluster cannot change today, move the validation workload to an isolated compatible environment. Rebuild vLLM for cu129 only when your team can maintain that fork.

Who should use this runbook

Use this guide if you are recovering Kimi K3 vLLM on an existing GPU cluster, approving a driver change, or trying to unblock Agent integration without waiting for a shared platform migration.

It is aimed at infrastructure engineers, platform owners, and application teams that need a decision within the same working day.

Last updated August 5, 2026. Facts were checked against the official Kimi K3 vLLM recipe, the vLLM Kimi K3 release post, and NVIDIA CUDA compatibility documentation.

The failed-start scene

Do not begin by changing five launch flags at once. Freeze the evidence first.

A failed Kimi K3 startup can look like an OOM, an NCCL failure, or a generic CUDA initialization error. Those symptoms are not interchangeable. If the container contains a CUDA 13 build but the node exposes an r575 driver, the version conflict happens before normal model-capacity tuning becomes meaningful.

The current vLLM recipe states that the Kimi K3 image is available as a CUDA 13, or cu130, build only. It also states that the host requires an r580 or newer NVIDIA driver. There is no official cu129 tag listed in the recipe, and the documented alternative is to build vLLM from the K3 branch against cu129 PyTorch yourself.

Save these artifacts before changing the node:

• Container image digest and tag.
• vLLM package or source commit.
• Model revision and launch command.
• Host output from nvidia-smi.
• Container output from python -c "import torch; print(torch.version.cuda)".
• Complete engine initialization logs.
• Container runtime, GPU visibility, and network configuration.
• The exact first failing line, not only the final traceback.

The important boundary is simple: do not label every startup failure “driver-related.” Confirm the image, driver branch, vLLM source, and first failing subsystem before selecting a recovery path.

Operational reminder: nvidia-smi reports the driver’s maximum supported CUDA level. It does not prove that every model kernel, container library, or vLLM dependency is compatible with the workload.

The first compatibility gate

The first check should answer one question: are you running the environment that the recipe describes?

The Kimi K3 recipe currently recommends the vllm/vllm-openai:kimi-k3 container. It identifies the build as cu130 and sets r580 or newer as the host requirement. NVIDIA’s CUDA 13 documentation also places CUDA 13.x compatibility in the r580-and-newer driver family.

Run checks that expose the full chain, not just the host driver:

nvidia-smi

docker inspect vllm/vllm-openai:kimi-k3 \
  --format '{{.RepoTags}} {{.Id}}'

docker run --rm --gpus all \
  vllm/vllm-openai:kimi-k3 \
  python -c "import torch; print(torch.__version__); print(torch.version.cuda)"

Then classify the result:

1. cu130 plus r575: this is a hard compatibility decision. Stop tuning batch size and context length.
2. cu130 plus r580 or newer: continue with image integrity, model revision, GPU visibility, and topology checks.
3. Custom cu129 build: record the source commit, PyTorch build, compiler, and kernel package versions. You now own the dependency chain.
4. Unknown image or floating tag: replace it with a pinned image digest before troubleshooting further.

A CUDA minor-version compatibility rule does not automatically make a major-version jump safe. NVIDIA documents CUDA 13.x as requiring the r580 family for the relevant compatibility path. It also warns that newer toolkit features can require newer driver features, even where a limited compatibility mode exists. See the NVIDIA CUDA compatibility guide for the distinction.

The same-day recovery choice

Your choice should follow the change boundary, not personal preference.

Path A: Upgrade the driver

This is the preferred production direction when the failure is conclusively cu130 versus r575.

Before the maintenance window, confirm:

• Every target GPU model is supported by the proposed driver branch.
• The container runtime can still expose GPUs after the change.
• Existing CUDA 12 workloads have a tested rollback path.
• NCCL, RDMA, storage, monitoring, and scheduling agents have a maintenance procedure.
• The cluster can drain or cordon nodes without interrupting protected workloads.
• The platform owner has a clear success and failure threshold.

NVIDIA’s driver documentation states that newer drivers maintain backward compatibility with applications built against older CUDA toolkits, but architecture support, library behavior, and workload-specific regressions still require testing. Use the NVIDIA driver and CUDA architecture matrix when checking existing workloads.

Advantages

• Aligns the host with the official Kimi K3 cu130 recipe.
• Reduces the amount of custom build maintenance.
• Gives you a cleaner long-term support story.
• Avoids treating a temporary cu129 fork as a production platform.

Risks

• A driver change is a node-level or cluster-level event.
• Existing workloads may depend on a specific branch, kernel module, or runtime behavior.
• The Kimi K3 engine can still fail later because of communication or capacity issues.

Path B: Rebuild for cu129

This path is not a free compatibility switch. It is a software maintenance project.

The official recipe identifies building from the K3 branch against cu129 PyTorch as an alternative for r575 hosts. That does not mean the result has the same validation status as the official container.

Use this path only if you can maintain:

• A pinned vLLM source revision.
• A reproducible PyTorch and CUDA build.
• FlashInfer and related kernel dependencies.
• A repeatable image build.
• Regression tests for model loading, tool calling, multimodal input, and long prompts.
• A process for rebuilding when the K3 branch changes.

Advantages

• Avoids immediate host-driver changes.
• Can fit a controlled development or pre-production workflow.
• Preserves an r575 node pool for workloads that cannot move yet.

Risks

• Build failures can replace runtime failures.
• A passing local build may not survive a clean rebuild.
• Kernel and dependency drift becomes your responsibility.
• The custom path may diverge from future official recipes.

Do not use a locally successful build as evidence that the shared cluster is ready for production. Treat it as a temporary compatibility branch until you have reproducible builds and regression results.

Path C: Switch environments

Use an isolated environment when the application deadline is closer than the platform change window.

An isolated environment should have its own:

• Driver-compatible GPU nodes.
• Container image and digest.
• Request endpoint.
• Log retention.
• Access controls.
• Cost or lease boundary.
• Rollback point.

This route is especially useful when an Agent team needs to validate tool calls, structured output, or multimodal workflows while the platform team reviews a driver change. You can keep application work moving without silently changing the shared cluster.

If you use the MacHTML console, keep the environment record explicit: image, node type, start time, test owner, and shutdown condition. The goal is not to hide the infrastructure decision. The goal is to isolate it.

The change-window control

A driver upgrade should be handled like a production change, even when the immediate goal is a test.

The GPU container runtime is part of the compatibility chain. Before changing the host driver, verify that your installed NVIDIA Container Toolkit version and runtime configuration follow the official NVIDIA Container Toolkit installation guide. A correct driver with a broken container runtime can produce a misleading “CUDA unavailable” failure.

Use a staged rollout:

1. Select one isolated node.
2. Drain it from normal scheduling.
3. Record the old driver package, kernel module state, and node labels.
4. Apply the proposed driver change according to your platform’s official procedure.
5. Reboot only within the approved window.
6. Run GPU visibility and container smoke tests.
7. Start Kimi K3 with the smallest supported serving command.
8. Run application-level requests.
9. Repeat on a second node if the first node passes.
10. Expand only after the baseline workloads pass.

A rollback trigger should be written before the change. Examples include failure of a protected workload, missing GPU devices, broken RDMA registration, inconsistent driver versions across nodes, or a Kimi K3 failure that cannot be separated from the platform change.

Experience rule: if the shared cluster cannot tolerate one-node experimentation, do not perform the first driver change on the shared cluster. Move the Kimi K3 validation to an isolated environment instead.

Use the MacHTML help center to keep the handoff record separate from the application team’s code changes. The infrastructure owner should be able to explain what changed without asking the Agent team to reconstruct the incident from chat messages.

The post-start cache check

A successful process start does not prove that migration succeeded.

The current vLLM Kimi K3 launch example explicitly includes --enable-prefix-caching. The vLLM release post also explains that Kimi K3 uses hybrid caching because its architecture combines recurrent KDA state with full-attention cache behavior.

Start with a controlled request pair:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Then send two requests with:

• The exact same system prompt.
• The exact same tool definitions.
• The same initial user prefix.
• A different final user instruction.
• Enough repetition to observe a cache decision.
• Captured request IDs and server logs.

If prefix caching appears inactive, check three separate causes:

1. Parameter cause: the flag was omitted, overridden, or not present in the actual process command.
2. Request cause: the prompts differ in whitespace, tool schema, system content, or message order.
3. Retention cause: the relevant KDA state was not retained at the boundary your workload reuses.

The vLLM release post describes prompt-end retention, interval-based retention, and selective retention behavior for hybrid models. It notes that retention is a policy decision, not a guarantee that every token position becomes a reusable checkpoint.

Do not upgrade the driver again because the cache hit rate is low. First prove that the requests share a prefix and that the cache policy can retain it.

The first capacity and communication pass

After startup and cache validation, increase load in controlled stages.

• Stage one: model load only.
• Stage two: one short text request.
• Stage three: one request at the target context range.
• Stage four: repeated identical prefixes.
• Stage five: the expected concurrent request count.
• Stage six: tool calling and structured output.
• Stage seven: multimodal input if your application needs it.
• Stage eight: cross-node traffic and the selected all-to-all backend.

The official recipe documents deepep_v2 for RDMA and flashinfer_nvlink_one_sided for NVLink. It also lists environment-specific settings for MNNVL systems and a fallback for certain mlx5 DMA-BUF registration failures. These are topology and kernel-path concerns, not proof that the driver upgrade was wrong.

Record each stage separately:

An OOM after migration does not automatically disprove the driver path. It may indicate model capacity, context length, concurrency, cache retention, or an unsuitable parallelism layout. Keep those failure chains separate.

The decision record after observation

Do not leave the temporary environment running indefinitely because nobody wants to make the final call.

At the end of the observation period, record four things:

• Whether the original startup failure is reproducible.
• Whether prefix caching behaves as expected for the real request pattern.
• Whether Agent calls produce correct tool and structured outputs.
• Whether the new environment creates more maintenance work than the shared platform can support.

Use this decision rule:

• Keep the upgraded path if the driver change passes node, workload, communication, and application regression.
• Keep the isolated path if it restores validation and the shared cluster still cannot accept the change.
• Keep the custom cu129 path only as a controlled transition if the team can reproduce the build and owns its dependency updates.
• Roll back if the change breaks protected workloads, produces inconsistent node behavior, or cannot be explained within the approved window.

Before closing the incident, complete this checklist:

• [ ] The image tag and digest are recorded.
• [ ] The host driver branch is recorded.
• [ ] The first failing subsystem is identified.
• [ ] The selected recovery path has an owner.
• [ ] The rollback trigger is written down.
• [ ] One isolated smoke test has passed.
• [ ] Model loading and short requests have passed.
• [ ] Prefix caching was tested with repeated identical prefixes.
• [ ] Agent tool calls and structured outputs were checked.
• [ ] Cross-node communication was tested if required.
• [ ] The final environment decision has an expiry or review date.

For teams that need to continue Kimi K3 and Agent integration while the shared GPU cluster remains frozen, a disposable and isolated environment is often the cleaner short-term choice. It avoids three common costs of the current setup: blocked change approvals, shared-node blast radius, and unclear rollback ownership. MacHTML can be considered when you need a temporary environment to restore validation first, then decide whether a permanent driver migration is justified. Review the available MacHTML environments before committing the workload to a long-term architecture.

FAQ