Kimi K3 vLLM OOM: Tune or Scale in 2026?

Last updated August 16, 2026. Version, hardware, driver, topology, and cache details were checked against the official Kimi K3 vLLM recipe. The official deployment post is cited in the relevant sections below.

The official recipe describes Kimi K3 as a 2.8-trillion-parameter MoE model with a 1,048,576-token context window. It lists CUDA 13, an R580-or-newer NVIDIA driver, and at least 8x GB300 GPUs for the NVIDIA path. (official Kimi K3 vLLM recipe)

Loading-stage OOM: do not start by lowering concurrency. Check the image, driver, hardware, and parallel topology first.

Runtime-stage OOM: reduce context pressure, concurrency, or cache retention, then replay the same request set. If the target workload still has no stable memory headroom, expand or migrate the serving environment.

This guide is for three groups:

• Researchers and POC teams checking whether Kimi K3 can initialize and return basic responses.
• Small Agent teams balancing context length, concurrent sessions, and prefix caching.
• Platform and production teams deciding whether to repair the current cluster, move to a supported topology, or add temporary capacity.

Separate weight-loading OOM from runtime OOM

A Kimi K3 vLLM OOM has two very different decision paths.

Read the log in order. Record the first memory failure. Note whether it occurs during model loading, graph capture, KV-cache allocation, prefill, or decode. Then check whether the server reports engine readiness. Finally, send one controlled request.

A process that starts successfully has only passed initialization. It has not passed capacity validation. Production evidence requires a returned request, repeated traffic, and enough memory margin for normal scheduling and recovery.

Kimi K3 uses 16 active experts out of 896 routed experts and combines Kimi Delta Attention with full-attention layers. The official vLLM material also describes a context window of up to one million tokens and a hybrid cache manager that handles recurrent state and paged KV blocks. (official Kimi K3 vLLM deployment post)

That design creates several hidden pressure points:

• Weight memory can fail before request scheduling starts.
• Long prompts can create a prefill spike even after initialization succeeds.
• Agent history and tool definitions can make the real prompt much larger than the user message.
• Repeated prefixes can benefit from caching, but retained state also consumes memory.
• Multi-node expert parallelism depends on the interconnect and backend, not simply the number of GPUs.

Can loading-stage OOM be fixed by lowering concurrency? Usually no. Concurrency affects request execution after the engine has initialized. If the process cannot place the weights or complete initialization, treat the incident as an environment or topology problem until the log shows otherwise.

For researchers and POC teams, prove execution before scale

A personal research setup or POC team has a limited goal. You need to verify the API, basic reasoning, tool calling, and perhaps multimodal input. You do not need to reproduce production throughput on the first attempt.

The official recipe points to the vllm/vllm-openai:kimi-k3 Docker image. It states that the image is CUDA 13-only and requires an R580-or-newer NVIDIA host driver. It also lists at least 8x GB300 GPUs for the NVIDIA path and at least 8x MI355X or MI350X hardware for the AMD path. (official Kimi K3 requirements)

Use this sequence:

1. Freeze the environment. Save the container digest, vLLM version, host driver, CUDA runtime, model revision, and launch command.
2. Confirm the topology. Do not infer support from aggregate VRAM. Match the hardware and parallel mode to the supported recipe.
3. Start with one API path. Avoid large synthetic batches, speculative decoding, and extra experiments until the base server reaches readiness.
4. Use a short representative prompt. This proves the request path. It does not prove long-context capacity.
5. Test tool calling separately. The official recipe warns that Kimi K3 can sometimes emit a tool-call format its parser does not expect. Validate the schema and retry path before declaring the POC complete. (official Kimi K3 deployment notes)
6. Record the stopping point. If the model cannot initialize on a supported topology, stop adding random memory flags. Move the test to a compliant environment.

A short prompt can hide the actual problem. It may not allocate enough KV state to reproduce a long-context failure. It may also avoid the tool schema, multimodal path, or repeated prefix pattern used by your application.

When should a POC team stop tuning? Stop when the OOM occurs during weight loading, the hardware is outside the supported path, or every workaround changes the test so much that it no longer represents the intended product. A POC should answer whether the workflow can run. It should not become an indefinite attempt to force an unsupported topology.

Use the MacHTML console to keep access notes, test records, and environment details separate from your development workstation. The important evidence remains the inference image, driver chain, hardware topology, and request result.

For small Agent teams, reduce pressure without destroying reuse

A small Agent service has a different memory pattern. Its requests often repeat:

• A fixed system prompt.
• Tool definitions.
• Repository or product context.
• Conversation history.
• A structured output format.

That repetition is why prefix caching matters. It is also why disabling every cache can produce a misleading result.

The official Kimi K3 deployment guidance says prefix caching is supported but disabled by default. You must explicitly enable it with --enable-prefix-caching. (official prefix caching guidance)

The official deployment post describes two retention approaches. Interval-based retention stores selected checkpoints. Prompt-end states are useful for multi-turn conversations. Selective retention can promote a prefix after repeated reuse instead of caching every one-off prompt. (official vLLM cache retention design)

Use the same request set before and after every change:

1. Capture real system prompts, tool definitions, typical history, output limits, and the longest normal request.
2. Run one request with prefix caching enabled.
3. Repeat the same prefixes to establish whether reuse occurs.
4. Lower concurrency while keeping prompts unchanged.
5. Lower the maximum context only if truncation or summarization is acceptable.
6. Adjust retention after measuring which prefixes are actually reused.
7. Restore one variable at a time and repeat the test.

Record peak memory, engine restarts, rejected requests, time to first token, output stability, and cache hits. Do not copy a memory value from another Kimi K3 deployment. Cache pressure depends on the hardware, software build, prompt distribution, sequence lengths, and topology.

What should you do when prefix caching causes memory pressure? Keep it when repeated prefixes are central to the real workload. First reduce retention or concurrency. Disable caching only when prefixes are mostly unique, the hit rate is low, or the memory cost damages the service more than recomputation does.

A one-request success is not a fix. A fix is a repeatable result from the same request distribution with stable memory and acceptable latency.

For platform teams, verify control over the full stack

An existing GPU cluster may look sufficient and still fail Kimi K3. The key question is whether you control every layer needed by the supported deployment path.

Check these areas:

• Host driver: the official recipe specifies R580 or newer for the CUDA 13 image.
• Container image: the Kimi K3 image is described as CUDA 13-only.
• GPU path: the recipe lists 8x GB300 or more for the NVIDIA path and 8x MI355X or MI350X or more for the AMD path.
• Interconnect: the recipe distinguishes RDMA and NVLink backends for cross-node communication.
• Node permissions: a container-only team may not be able to upgrade drivers, load RDMA modules, or alter NCCL settings.
• Image lifecycle: the deployment post says the current path depends on complex pre-release dependencies and recommends Docker images. (official vLLM deployment documentation)

Is self-building worthwhile when the cluster misses the Kimi K3 requirements? It can make sense when you control the host, compiler toolchain, driver, network, and maintenance window. It is a weak long-term option when you can modify only the container. The result may be a custom environment that breaks after the next vLLM, CUDA, or kernel change.

Document what your team can change:

• Container only.
• Container and launch flags.
• Host driver and kernel.
• Network and RDMA stack.
• Node placement and parallel topology.
• Hardware procurement.

If the team controls only the first two layers, repeated self-building may cost more than moving the validation to a compliant environment.

Use the MacHTML help center to document access ownership and operational handoffs, but keep the Kimi K3 decision tied to the supported GPU environment. A remote terminal cannot compensate for an incompatible driver or missing interconnect.

For production teams, target traffic decides whether to scale

Production capacity is not proven by an idle server. Define the workload the service must survive:

• Expected prompt and output lengths.
• Concurrent Agent sessions.
• Repeated system and tool prefixes.
• Multimodal request frequency.
• Required latency during normal and peak traffic.
• Restart and worker-loss recovery behavior.
• Whether prefill and decode compete for the same resources.

The official vLLM deployment post describes expert and data parallel serving across nodes, plus prefill/decode disaggregation. One validated example routes TEP8 prefill to DEP16 decode with NIXL used for KV transfer. (official Kimi K3 serving topology) This is a tested topology example, not a universal GPU calculator.

The same post reports 118 tokens per second without speculative decoding and 370 tokens per second with DSpark on 16 NVIDIA GB300 GPUs. Those figures are tied to the published benchmark topology and workload. They are not a guarantee for your cluster. (official Kimi K3 benchmark results)

Run production validation in five stages:

1. Load the weights and wait for engine readiness.
2. Send one request from the real distribution.
3. Sweep concurrency while preserving prompt shapes and output limits.
4. Replay multi-turn Agent sessions with repeated prefixes.
5. Inject a worker or node failure and record recovery behavior.

Scale when the target load remains unstable after reasonable reductions, when normal traffic leaves no reliable memory margin, or when queueing and recovery requirements cannot be met without breaking the product contract.

Do not scale because one unsupported request fails if your service intentionally rejects that request class. Do not call the system healthy because a short synthetic prompt works when real Agent sessions use much longer histories.

How do you know an Agent service has reached the expansion point? You have reached it when the target request set requires permanent context or concurrency reductions, prefix retention must be disabled despite repeated prefixes, or normal traffic leaves no headroom for retries and recovery. That is a capacity decision, not merely a tuning issue.

Convert the retest into a tune, migrate, or scale decision

Use this checklist after every meaningful change:

• [ ] The failure stage is recorded from the vLLM log.
• [ ] Engine readiness is recorded.
• [ ] One controlled request returns successfully.
• [ ] The container digest and model revision are saved.
• [ ] The host driver and CUDA path are saved.
• [ ] GPU type, node count, parallel settings, and interconnect are saved.
• [ ] The request sample includes real system prompts and tool definitions.
• [ ] Context, concurrency, and cache settings are recorded.
• [ ] Peak memory and cache behavior are captured.
• [ ] The same request set is used before and after tuning.
• [ ] Production testing includes a recovery scenario.
• [ ] The next action has a written trigger: keep, migrate, or scale.

Classify the result:

• Tune and keep: the model loads on a supported topology, low-load validation passes, and the intended use case accepts the tested limits.
• Monitor and operate: reasonable reductions produce stable target behavior, but the service needs alerts for memory, queueing, cache reuse, and recovery.
• Migrate or scale: weights cannot load, the environment is unsupported, or production traffic leaves insufficient headroom.

A temporary comparison environment is safer than changing the only production cluster. Reproduce the same image, launch parameters, request sample, and measurement window on a compliant environment. Then determine whether the original failure came from compatibility, topology, or capacity.

Your current GPU cluster may appear cheaper, but it can carry hidden costs: driver coordination, restricted container privileges, network changes, procurement delays, and risky maintenance windows. MacHTML rental is not a replacement for the official Kimi K3 GPU topology, so do not use a rented Mac as evidence that Kimi K3 inference will fit. It can still support temporary Apple-side integration tests, API clients, CI control jobs, or an isolated validation workstation while the GPU decision is pending.

For that narrow use case, a MacHTML pricing option can be easier to schedule than modifying the only cluster. If you need actual Kimi K3 serving capacity, choose a compliant NVIDIA or AMD environment and validate the exact workload. If you need a temporary Mac environment around that workflow, rent it for the test window instead of buying hardware that will sit idle after the investigation.