How to serve LLMs in production: a guide to GPU memory, KV cache, and sizing

TL;DR

Qwen3.5-4B in FP8 needs roughly 4.9 GiB for weights and 16 KiB per token of KV cache, plus a fixed recurrent state of about 48 MiB per sequence. On a 24 GB L4 that supports somewhere around 47 concurrent users at 16k context. MoE models like Qwen3.6-35B-A3B do not save weight memory, only compute, but they can hold a much smaller KV cache per token.

For multi-user production serving, use vLLM, SGLang, or TensorRT-LLM. vLLM achieves 2-4x higher throughput than prior systems like FasterTransformer and Orca through its paged attention design. Do not default to Ollama or llama.cpp for multi-user production: both are built for low concurrency, not the continuous batching multi-user serving needs.

What is GPU VRAM and why does it matter for serving LLMs?

GPU VRAM (Video RAM) is the high-bandwidth memory on a GPU. It is the only memory the GPU can read from at full speed during inference. When serving an LLM, three things compete for VRAM:

1. Model weights: the parameters of the model. Loaded once at startup. Fixed size.
2. KV cache: the model's internal state for each active conversation. Grows linearly with conversation length, per user.
3. Inference engine overhead: CUDA kernels, scheduler state, activation memory. Typically 1-2 GB.

If the sum exceeds VRAM, requests cannot be served. There is no swap-to-RAM equivalent that maintains performance.

According to vLLM's blog, inference systems before paged attention wasted 60% to 80% of KV cache memory due to fragmentation and over-reservation. This is why modern serving engines have become essential rather than optional.

What GPUs are commonly used for LLM serving?

The GPUs commonly used for production LLM serving span from the aging T4 up through the frontier-class B200. Each has different VRAM capacity, memory bandwidth, and numeric-format support.

The B200 is named at 180 GB, not 192 GB — the 192 GB figure traces to the original Blackwell announcement and other SKUs. T4 bandwidth is genuinely inconsistent in NVIDIA's own docs (product brief 320 GB/s, datasheet 300 GB/s), hence the range.

VRAM determines total capacity. Compute capability determines which numeric formats and kernel optimisations (BF16, FP8, FlashAttention) are supported. Older GPUs like the T4 are limited to older, slower code paths.

How much memory does an LLM's weights occupy?

Model weight memory equals parameter count multiplied by bytes per parameter. Three common precisions:

• BF16: 2 bytes per parameter
• FP8: 1 byte per parameter
• INT4 (AWQ/GPTQ): 0.5 bytes per parameter

A 4 billion parameter model in BF16 occupies 8 GB. The same model in FP8 occupies 4 GB. In INT4, approximately 2.5 GB.

This cost is your basic starting fixed cost before you start serving any users.

Weight sizes for common LLMs

INT4 figures include roughly 15% overhead for quantization scales and zero-points. Treat the MoE row as a floor, since AWQ typically leaves embeddings and vision layers unquantized.

Qwen3.6 is the current family but its open-weight releases start at 27B. The full Qwen3.6 open-weight lineup is exactly four repos, all Apache 2.0: Qwen3.6-27B, Qwen3.6-27B-FP8, Qwen3.6-35B-A3B, Qwen3.6-35B-A3B-FP8. Qwen3.5-4B and 9B remain the current Qwen models at small sizes, which is why the worked example below uses one.

Do Mixture of Experts (MoE) models save GPU memory?

No. MoE models save compute, not memory. A model named "35B-A3B" has 35 billion total parameters but only 3 billion active per token. All 35 billion parameters must be loaded into VRAM because any expert may be needed for the next token.

Memory cost: dominated by total parameters (70 GB in BF16 for a 35B model).

Compute cost: dominated by active parameters. Faster than a dense 35B model.

KV cache: set by the attention configuration, not the expert count. Qwen3.6-35B-A3B uses only 2 key-value heads and holds 20 KiB per token, less than the 4B model's 32 KiB. Weight footprint and KV footprint scale independently.

A common mistake is provisioning a GPU based on the active parameter count. An engineer expecting Qwen3.6-35B-A3B to fit on a 24 GB L4 will find the model fails to load: it needs 35 GB just for the weights in FP8.

What is the KV cache in an LLM?

The KV cache (Key-Value cache) is the model's internal mathematical state for every token in an active conversation. It comes from the attention mechanism, where each token's key and value vectors must be retained so future tokens can attend to them.

The KV cache grows linearly with conversation length. Every new token adds a fixed amount of memory per user. vLLM's blog cites LLaMA-13B as an example: a single sequence's KV cache can occupy up to 1.7 GB of GPU memory.

How large is the KV cache per token?

KV cache size per token depends on the model's architecture (number of layers, attention heads, head dimension, and whether it uses Grouped Query Attention) — and, for hybrid models like Qwen3.5 and Qwen3.6, on how many of those layers actually hold a per-token cache at all. Qwen3.5 and Qwen3.6 use a hybrid attention pattern: 3 Gated DeltaNet (linear attention) layers followed by 1 full-attention layer (full_attention_interval: 4). Only the full-attention layers hold a KV cache that grows with context; the other three hold a fixed-size recurrent state that does not grow at all. The correct formula is 2 × full_attention_layers × num_key_value_heads × head_dim × bytes_per_element — computing KV cache as if every layer were full attention overstates it by roughly 4x.

Per-token figures come from each model's config.json using 2 × full-attention layers × KV heads × head dimension × bytes. The recurrent state figures assume the FP32 dtype the Qwen configs declare (mamba_ssm_dtype: float32) for the linear-attention layers. If your engine holds that state in BF16 they halve. A small causal-convolution state of 1 to 4 MiB per sequence is not included. These are also no-MTP figures: all four configs carry mtp_num_hidden_layers: 1, and enabling speculative decoding adds the draft layer's own KV cache on top.

For Qwen3.5-4B in FP8, this means:

• 1,000 tokens of context = 16 MiB
• 16,000 tokens of context = 250 MiB
• 24,000 tokens of context = 375 MiB

Per user, plus the fixed ~48 MiB recurrent state per sequence. The KV cache size for 16 concurrent users at 24k context each is approximately 6.6 GiB.

This is why "16 concurrent users" is not a complete specification without also stating context length.

How do you calculate VRAM needed to serve an LLM?

The formula for estimating VRAM requirements:

VRAM ≈ weights + 2 GB overhead + (peak_concurrent × p95_context × kv_per_token) × 1.15

The 1.15 factor is a safety margin for activation spikes and engine overhead variance.

Worked example: Qwen3.5-4B on an L4 (24 GB)

Everything here is in GiB, because mixing GB and GiB is an easy way to overstate capacity by several users. Engine overhead is a typical measured range, not a specification: read your own figure from vLLM's startup log line reporting GPU KV cache size. Qwen ships no FP8 checkpoint below 27B, so the 4.9 GiB assumes you quantized the model yourself. Note also that the parameter count is 4.66B rather than a round 4B, and includes a vision tower.

Treat these counts as worst-case steady state, not a capacity guarantee. vLLM pages KV on demand rather than reserving each user's maximum context, so real headroom is usually better.

Running the same model in BF16 instead pushes weights to roughly 8.7 GiB and doubles per-user KV to about 0.54 GiB, which brings the same card down to somewhere around 18 concurrent users at 16k context.

Switching to Qwen3.5-9B roughly doubles the weight cost. Per-token KV is identical between the two models, since they share the same attention configuration, so the entire difference in capacity comes from weights eating the KV pool.

The same hardware has very different capacity depending on the model and precision chosen.

What is paged attention?

Paged attention is a memory management technique introduced by Kwon and colleagues at SOSP 2023 alongside the vLLM serving engine. It divides KV cache memory into fixed-size blocks that map to non-contiguous physical memory, borrowing ideas from virtual memory and paging in operating systems. Blocks are allocated dynamically as conversations grow, replacing the older approach of reserving maximum-possible memory per user upfront.

Why paged attention matters

Without paged attention, an engine serving 16 users at 32k maximum context would reserve 16 × 32k worth of memory at all times, even if no user had used that much yet. This wastes 60-80% of VRAM in typical workloads (the same range vLLM's blog reports).

With paged attention, a user with 2,000 tokens consumes approximately 125 blocks. A user with 16,000 tokens consumes approximately 1,000 blocks. The total pool is shared across all users.

The vLLM paper reported 2-4x higher throughput than FasterTransformer and Orca on equivalent workloads, with gains growing as sequence lengths increased. LMSYS, the team behind Chatbot Arena, cut GPU count by 50% while serving up to 5x more traffic after migrating to vLLM. Paged attention is the primary reason vLLM and SGLang outperform older serving frameworks.

What is continuous batching?

Continuous batching is a scheduling technique where new requests join the active batch as soon as a slot becomes available, rather than waiting for a fixed batch window. Old requests leave the batch as they complete.

This means the GPU is never idle waiting for a batch to fill, and new requests never wait for an arbitrary scheduling tick.

Continuous batching is standard in vLLM, SGLang, and TensorRT-LLM.

What is prefix caching and when does it help?

Prefix caching is a technique where KV cache blocks are reused across requests that share the same starting prefix. If 1,000 requests all share the same 2k system prompt, the KV for that prompt is computed once and reused 1,000 times.

When prefix caching helps

• Agent backends with stable system prompts and tool definitions
• Chatbots with a fixed persona
• RAG systems where retrieved chunks are stable
• Multi-turn conversations (each turn reuses the prefix from prior turns)

When prefix caching does not help

• Requests with fully unique prefixes
• Workloads where the prefix changes per request
• Single-shot generation with random prompts

Prefix caching in vLLM

Prefix caching is enabled by default in vLLM's V1 engine. There is no flag to add. Disable it with --no-enable-prefix-caching if your workload has no shared prefixes, since it costs throughput when nothing can be reused. Cache hit rates appear in the engine logs.

Prefix caching can reduce TTFT (Time To First Token) by 4x to 8x on cacheable workloads with zero code changes. Anthropic exposes a user-controlled version of this through its API as prompt caching, via the cache_control parameter. OpenAI applies prompt caching automatically for prompts above 1,024 tokens, with similar effect but no manual control.

What happens when a GPU runs out of memory during inference?

A well-configured inference engine handles overflow through a graceful cascade. A misconfigured one crashes.

OOM (Out Of Memory) crashes are almost always a configuration error, not a workload problem. The two most common causes:

1. --max-num-seqs is set higher than the KV pool can support at realistic context length
2. --gpu-memory-utilization is set too aggressively (above 0.95)

Engines should queue or swap. Never crash.

Which inference engine should you use for serving LLMs?

The right inference engine depends on whether you are serving one user (local development) or many concurrent users (production).

Multi-user production engines

Low-concurrency engines

Pointing 16 concurrent users at Ollama or llama.cpp will cause requests to queue serially. Latency degrades rapidly. This is the most common early mistake in LLM serving deployments.

vLLM flags and deployment sizing

The full vLLM flag reference and the deployment sizing checklist live in Part 4, the LLM serving cheat sheet. This post covers the concepts they rest on rather than repeating the tables.

Common GPU sizing mistakes

Mistake 1: Sizing for average instead of peak concurrency

Average concurrency does not capture the worst moment. Peak concurrency dictates VRAM requirements.

Mistake 2: Sizing for average context length

P95 context length, not average, drives KV pressure. Long-tail users are the ones that cause OOM.

Mistake 3: Ignoring prefix overlap

Prefix caching can reduce effective per-user KV by 50-90% when prompts share stable prefixes. Sizing without accounting for this overestimates GPU requirements.

Mistake 4: Treating MoE active parameters as memory cost

MoE models occupy VRAM equal to their total parameter count, not active. A 35B-A3B model needs 70 GB in BF16, not 3 GB.

Mistake 5: Setting --gpu-memory-utilization above 0.95

Leaves no headroom for activation spikes during prefill. Causes OOM under moderate load.

Frequently asked questions

Can I serve Qwen3.5-9B on an L4 GPU?

Yes, for low concurrency. Qwen3.5-9B in BF16 occupies 18 GB of weights on an L4's 24 GB VRAM. After engine overhead (2 GB), approximately 4 GB remain for KV cache. At 16k context, this supports approximately 3 concurrent users. Using FP8 weights frees 9 GB and supports approximately 10 concurrent users at 16k context.

Why does my LLM server OOM under moderate load?

The three most likely causes:

1. --max-num-seqs set higher than the KV pool can support at realistic context length
2. --gpu-memory-utilization set above 0.95, leaving no headroom for activation spikes
3. --max-model-len allowing context lengths that do not fit at peak concurrency

Is vLLM better than Ollama for production?

For multi-user production, yes. vLLM implements paged attention and continuous batching, which Ollama does not. Pointing more than a few concurrent users at Ollama causes requests to queue serially and latency to degrade rapidly. The vLLM project documents 2-4x throughput improvements over alternative serving systems.

Ollama is appropriate for local development, demos, and low-concurrency workloads.

Does FP8 quantization hurt model quality?

FP8 W8A8 (weights and activations) is effectively lossless: roughly 0.6 percentage points on MMLU-Pro, inside typical eval noise. FP8 KV cache is the lossier piece — it shows a measurable effect at very long context, though usually still small enough to be imperceptible in most production workloads. FP8 requires a compute capability ≥ 8.9 GPU (L4, L40S, H100, H200, B200). It is not supported on A100.

How do I reduce GPU memory usage when serving LLMs?

In order of impact:

1. Quantize weights to FP8 (cuts weight memory by 50%) or INT4 (cuts by 75%)
2. Enable FP8 KV cache (cuts KV memory by 50% on L4 or H100)
3. Reduce --max-num-seqs to a realistic concurrency level
4. Reduce --max-model-len to actual P99 context, not model maximum
5. Use a smaller model if quality permits

Further reading

• CNCF, 2026. "Kubernetes Established as the De Facto Operating System for AI as Production Use Hits 82% in 2025 CNCF Annual Cloud Native Survey." 82% measures container users running Kubernetes in production overall; the AI-specific figure in the same survey is 66% of organizations running generative AI using Kubernetes for inference. CNCF Annual Survey announcement
• CNCF TAG Runtime. "Cloud Native AI White Paper." https://tag-runtime.cncf.io/wgs/cnaiwg/whitepapers/cloudnativeai/

Series navigation

• Part 1: How to serve LLMs in production (this post)
• Part 2: Anatomy of an LLM request
• Part 3: Building LLM applications
• Part 4: LLM serving cheat sheet
• Part 5: Common mistakes when serving LLMs in production

Related reading

• The Future Economics of Enterprise AI Deployment