How to serve LLMs in production: a guide to GPU memory, KV cache, and sizing

TL;DR

Serving an LLM on a GPU requires three things to share VRAM: the model weights (fixed cost), the KV cache (grows with conversation length per user), and the inference engine's overhead. Total VRAM needed is approximately:

VRAM ≈ weights + 2 GB overhead + (peak_concurrent × p95_context × kv_per_token) × 1.15

A Qwen3.5-4B model in BF16 needs 8 GB for weights and approximately 74 KB per token of KV cache. On a 24 GB L4 GPU, this supports approximately 12 concurrent users at 16k context length. MoE (Mixture of Experts) models like Qwen3.5-35B-A3B do not save memory; only compute.

For multi-user production serving, use vLLM, SGLang, or TensorRT-LLM. vLLM achieves 2-4x higher throughput than prior systems like FasterTransformer and Orca through its paged attention design. Do not use Ollama or llama.cpp in production: they are single-user serving engines.

What is GPU VRAM and why does it matter for serving LLMs?

GPU VRAM (Video RAM) is the high-bandwidth memory on a GPU. It is the only memory the GPU can read from at full speed during inference. When serving an LLM, three things compete for VRAM:

1. Model weights: the parameters of the model. Loaded once at startup. Fixed size.
2. KV cache: the model's internal state for each active conversation. Grows linearly with conversation length, per user.
3. Inference engine overhead: CUDA kernels, scheduler state, activation memory. Typically 1-2 GB.

If the sum exceeds VRAM, requests cannot be served. There is no swap-to-RAM equivalent that maintains performance.

Before vLLM's paged attention design, existing inference systems wasted 60% to 80% of KV cache memory due to fragmentation and over-reservation. This is why modern serving engines have become essential rather than optional.

What GPUs are commonly used for LLM serving?

The four most common GPUs for production LLM serving are the T4, L4, A100, and H100. Each has different VRAM capacity and supports different numeric formats.

VRAM determines total capacity. Compute capability determines which numeric formats and kernel optimisations (BF16, FP8, FlashAttention) are supported. Older GPUs like the T4 are limited to older, slower code paths.

How much memory does an LLM's weights occupy?

Model weight memory equals parameter count multiplied by bytes per parameter. Three common precisions:

• BF16: 2 bytes per parameter
• FP8: 1 byte per parameter
• INT4 (AWQ/GPTQ): 0.5 bytes per parameter

A 4 billion parameter model in BF16 occupies 8 GB. The same model in FP8 occupies 4 GB. In INT4, approximately 2.5 GB.

This cost is your basic starting fixed cost before you start serving any users.

Weight sizes for common LLMs

Do Mixture of Experts (MoE) models save GPU memory?

No. MoE models save compute, not memory. A model named "35B-A3B" has 35 billion total parameters but only 3 billion active per token. All 35 billion parameters must be loaded into VRAM because any expert may be needed for the next token.

Memory cost: dominated by total parameters (35 GB in BF16 for a 35B model).

Compute cost: dominated by active parameters. Faster than a dense 35B model.

KV cache: similar to a dense model of comparable shape. MoE affects feed-forward layers, not attention.

A common mistake is provisioning a GPU based on the active parameter count. An engineer expecting Qwen3.5-35B-A3B to fit on a 24 GB L4 will find the model fails to load: it needs 35 GB just for the weights in FP8.

What is the KV cache in an LLM?

The KV cache (Key-Value cache) is the model's internal mathematical state for every token in an active conversation. It comes from the attention mechanism, where each token's key and value vectors must be retained so future tokens can attend to them.

The KV cache grows linearly with conversation length. Every new token adds a fixed amount of memory per user. For LLaMA-13B, a single sequence's KV cache can occupy up to 1.7 GB of GPU memory.

How large is the KV cache per token?

KV cache size per token depends on the model's architecture (number of layers, attention heads, head dimension, and whether it uses Grouped Query Attention).

For Qwen3.5-4B in FP8, this means:

• 1,000 tokens of context = 74 MB
• 16,000 tokens of context = 1.17 GB
• 24,000 tokens of context = 1.75 GB

Per user. The KV cache size for 16 concurrent users at 24k context each is approximately 28 GB.

This is why "16 concurrent users" is not a complete specification without also stating context length.

How do you calculate VRAM needed to serve an LLM?

The formula for estimating VRAM requirements:

VRAM ≈ weights + 2 GB overhead + (peak_concurrent × p95_context × kv_per_token) × 1.15

The 1.15 factor is a safety margin for activation spikes and engine overhead variance.

Worked example: Qwen3.5-4B on an L4 (24 GB)

Switching weights to FP8 frees approximately 4 GB, supporting approximately 15 concurrent users at 16k context.

Switching to a 9B model in BF16 increases weight cost to 18 GB, leaving only approximately 4 GB for the KV pool. This supports approximately 3 concurrent users at 16k context.

The same hardware has very different capacity depending on the model and precision chosen.

What is paged attention?

Paged attention is a memory management technique introduced by Kwon and colleagues at SOSP 2023 alongside the vLLM serving engine. It divides KV cache memory into fixed-size blocks that map to non-contiguous physical memory, borrowing ideas from virtual memory and paging in operating systems. Blocks are allocated dynamically as conversations grow, replacing the older approach of reserving maximum-possible memory per user upfront.

Why paged attention matters

Without paged attention, an engine serving 16 users at 32k maximum context would reserve 16 × 32k worth of memory at all times, even if no user had used that much yet. This wastes 80-90% of VRAM in typical workloads.

With paged attention, a user with 2,000 tokens consumes approximately 125 blocks. A user with 16,000 tokens consumes approximately 1,000 blocks. The total pool is shared across all users.

The vLLM paper reported 2-4x higher throughput than FasterTransformer and Orca on equivalent workloads, with gains growing as sequence lengths increased. LMSYS, the team behind Chatbot Arena, cut GPU count by 50% while serving 2-3x more requests per second after migrating to vLLM. Paged attention is the primary reason vLLM and SGLang outperform older serving frameworks.

What is continuous batching?

Continuous batching is a scheduling technique where new requests join the active batch as soon as a slot becomes available, rather than waiting for a fixed batch window. Old requests leave the batch as they complete.

This means the GPU is never idle waiting for a batch to fill, and new requests never wait for an arbitrary scheduling tick.

Continuous batching is standard in vLLM, SGLang, and TensorRT-LLM.

What is prefix caching and when does it help?

Prefix caching is a technique where KV cache blocks are reused across requests that share the same starting prefix. If 1,000 requests all share the same 2k system prompt, the KV for that prompt is computed once and reused 1,000 times.

When prefix caching helps

• Agent backends with stable system prompts and tool definitions
• Chatbots with a fixed persona
• RAG systems where retrieved chunks are stable
• Multi-turn conversations (each turn reuses the prefix from prior turns)

When prefix caching does not help

• Requests with fully unique prefixes
• Workloads where the prefix changes per request
• Single-shot generation with random prompts

How to enable prefix caching in vLLM

Add the flag --enable-prefix-caching when starting the vLLM server. Cache hit rates appear in the engine logs.

Prefix caching can reduce TTFT (Time To First Token) by 5x to 20x on cacheable workloads with zero code changes. Anthropic and OpenAI both expose user-controlled versions of this through their APIs as prompt caching, with similar effect.

What happens when a GPU runs out of memory during inference?

A well-configured inference engine handles overflow through a graceful cascade. A misconfigured one crashes.

OOM (Out Of Memory) crashes are almost always a configuration error, not a workload problem. The two most common causes:

1. --max-num-seqs is set higher than the KV pool can support at realistic context length
2. --gpu-memory-utilization is set too aggressively (above 0.95)

Engines should queue or swap. Never crash.

Which inference engine should you use for serving LLMs?

The right inference engine depends on whether you are serving one user (local development) or many concurrent users (production).

Multi-user production engines

Single-user engines

Pointing 16 concurrent users at Ollama or llama.cpp will cause requests to queue serially. Latency degrades rapidly. This is the most common early mistake in LLM serving deployments.

Essential vLLM flags for production serving

Full flag reference is available in the official vLLM documentation.

How to size a new LLM deployment: checklist

Before provisioning a GPU, answer these questions. Each one is a number, not an estimate.

1. Which model? Exact name and size (e.g., "Qwen3.5-9B," not "a 9B model").
2. Which precision? BF16, FP8, or AWQ-4bit.
3. Average concurrent in-flight requests? Not requests per second. Concurrent.
4. Peak concurrent? Realistic worst-moment count.
5. Average context length? Measured from real or representative traffic.
6. P95 context length? Long-tail users dominate KV pressure.
7. Prefix overlap? What fraction of each request is shared (system prompt, tools, RAG)?
8. Latency budget? P50 and P95 TTFT targets.

If the calculated VRAM exceeds the GPU, four levers are available: smaller model, lower precision, fewer concurrent users, shorter contexts, or a bigger GPU.

Common GPU sizing mistakes

Mistake 1: Sizing for average instead of peak concurrency

Average concurrency does not capture the worst moment. Peak concurrency dictates VRAM requirements.

Mistake 2: Sizing for average context length

P95 context length, not average, drives KV pressure. Long-tail users are the ones that cause OOM.

Mistake 3: Ignoring prefix overlap

Prefix caching can reduce effective per-user KV by 50-90% when prompts share stable prefixes. Sizing without accounting for this overestimates GPU requirements.

Mistake 4: Treating MoE active parameters as memory cost

MoE models occupy VRAM equal to their total parameter count, not active. A 35B-A3B model needs 35 GB in BF16, not 3 GB.

Mistake 5: Setting --gpu-memory-utilization above 0.95

Leaves no headroom for activation spikes during prefill. Causes OOM under moderate load.

Frequently asked questions

Can I serve Qwen3.5-9B on an L4 GPU?

Yes, for low concurrency. Qwen3.5-9B in BF16 occupies 18 GB of weights on an L4's 24 GB VRAM. After engine overhead (2 GB), approximately 4 GB remain for KV cache. At 16k context, this supports approximately 3 concurrent users. Using FP8 weights frees 9 GB and supports approximately 10 concurrent users at 16k context.

Why does my LLM server OOM under moderate load?

The three most likely causes:

1. --max-num-seqs set higher than the KV pool can support at realistic context length
2. --gpu-memory-utilization set above 0.95, leaving no headroom for activation spikes
3. --max-model-len allowing context lengths that do not fit at peak concurrency

Is vLLM better than Ollama for production?

For multi-user production, yes. vLLM implements paged attention and continuous batching, which Ollama does not. Pointing more than a few concurrent users at Ollama causes requests to queue serially and latency to degrade rapidly. The vLLM project documents 2-4x throughput improvements over alternative serving systems.

Ollama is appropriate for local development, demos, and single-user workloads.

Does FP8 quantization hurt model quality?

FP8 typically reduces benchmark scores by 1-3 percentage points compared to BF16. For most production workloads, this difference is not perceptible to users. FP8 also requires a compute capability ≥ 8.9 GPU (L4, H100). It is not supported on A100.

How do I reduce GPU memory usage when serving LLMs?

In order of impact:

1. Quantize weights to FP8 (cuts weight memory by 50%) or INT4 (cuts by 75%)
2. Enable FP8 KV cache (cuts KV memory by 50% on L4 or H100)
3. Reduce --max-num-seqs to a realistic concurrency level
4. Reduce --max-model-len to actual P99 context, not model maximum
5. Use a smaller model if quality permits

Further reading

• CNCF, 2026. "Kubernetes Established as the De Facto Operating System for AI as Production Use Hits 82%." CNCF Annual Survey announcement
• CNCF TAG Runtime. "Cloud Native AI White Paper." https://tag-runtime.cncf.io/wgs/cnaiwg/whitepapers/cloudnativeai/

Series navigation

• Part 1: How to serve LLMs in production (this post)
• Part 2: Anatomy of an LLM request
• Part 3: Building LLM applications
• Part 4: LLM serving cheat sheet
• Part 5: Common mistakes when serving LLMs in production

Related reading

• The Future Economics of Enterprise AI Deployment