# What is a KV cache, and why does it eat my VRAM? > The KV cache is memory a model uses to remember every token already in the conversation, and it grows with context length. In October 2026, Qwen3 32B needs 0.26 MB per token, so 32,768 tokens add 8.6 GB to its 19.8 GB of Q4_K_M weights: too much for a 24 GB card. Canonical: https://thelocalbench.com/learn/kv-cache · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0 Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder Reviewed 2026-10-12 · REV 2026.10 ## The plain version When a model writes a reply, it produces one token (roughly a word piece) at a time. To choose each new token it has to "look back" at everything before it: your prompt, the documents you pasted, and its own reply so far. That looking-back step is called **attention**. Attention works by comparing the new token with a short summary of every earlier token. Each summary has two parts, a **key** (what this token is about) and a **value** (what it contributes). Working those out is expensive, so the model computes them once per token and keeps them. That store is the **key-value cache**, or KV cache. **An everyday analogy.** Imagine a court stenographer who, instead of re-reading the whole transcript before every sentence, keeps an index card for each sentence already spoken. The cards make each new sentence fast to write. But the stack of cards grows with every sentence, and in a long trial the stack can get bigger than the rulebook on the desk. The rulebook is the model's weights. The cards are the KV cache. **Why it surprises people.** The weights are a fixed size: download a 19.8 GB file and that's what it takes. The KV cache isn't in the file. It starts empty and grows with every token in the conversation, for every user at once. A model that loads comfortably can run out of memory an hour later when a long document pushes the context to 32,000 tokens. **How big it gets.** For Qwen3 32B, every token costs about 0.26 MB of cache at the default 16-bit precision. That sounds small until you multiply: - 4,096 tokens (a long chat): about 1.1 GB - 32,768 tokens (a big document): about 8.6 GB - 131,072 tokens (a whole codebase, with YaRN context extension): about 34 GB, larger than the model itself > **Why it matters for your rig** > Qwen3 32B at Q4_K_M on a 24 GB card: 19.8 GB of weights plus 8.6 GB of cache at 32k context is **28.4 GB**, so it doesn't fit. Options: keep context at 8k (2.1 GB of cache, a tight fit), store the cache at 8 bits (4.3 GB at 32k, still too much, but a tight fit at 16k), or switch to **Gemma 3 27B**, whose design keeps its 32k cache to about 3.1 GB. *[Diagram: memory by context length — see the HTML page]* **Three things that keep the cache small**, and that explain why two models of the same size can need very different memory: 1. **Shared key-value heads (GQA).** Most current models let several attention "heads" share one set of keys and values. Qwen3 32B has 64 query heads but only 8 KV heads, an 8× saving built into the model. 2. **Short-memory layers (sliding window).** Some layers only look at the last few hundred or thousand tokens, so their part of the cache stops growing. Gemma 3 does this on five of every six layers. 3. **Lower-precision cache.** Runtimes can store the cache at 8 bits instead of 16, halving it with a very small quality cost. This is a setting you choose, separate from the model's quant. You don't control the first two; they're part of the model's architecture, which is why every datasheet on this site lists layers, KV heads and the sliding window, verified against the model's own `config.json`. ## The technical layer ### The formula For standard attention with GQA (brief §7.3 formula 2): ``` KV_bytes = 2 × n_layers × n_kv_heads × head_dim × ctx × concurrency × bytes_per_elem ``` The 2 is one tensor for keys and one for values. `bytes_per_elem` is 2 for F16, 1 for q8_0 and about 0.56 for q4_0 (4 bits plus a scale per block of 32). For Qwen3 32B (64 layers, 8 KV heads, head_dim 128): ``` 2 × 64 × 8 × 128 × 2 B = 262,144 B per token ≈ 0.262 MB × 32,768 tokens = 8.59 GB ``` ### Sliding-window layers Layers with a window `w` only cache `min(ctx, w)` tokens. Gemma 3 27B has 62 layers, 16 KV heads and head_dim 128, with `sliding_window = 1024` and `sliding_window_pattern = 6` (every sixth layer is global): 10 global and 52 local layers. At 32k context: ``` global: 10 × 32,768 × (2 × 16 × 128 × 2 B) = 2.68 GB local: 52 × 1,024 × (2 × 16 × 128 × 2 B) = 0.44 GB total ≈ 3.12 GB (vs 16.6 GB if every layer were global) ``` gpt-oss does the same with a 128-token window on alternating layers; at 128k context gpt-oss-120b's cache is about 4.8 GB. ### MLA and linear attention DeepSeek-style **multi-head latent attention** caches a compressed latent of `kv_lora_rank + qk_rope_head_dim` elements per layer per token, instead of `2 × n_kv_heads × head_dim`. Hybrid models with linear-attention or state-space layers keep a constant-size state for those layers. The configurator handles both cases with separate formulas. ### Runtime flags - **llama.cpp**: `-c 32768` sets context; `-ctk q8_0 -ctv q8_0` quantizes the cache (V-cache quantization needs `-fa on`); `--swa-full` disables the sliding-window saving (avoid unless a frontend needs it); `-np 4` splits the context across 4 parallel slots. - **Ollama**: `num_ctx` in the Modelfile or API; `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`. - **LM Studio**: context length slider; "K/V cache quantization" under advanced settings. - **vLLM**: `--max-model-len`, `--kv-cache-dtype fp8`; it pre-allocates a share of VRAM set by `--gpu-memory-utilization` and pages the cache in blocks. - **MLX**: `--max-kv-size` bounds a rotating cache; `mlx_lm` supports quantized KV via `--kv-bits`. ### Speed, not just capacity The cache is also *read* for every generated token, so a large cache slows decoding: `bytes_per_token = W_active + KV(ctx)`. That's why the datasheets show tok/s against context depth, and why a 32k-context run on a 24 GB card can be 20–25% slower than an empty-context benchmark suggests. ### Pitfalls - **Ollama's default context is small**; long prompts are silently truncated unless you raise `num_ctx`. - **Concurrency multiplies the cache.** Four users at 8k cost the same as one at 32k. - **q4_0 cache** saves more but can visibly hurt long-context recall on some models. Start with q8_0. ## Sources - [Qwen/Qwen3-32B config.json](https://huggingface.co/Qwen/Qwen3-32B/blob/main/config.json) - [Gemma 3 config.json (sliding_window, sliding_window_pattern)](https://huggingface.co/unsloth/gemma-3-27b-it/blob/main/config.json) - [Ainslie et al., GQA (2023)](https://arxiv.org/abs/2305.13245) - [DeepSeek-V2 paper (multi-head latent attention)](https://arxiv.org/abs/2405.04434) - [llama.cpp server and CLI options (-c, -ctk, -ctv, -fa)](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) Try it: https://thelocalbench.com/configurator?model=qwen3-32b&quant=Q4_K_M&ctx=32768