# What do context length, RoPE and YaRN mean, and how much context do you really get?

> Context length is how many tokens a model can consider at once. Models encode position with RoPE, and methods such as YaRN stretch it beyond the trained length: Qwen3-30B-A3B is 32,768 tokens native and 131,072 with YaRN. Your runtime's setting, and the KV-cache memory it needs, decide what you actually get.

Canonical: https://thelocalbench.com/learn/context-length · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0

Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder

Reviewed 2026-10-13 · REV 2026.10

## The plain version

A model's **context** is everything it can "see" while writing the next token: the system prompt, your messages, its replies, any pasted documents. It's measured in tokens (roughly ¾ of an English word each). When a conversation outgrows the context, the oldest parts fall off, or the app trims them.

**Three different numbers get called "context length":**

1. **Trained (native):** what the model saw during training. Qwen3-30B-A3B: 32,768 tokens.
2. **Extended:** what the model card says it supports with **RoPE scaling**. Qwen3-30B-A3B with YaRN: 131,072.
3. **Configured:** what your runtime actually allocated. Ollama picks this from your VRAM, and below 24 GiB it's 4K tokens.

The third number is the one you get, and it's usually the smallest.

**What's RoPE?** Transformers need to know word *order*. Most modern models use *rotary position embeddings* (RoPE): each position rotates parts of the attention vectors by angles that grow with distance, so the model can tell how far apart two tokens are. Past the trained length, the model meets rotation angles it has never seen, and quality collapses.

**What's YaRN?** RoPE scaling methods squeeze longer sequences into the angle range the model knows. Simple "linear" scaling compresses everything evenly, which blurs fine-grained position. YaRN compresses the slow-rotating dimensions more and the fast ones less, and adjusts attention temperature. Its paper reports reaching long contexts with 10× fewer training tokens than earlier methods. Many models ship already trained for it, and you switch it on at load time.

<RopeDiagram />

**An everyday analogy.** A ruler marked to 30 cm. Linear scaling relabels it as 120 cm, with every mark four times further apart, so you lose precision everywhere. YaRN keeps the millimetre marks sharp and stretches only the coarse ones.

> **Why it matters for your rig**
>

Context costs memory: every token adds to the [KV cache](/learn/kv-cache). On a 24 GB card, a 4-bit 30B-class model leaves a few gigabytes for it, which is plenty for 32K but tight for 128K unless the model uses heavy GQA or you quantize the cache. Long prompts also cost time to first token. And leaving YaRN on when you only need short contexts can slightly hurt quality, which is why Qwen recommends enabling it only when needed.



**Long context isn't long memory.** Even within their window, models recall facts from the middle of very long prompts less reliably than from the beginning or the end. For large document collections, [RAG](/learn/rag-and-embeddings) usually beats pasting everything in.

## The technical layer

### RoPE in one line

For a pair of dimensions *i* at position *m*, rotate by θᵢ·m with θᵢ = base^(−2i/d). Linear scaling divides *m* by factor *s*. YaRN interpolates per dimension according to wavelength, and scales attention logits by a temperature that grows with *s*.

### Turning it on in llama.cpp

```
llama-server -m Qwen3-30B-A3B-Q4_K_M.gguf -c 131072 \
  --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768 --port 8080
```

`-c` sets the allocated context (0 means take it from the model). `--rope-scale 4` expands by 4×, and `--yarn-orig-ctx` is the native length. Many GGUFs carry the right settings in their metadata, so try plain `-c` first.

### Ollama

```
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
```

Or set `num_ctx` per request or in a Modelfile. Check what was really allocated, and whether the model spilled to the CPU, with `ollama ps`.

### Memory per token

```
KV_bytes/token = 2 × layers × kv_heads × head_dim × bytes_per_element
```

Qwen3-30B-A3B (48 layers, 4 KV heads, head_dim 128, f16): 2 × 48 × 4 × 128 × 2 ≈ 96 KB per token, or about 12 GB at 128K. With `-ctk q8_0 -ctv q8_0` it's about half that.

### Pitfalls

- **Silent truncation.** Many apps drop old messages without telling you. If a model "forgets" the start of a document, check the configured context first.
- **Sliding-window models.** Gemma 3 uses local attention on most layers, so its cache grows more slowly, but those layers can only see recent tokens.
- **Don't scale past the card.** A factor beyond what the model was trained for produces fluent nonsense.

## Sources

- [Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2104.09864)](https://arxiv.org/abs/2104.09864)
- [Peng et al., "YaRN: Efficient Context Window Extension" (arXiv 2309.00071)](https://arxiv.org/abs/2309.00071)
- [Qwen3-30B-A3B model card: native 32,768, 131,072 with YaRN](https://huggingface.co/Qwen/Qwen3-30B-A3B)
- [llama.cpp server README: -c, --rope-scaling, --rope-scale, --yarn-orig-ctx](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- [Ollama docs: context length defaults](https://docs.ollama.com/context-length)

Try it: https://thelocalbench.com/configurator?mode=hardware&hw=rtx-3090-24g-used