Skip to main contentThe Local Bench

Explainer · REV 2026.10 · reviewed 2026-10-13 · 8 min

What do context length, RoPE and YaRN mean, and how much context do you really get?

Why "128K context" on a model card doesn't mean 128K in your app, how RoPE scaling stretches a model past its training, and what each extra token costs in memory and speed.

Discrete graphics card

The 50-word answer

Context length is how many tokens a model can consider at once. Models encode position with RoPE, and methods such as YaRN stretch it beyond the trained length: Qwen3-30B-A3B is 32,768 tokens native and 131,072 with YaRN. Your runtime's setting, and the KV-cache memory it needs, decide what you actually get.

The plain version

A model's context is everything it can "see" while writing the next token: the system prompt, your messages, its replies, any pasted documents. It's measured in tokens (roughly ¾ of an English word each). When a conversation outgrows the context, the oldest parts fall off, or the app trims them.

Three different numbers get called "context length":

  1. Trained (native): what the model saw during training. Qwen3-30B-A3B: 32,768 tokens.
  2. Extended: what the model card says it supports with RoPE scaling. Qwen3-30B-A3B with YaRN: 131,072.
  3. Configured: what your runtime actually allocated. Ollama picks this from your VRAM, and below 24 GiB it's 4K tokens.
The third number is the one you get, and it's usually the smallest.

What's RoPE? Transformers need to know word order. Most modern models use rotary position embeddings (RoPE): each position rotates parts of the attention vectors by angles that grow with distance, so the model can tell how far apart two tokens are. Past the trained length, the model meets rotation angles it has never seen, and quality collapses.

What's YaRN? RoPE scaling methods squeeze longer sequences into the angle range the model knows. Simple "linear" scaling compresses everything evenly, which blurs fine-grained position. YaRN compresses the slow-rotating dimensions more and the fast ones less, and adjusts attention temperature. Its paper reports reaching long contexts with 10× fewer training tokens than earlier methods. Many models ship already trained for it, and you switch it on at load time.

Qwen3 30B-A3B: native versus YaRN-extended contextNative context 32,768 tokens. With YaRN factor 4 the model card documents 131,072 tokens.Native (trained)32,768YaRN × 4131,072 tokens
Fig. 1 · Qwen3 30B-A3B's model card lists 32,768 tokens native and 131,072 with YaRN (factor 4). Every extra token also costs KV-cache memory; see the KV cache explainer.

An everyday analogy. A ruler marked to 30 cm. Linear scaling relabels it as 120 cm, with every mark four times further apart, so you lose precision everywhere. YaRN keeps the millimetre marks sharp and stretches only the coarse ones.

Long context isn't long memory. Even within their window, models recall facts from the middle of very long prompts less reliably than from the beginning or the end. For large document collections, RAG usually beats pasting everything in.

  • RoPE in one line

    For a pair of dimensions i at position m, rotate by θᵢ·m with θᵢ = base^(−2i/d). Linear scaling divides m by factor s. YaRN interpolates per dimension according to wavelength, and scales attention logits by a temperature that grows with s.

    Turning it on in llama.cpp

    llama-server -m Qwen3-30B-A3B-Q4_K_M.gguf -c 131072 \
  --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768 --port 8080

    -c sets the allocated context (0 means take it from the model). --rope-scale 4 expands by 4×, and --yarn-orig-ctx is the native length. Many GGUFs carry the right settings in their metadata, so try plain -c first.

    Ollama

    OLLAMA_CONTEXT_LENGTH=32768 ollama serve

    Or set num_ctx per request or in a Modelfile. Check what was really allocated, and whether the model spilled to the CPU, with ollama ps.

    Memory per token

    KV_bytes/token = 2 × layers × kv_heads × head_dim × bytes_per_element

    Qwen3-30B-A3B (48 layers, 4 KV heads, head_dim 128, f16): 2 × 48 × 4 × 128 × 2 ≈ 96 KB per token, or about 12 GB at 128K. With -ctk q8_0 -ctv q8_0 it's about half that.

    Pitfalls

    • Silent truncation. Many apps drop old messages without telling you. If a model "forgets" the start of a document, check the configured context first.
    • Sliding-window models. Gemma 3 uses local attention on most layers, so its cache grows more slowly, but those layers can only see recent tokens.
    • Don't scale past the card. A factor beyond what the model was trained for produces fluent nonsense.

Sources

  1. Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2104.09864)
  2. Peng et al., "YaRN: Efficient Context Window Extension" (arXiv 2309.00071)
  3. Qwen3-30B-A3B model card: native 32,768, 131,072 with YaRN
  4. llama.cpp server README: -c, --rope-scaling, --rope-scale, --yarn-orig-ctx
  5. Ollama docs: context length defaults

Other explainers: KV cache · Memory bandwidth · Mixture of experts · GGUF vs EXL vs AWQ · Quantization · RAG and embeddings · Speculative decoding · Tool calling and MCP