# Learn > Glossary and explainers. Canonical: https://thelocalbench.com/learn · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0 Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder ## Explainers - [What do context length, RoPE and YaRN mean, and how much context do you really get?](/learn/context-length.md): Context length is how many tokens a model can consider at once. Models encode position with RoPE, and methods such as YaRN stretch it beyond the trained length: Qwen3-30B-A3B is 32,768 tokens native and 131,072 with YaRN. Your runtime's setting, and the KV-cache memory it needs, decide what you actually get. - [What is a KV cache, and why does it eat my VRAM?](/learn/kv-cache.md): The KV cache is memory a model uses to remember every token already in the conversation, and it grows with context length. In October 2026, Qwen3 32B needs 0.26 MB per token, so 32,768 tokens add 8.6 GB to its 19.8 GB of Q4_K_M weights: too much for a 24 GB card. - [Why is memory bandwidth the speed limit for local AI?](/learn/memory-bandwidth.md): Each generated token requires reading every active weight from memory once, so tokens per second can't exceed bandwidth divided by gigabytes read per token. In October 2026, a used RTX 3090 (936 GB/s) running Gemma 3 27B at Q4_K_M (16.6 GB) tops out near 57 tok/s; real runs reach about 60% of that. - [What is a mixture-of-experts model, and why does "30B-A3B" run so fast?](/learn/mixture-of-experts.md): A mixture-of-experts (MoE) model splits each layer into many expert sub-networks and runs only a few per token. You must store every expert, but each token reads only the active ones, so Qwen3-30B-A3B (30.5B total, 3.3B active) needs memory like a 30B model yet decodes at speeds closer to a 3B one. - [GGUF, EXL2/EXL3, AWQ or GPTQ — which quantized file should you download?](/learn/quantization-formats.md): Choose by runtime, not by name. GGUF is for llama.cpp, Ollama and LM Studio, and runs on CPUs, Macs and any GPU. EXL2 and EXL3 are for ExLlama on NVIDIA GPUs and allow any average bit rate. AWQ and GPTQ are 4-bit GPU formats for vLLM-style servers. At similar bits per weight, quality differences are small. - [What is quantization, and how much does Q4 hurt?](/learn/quantization.md): Quantization stores a model's weights in fewer bits, typically 4 to 8 instead of 16. In October 2026, Gemma 3 27B shrinks from 54.0 GB in BF16 to 16.6 GB at Q4_K_M, enough to fit a 24 GB card. Quality loss is small at 5 to 8 bits, acceptable at 4, noticeable at 3. - [What are RAG and embeddings, and how does a local model "read" your documents?](/learn/rag-and-embeddings.md): Retrieval-augmented generation (RAG) turns your documents into embeddings (lists of numbers that capture meaning), finds the passages closest to your question, and pastes them into the model's prompt. The model never learns your files; it reads the few relevant pieces each time. Retrieval quality, not model size, usually decides the result. - [What is speculative decoding, and is it a free speed-up?](/learn/speculative-decoding.md): Speculative decoding lets a small, fast draft model guess the next few tokens, then has the large model check all the guesses in one pass. Accepted guesses come out almost for free, so the output is identical to the large model's own, but typically 1.5–3× faster when the draft guesses well. - [What are tool calling and MCP, and can a local model use them?](/learn/tool-calling-and-mcp.md): With tool calling, the model replies with a structured request (a function name plus JSON arguments) instead of prose. Your app runs the function and returns the result. MCP is an open protocol, built on JSON-RPC 2.0, that lets any compatible app plug into servers offering tools. Local models handle it well if they were trained for it. ## Glossary ### AWQ and GPTQ Calibrated, mostly 4-bit GPU quantization methods. GPTQ uses approximate second-order information layer by layer; AWQ protects the roughly 1% of weights that matter most to activations. Both are first-class in vLLM and SGLang. Seen as: -AWQ, -GPTQ, w4a16 Go deeper: /learn/quantization-formats ### Bits per weight (bpw) The average number of bits used to store each parameter in a quantized file, including scales and metadata. Q4_K_M averages about 4.85 bpw, not 4. Multiply parameters by bpw and divide by 8 to estimate the file size in bytes. Seen as: 4.85 bpw, BPW Go deeper: /learn/quantization ### Context window The maximum number of tokens a model can attend to at once, prompt and reply combined. A longer window does not cost memory until you use it; the KV cache grows with the tokens actually in context. Seen as: 128K, --ctx-size 32768, num_ctx Go deeper: /learn/kv-cache ### Decode The phase where the model generates the reply one token at a time. Each token requires reading the active weights and the KV cache, so decode speed (tok/s) is mostly set by memory bandwidth. llama-bench reports it as tg128. Seen as: tg128, eval rate, tok/s Go deeper: /learn/memory-bandwidth ### Embeddings Lists of numbers that represent the meaning of a text, made by a small embedding model. Similar meanings give nearby vectors, which is how RAG finds relevant passages without matching keywords. Seen as: nomic-embed-text, /api/embed, --embedding Go deeper: /learn/rag-and-embeddings ### EXL2 / EXL3 Quantized formats for the ExLlama runtimes on NVIDIA GPUs. EXL2 mixes 2- to 8-bit pieces to hit any average bits per weight; EXL3, from ExLlamaV3, is based on QTIP. Usually served through TabbyAPI. Seen as: -exl2, -exl3, 4.65bpw Go deeper: /learn/quantization-formats ### Flash attention An attention algorithm that computes results in tiles without writing the full attention matrix to memory. In llama.cpp it reduces the compute buffer, speeds up long contexts and is required to quantize the V cache. Seen as: -fa, --flash-attn on Go deeper: /learn/kv-cache ### GGUF The single-file model format used by llama.cpp and the tools built on it (Ollama, LM Studio, llamafile). It stores the quantized tensors plus metadata such as the architecture, tokenizer and quant type, so a runtime can load it without extra config files. Seen as: model-Q4_K_M.gguf, -m model.gguf Go deeper: /learn/quantization ### Grouped-query attention (GQA) An attention design where several query heads share one key/value head. Llama 3.1 8B has 32 query heads but 8 KV heads, which cuts the KV cache to a quarter of what full multi-head attention would need. Seen as: num_key_value_heads, n_head_kv Go deeper: /learn/kv-cache ### Importance matrix (imatrix) A calibration file that records which weights matter most on sample text. Quantizing with an imatrix spends precision where it counts, which noticeably improves quality at 3 and 4 bits for the same file size. Seen as: --imatrix, i1-Q4_K_M, IQ4_XS Go deeper: /learn/quantization ### K-quants llama.cpp's family of block quantization schemes (Q3_K, Q4_K, Q5_K, Q6_K). Weights are grouped into super-blocks with their own scales, and the _S/_M/_L suffix says how many sensitive tensors are kept at higher precision. Seen as: Q4_K_M, Q5_K_S, Q6_K Go deeper: /learn/quantization ### KL-divergence A measure of how far a quantized model's next-token probabilities drift from the full-precision model's on the same text. Zero means identical. It is a more direct test of quantization damage than perplexity, and is how The Local Bench reports quant fidelity. Seen as: --kl-divergence, KLD 0.017 Go deeper: /learn/quantization ### KV cache The memory where a transformer keeps the attention keys and values for every token it has already read, so it doesn't recompute them. It grows linearly with context length and with the number of concurrent users, and on long contexts it can rival the weights in size. Seen as: --ctx-size, -ctk q8_0, --cache-type-k Go deeper: /learn/kv-cache ### MCP (Model Context Protocol) An open protocol, using JSON-RPC 2.0, that lets AI apps connect to servers offering tools, resources and prompts. Any MCP-capable app can then use any MCP server, local or remote. Seen as: MCP server, mcpServers, stdio Go deeper: /learn/tool-calling-and-mcp ### Memory bandwidth How many gigabytes per second the processor can read from its memory. For single-user text generation it is usually the speed limit, because every new token requires reading all active weights once. Ceiling tok/s ≈ bandwidth ÷ gigabytes read per token. Seen as: GB/s, TB/s, memory bus Go deeper: /learn/memory-bandwidth ### Mixture of experts (MoE) A model split into many expert sub-networks, of which only a few run for each token. You must hold all of them in memory, but only the active ones are read per token, so a 30B MoE with 3B active decodes about as fast as a 3B dense model. Seen as: 30B-A3B, num_experts, experts_per_token Go deeper: /learn/mixture-of-experts ### Multi-head latent attention (MLA) DeepSeek's attention design, which compresses keys and values into a small shared latent vector per token. It shrinks the KV cache far more than GQA, so memory per token depends on the latent rank rather than the number of heads. Seen as: kv_lora_rank, qk_rope_head_dim Go deeper: /learn/kv-cache ### Offloading Running some layers on the GPU and the rest on the CPU when a model doesn't fit in VRAM. It works, but each token's time is the sum of both parts, so even a small CPU share can halve the speed. Seen as: -ngl 40, --n-gpu-layers, num_gpu Go deeper: /learn/memory-bandwidth ### Perplexity How surprised a model is by a test text, on average; lower is better. Useful for comparing quants of the same model, but not across models with different tokenizers, and it can hide errors that KL-divergence catches. Seen as: PPL, llama-perplexity ### Prefill The phase where the model reads your whole prompt in parallel before replying. It is limited by compute (TFLOPS) rather than bandwidth, and it sets how long you wait before the first token on long prompts. llama-bench reports it as pp512. Seen as: pp512, prompt eval rate Go deeper: /learn/memory-bandwidth ### Quantization Storing a model's weights with fewer bits per number, for example 4 instead of 16. It shrinks the file and the memory needed roughly in proportion, and speeds up generation on bandwidth-limited hardware, at the cost of a small, measurable loss of accuracy. Seen as: Q4_K_M, Q8_0, MXFP4, --quantize Go deeper: /learn/quantization ### RAG Retrieval-augmented generation. Your documents are split into chunks and embedded; the chunks closest to a question are pasted into the prompt so the model answers from them. Nothing is retrained. Seen as: knowledge base, #documents, top-k Go deeper: /learn/rag-and-embeddings ### RoPE scaling (YaRN) Techniques that stretch a model's positional encoding to run beyond its trained context, such as Qwen3's 32K native window extended to 128K with YaRN. Quality can drop on short prompts if scaling is left on unnecessarily. Seen as: --rope-scaling yarn, rope_scaling Go deeper: /learn/context-length ### Sliding-window attention Layers that only attend to the most recent N tokens instead of the whole context. Their KV cache stops growing at N tokens. Gemma 3 uses a 1,024-token window on five of every six layers; gpt-oss uses 128 tokens on half of its layers. Seen as: sliding_window, SWA, --swa-full Go deeper: /learn/kv-cache ### Speculative decoding A speed-up where a small draft model guesses several tokens and the large model verifies them in one pass. Accepted guesses come almost free, and the output is identical to the large model's. Typical gains are 1.5–3× on predictable text. Seen as: -md, --spec-draft-n-max, draft model, EAGLE Go deeper: /learn/speculative-decoding ### Time to first token (TTFT) The delay between sending a prompt and seeing the first token of the reply. Roughly prompt tokens ÷ prefill speed. It matters most for RAG and coding agents, which send prompts of 8k–16k tokens. Seen as: TTFT, first token latency Go deeper: /learn/memory-bandwidth ### Tool calling A model's ability to answer with a structured request to run a function, such as a search or a calculator, instead of plain text. The app runs the function and feeds the result back. It needs a model and chat template trained for it. Seen as: function calling, tools, tool_calls, --jinja Go deeper: /learn/tool-calling-and-mcp ### Unified memory A single pool of memory shared by CPU and GPU, as on Apple Silicon, AMD Strix Halo and NVIDIA DGX Spark. It offers large capacity (up to 512 GB) but the OS reserves part of it; we assume 75% is usable by the GPU by default. Seen as: 128 GB unified, iogpu.wired_limit_mb Go deeper: /learn/memory-bandwidth ### VRAM The memory on a discrete graphics card. A model runs at full speed only if its weights, KV cache and runtime overhead all fit in VRAM; whatever spills over to system RAM runs at system-RAM speed. Seen as: 24 GB, -ngl 99, out of memory Go deeper: /learn/memory-bandwidth