{"as_of":"2026-10-12","license":"CC-BY-4.0","license_url":"https://creativecommons.org/licenses/by/4.0/","cite_as":"The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0","source_url":"https://thelocalbench.com","terms":[{"term":"AWQ and GPTQ","slug":"awq-gptq","definition":"Calibrated, mostly 4-bit GPU quantization methods. GPTQ uses approximate second-order information layer by layer; AWQ protects the roughly 1% of weights that matter most to activations. Both are first-class in vLLM and SGLang.","seen_as":["-AWQ","-GPTQ","w4a16"],"explainer":"quantization-formats","related":["quantization","gguf"],"url":"/learn/quantization-formats"},{"term":"Bits per weight (bpw)","slug":"bits-per-weight","definition":"The average number of bits used to store each parameter in a quantized file, including scales and metadata. Q4_K_M averages about 4.85 bpw, not 4. Multiply parameters by bpw and divide by 8 to estimate the file size in bytes.","seen_as":["4.85 bpw","BPW"],"explainer":"quantization","related":["quantization","k-quants","gguf"],"url":"/learn/quantization"},{"term":"Context window","slug":"context-window","definition":"The maximum number of tokens a model can attend to at once, prompt and reply combined. A longer window does not cost memory until you use it; the KV cache grows with the tokens actually in context.","seen_as":["128K","--ctx-size 32768","num_ctx"],"explainer":"kv-cache","related":["kv-cache","rope-scaling","sliding-window-attention"],"url":"/learn/kv-cache"},{"term":"Decode","slug":"decode","definition":"The phase where the model generates the reply one token at a time. Each token requires reading the active weights and the KV cache, so decode speed (tok/s) is mostly set by memory bandwidth. llama-bench reports it as tg128.","seen_as":["tg128","eval rate","tok/s"],"explainer":"memory-bandwidth","related":["prefill","memory-bandwidth","ttft"],"url":"/learn/memory-bandwidth"},{"term":"Embeddings","slug":"embeddings","definition":"Lists of numbers that represent the meaning of a text, made by a small embedding model. Similar meanings give nearby vectors, which is how RAG finds relevant passages without matching keywords.","seen_as":["nomic-embed-text","/api/embed","--embedding"],"explainer":"rag-and-embeddings","related":["rag"],"url":"/learn/rag-and-embeddings"},{"term":"EXL2 / EXL3","slug":"exl2","definition":"Quantized formats for the ExLlama runtimes on NVIDIA GPUs. EXL2 mixes 2- to 8-bit pieces to hit any average bits per weight; EXL3, from ExLlamaV3, is based on QTIP. Usually served through TabbyAPI.","seen_as":["-exl2","-exl3","4.65bpw"],"explainer":"quantization-formats","related":["quantization","bits-per-weight"],"url":"/learn/quantization-formats"},{"term":"Flash attention","slug":"flash-attention","definition":"An attention algorithm that computes results in tiles without writing the full attention matrix to memory. In llama.cpp it reduces the compute buffer, speeds up long contexts and is required to quantize the V cache.","seen_as":["-fa","--flash-attn on"],"explainer":"kv-cache","related":["kv-cache"],"url":"/learn/kv-cache"},{"term":"GGUF","slug":"gguf","definition":"The single-file model format used by llama.cpp and the tools built on it (Ollama, LM Studio, llamafile). It stores the quantized tensors plus metadata such as the architecture, tokenizer and quant type, so a runtime can load it without extra config files.","seen_as":["model-Q4_K_M.gguf","-m model.gguf"],"explainer":"quantization","related":["quantization","k-quants","llama-cpp"],"url":"/learn/quantization"},{"term":"Grouped-query attention (GQA)","slug":"gqa","definition":"An attention design where several query heads share one key/value head. Llama 3.1 8B has 32 query heads but 8 KV heads, which cuts the KV cache to a quarter of what full multi-head attention would need.","seen_as":["num_key_value_heads","n_head_kv"],"explainer":"kv-cache","related":["kv-cache","mla"],"url":"/learn/kv-cache"},{"term":"Importance matrix (imatrix)","slug":"imatrix","definition":"A calibration file that records which weights matter most on sample text. Quantizing with an imatrix spends precision where it counts, which noticeably improves quality at 3 and 4 bits for the same file size.","seen_as":["--imatrix","i1-Q4_K_M","IQ4_XS"],"explainer":"quantization","related":["k-quants","quantization","kl-divergence"],"url":"/learn/quantization"},{"term":"K-quants","slug":"k-quants","definition":"llama.cpp's family of block quantization schemes (Q3_K, Q4_K, Q5_K, Q6_K). Weights are grouped into super-blocks with their own scales, and the _S/_M/_L suffix says how many sensitive tensors are kept at higher precision.","seen_as":["Q4_K_M","Q5_K_S","Q6_K"],"explainer":"quantization","related":["quantization","bits-per-weight","imatrix"],"url":"/learn/quantization"},{"term":"KL-divergence","slug":"kl-divergence","definition":"A measure of how far a quantized model's next-token probabilities drift from the full-precision model's on the same text. Zero means identical. It is a more direct test of quantization damage than perplexity, and is how The Local Bench reports quant fidelity.","seen_as":["--kl-divergence","KLD 0.017"],"explainer":"quantization","related":["quantization","perplexity"],"url":"/learn/quantization"},{"term":"KV cache","slug":"kv-cache","definition":"The memory where a transformer keeps the attention keys and values for every token it has already read, so it doesn't recompute them. It grows linearly with context length and with the number of concurrent users, and on long contexts it can rival the weights in size.","seen_as":["--ctx-size","-ctk q8_0","--cache-type-k"],"explainer":"kv-cache","related":["context-window","gqa","sliding-window-attention","flash-attention"],"url":"/learn/kv-cache"},{"term":"MCP (Model Context Protocol)","slug":"mcp","definition":"An open protocol, using JSON-RPC 2.0, that lets AI apps connect to servers offering tools, resources and prompts. Any MCP-capable app can then use any MCP server, local or remote.","seen_as":["MCP server","mcpServers","stdio"],"explainer":"tool-calling-and-mcp","related":["tool-calling"],"url":"/learn/tool-calling-and-mcp"},{"term":"Memory bandwidth","slug":"memory-bandwidth","definition":"How many gigabytes per second the processor can read from its memory. For single-user text generation it is usually the speed limit, because every new token requires reading all active weights once. Ceiling tok/s ≈ bandwidth ÷ gigabytes read per token.","seen_as":["GB/s","TB/s","memory bus"],"explainer":"memory-bandwidth","related":["decode","unified-memory","vram","mixture-of-experts"],"url":"/learn/memory-bandwidth"},{"term":"Mixture of experts (MoE)","slug":"mixture-of-experts","definition":"A model split into many expert sub-networks, of which only a few run for each token. You must hold all of them in memory, but only the active ones are read per token, so a 30B MoE with 3B active decodes about as fast as a 3B dense model.","seen_as":["30B-A3B","num_experts","experts_per_token"],"explainer":"mixture-of-experts","related":["memory-bandwidth","decode"],"url":"/learn/mixture-of-experts"},{"term":"Multi-head latent attention (MLA)","slug":"mla","definition":"DeepSeek's attention design, which compresses keys and values into a small shared latent vector per token. It shrinks the KV cache far more than GQA, so memory per token depends on the latent rank rather than the number of heads.","seen_as":["kv_lora_rank","qk_rope_head_dim"],"explainer":"kv-cache","related":["gqa","kv-cache"],"url":"/learn/kv-cache"},{"term":"Offloading","slug":"offloading","definition":"Running some layers on the GPU and the rest on the CPU when a model doesn't fit in VRAM. It works, but each token's time is the sum of both parts, so even a small CPU share can halve the speed.","seen_as":["-ngl 40","--n-gpu-layers","num_gpu"],"explainer":"memory-bandwidth","related":["vram","memory-bandwidth"],"url":"/learn/memory-bandwidth"},{"term":"Perplexity","slug":"perplexity","definition":"How surprised a model is by a test text, on average; lower is better. Useful for comparing quants of the same model, but not across models with different tokenizers, and it can hide errors that KL-divergence catches.","seen_as":["PPL","llama-perplexity"],"explainer":null,"related":["kl-divergence","quantization"],"url":"/learn#perplexity"},{"term":"Prefill","slug":"prefill","definition":"The phase where the model reads your whole prompt in parallel before replying. It is limited by compute (TFLOPS) rather than bandwidth, and it sets how long you wait before the first token on long prompts. llama-bench reports it as pp512.","seen_as":["pp512","prompt eval rate"],"explainer":"memory-bandwidth","related":["decode","ttft"],"url":"/learn/memory-bandwidth"},{"term":"Quantization","slug":"quantization","definition":"Storing a model's weights with fewer bits per number, for example 4 instead of 16. It shrinks the file and the memory needed roughly in proportion, and speeds up generation on bandwidth-limited hardware, at the cost of a small, measurable loss of accuracy.","seen_as":["Q4_K_M","Q8_0","MXFP4","--quantize"],"explainer":"quantization","related":["bits-per-weight","gguf","k-quants","kl-divergence"],"url":"/learn/quantization"},{"term":"RAG","slug":"rag","definition":"Retrieval-augmented generation. Your documents are split into chunks and embedded; the chunks closest to a question are pasted into the prompt so the model answers from them. Nothing is retrained.","seen_as":["knowledge base","#documents","top-k"],"explainer":"rag-and-embeddings","related":["embeddings","context-window"],"url":"/learn/rag-and-embeddings"},{"term":"RoPE scaling (YaRN)","slug":"rope-scaling","definition":"Techniques that stretch a model's positional encoding to run beyond its trained context, such as Qwen3's 32K native window extended to 128K with YaRN. Quality can drop on short prompts if scaling is left on unnecessarily.","seen_as":["--rope-scaling yarn","rope_scaling"],"explainer":"context-length","related":["context-window"],"url":"/learn/context-length"},{"term":"Sliding-window attention","slug":"sliding-window-attention","definition":"Layers that only attend to the most recent N tokens instead of the whole context. Their KV cache stops growing at N tokens. Gemma 3 uses a 1,024-token window on five of every six layers; gpt-oss uses 128 tokens on half of its layers.","seen_as":["sliding_window","SWA","--swa-full"],"explainer":"kv-cache","related":["kv-cache","context-window"],"url":"/learn/kv-cache"},{"term":"Speculative decoding","slug":"speculative-decoding","definition":"A speed-up where a small draft model guesses several tokens and the large model verifies them in one pass. Accepted guesses come almost free, and the output is identical to the large model's. Typical gains are 1.5–3× on predictable text.","seen_as":["-md","--spec-draft-n-max","draft model","EAGLE"],"explainer":"speculative-decoding","related":["decode","memory-bandwidth"],"url":"/learn/speculative-decoding"},{"term":"Time to first token (TTFT)","slug":"ttft","definition":"The delay between sending a prompt and seeing the first token of the reply. Roughly prompt tokens ÷ prefill speed. It matters most for RAG and coding agents, which send prompts of 8k–16k tokens.","seen_as":["TTFT","first token latency"],"explainer":"memory-bandwidth","related":["prefill","decode"],"url":"/learn/memory-bandwidth"},{"term":"Tool calling","slug":"tool-calling","definition":"A model's ability to answer with a structured request to run a function, such as a search or a calculator, instead of plain text. The app runs the function and feeds the result back. It needs a model and chat template trained for it.","seen_as":["function calling","tools","tool_calls","--jinja"],"explainer":"tool-calling-and-mcp","related":["mcp"],"url":"/learn/tool-calling-and-mcp"},{"term":"Unified memory","slug":"unified-memory","definition":"A single pool of memory shared by CPU and GPU, as on Apple Silicon, AMD Strix Halo and NVIDIA DGX Spark. It offers large capacity (up to 512 GB) but the OS reserves part of it; we assume 75% is usable by the GPU by default.","seen_as":["128 GB unified","iogpu.wired_limit_mb"],"explainer":"memory-bandwidth","related":["vram","memory-bandwidth"],"url":"/learn/memory-bandwidth"},{"term":"VRAM","slug":"vram","definition":"The memory on a discrete graphics card. A model runs at full speed only if its weights, KV cache and runtime overhead all fit in VRAM; whatever spills over to system RAM runs at system-RAM speed.","seen_as":["24 GB","-ngl 99","out of memory"],"explainer":"memory-bandwidth","related":["unified-memory","offloading","memory-bandwidth"],"url":"/learn/memory-bandwidth"}]}