Skip to main contentThe Local Bench

9 explainers · 29 terms

Learn

The jargon of local AI, explained in terms of your rig. Short definitions here; deep explainers when you want the formulas.

Try a local model

A small first run before the terminology.

Run the tested CPU generation demo. It produces a short story using a 61 MB model. No GPU or account required; this is a setup check, not a useful chat assistant.

Explainers

Each starts with a 50-word answer, then a plain-language layer, then the technical details.

Coming next

Explainers we're writing. Planned topics may change order.

  • Training · LoRA and QLoRA: fine-tuning on one GPU
  • Models · Reasoning models: what "thinking" tokens cost you
  • Training · Distillation: how small models learn from big ones
  • Models · Tokenizers: why tokens are not words (and why it matters for price and speed)
  • Apps · Sampling: temperature, top-p, min-p and repetition penalties
  • Hardware · VRAM math: weights + KV cache + overhead, worked examples
  • Hardware · Unified memory: Apple Silicon, Strix Halo and DGX Spark
  • Hardware · NPUs: what the AI chip in your laptop can and cannot run
  • Hardware · PCIe lanes and multi-GPU boards: when x4 is fine
  • Hardware · Power, thermals and noise: power-limiting a GPU for 24/7 use
  • Hardware · Multi-GPU: tensor vs pipeline (layer) parallelism
  • Speed · vLLM and paged attention: serving a household or a team
  • Models · Evals and benchmarks: reading leaderboards without being fooled
  • Apps · Agent harnesses: what Aider, OpenHands and Goose add on top of a model
  • Speed · KV-cache quantization and flash attention
  • Models · Vision-language models: mmproj files and image tokens
  • Apps · Local speech: Whisper, TTS and voice-assistant latency
  • Speed · Batching: why one user and ten users need different hardware
  • Models · Open-weight licences: Apache, Llama, Gemma and what you may ship

Glossary

Definitions of 60 words or fewer. Link to any term with /learn#term.

AWQ and GPTQ

Calibrated, mostly 4-bit GPU quantization methods. GPTQ uses approximate second-order information layer by layer; AWQ protects the roughly 1% of weights that matter most to activations. Both are first-class in vLLM and SGLang.

You'll see it as: -AWQ · -GPTQ · w4a16

Go deeper: GGUF vs EXL vs AWQ · Related: Quantization, GGUF

Bits per weight (bpw)

The average number of bits used to store each parameter in a quantized file, including scales and metadata. Q4_K_M averages about 4.85 bpw, not 4. Multiply parameters by bpw and divide by 8 to estimate the file size in bytes.

You'll see it as: 4.85 bpw · BPW

Go deeper: Quantization · Related: Quantization, K-quants, GGUF

Context window

The maximum number of tokens a model can attend to at once, prompt and reply combined. A longer window does not cost memory until you use it; the KV cache grows with the tokens actually in context.

You'll see it as: 128K · --ctx-size 32768 · num_ctx

Go deeper: KV cache · Related: KV cache, RoPE scaling (YaRN), Sliding-window attention

Decode

The phase where the model generates the reply one token at a time. Each token requires reading the active weights and the KV cache, so decode speed (tok/s) is mostly set by memory bandwidth. llama-bench reports it as tg128.

You'll see it as: tg128 · eval rate · tok/s

Go deeper: Memory bandwidth · Related: Prefill, Memory bandwidth, Time to first token (TTFT)

Embeddings

Lists of numbers that represent the meaning of a text, made by a small embedding model. Similar meanings give nearby vectors, which is how RAG finds relevant passages without matching keywords.

You'll see it as: nomic-embed-text · /api/embed · --embedding

Go deeper: RAG and embeddings · Related: RAG

EXL2 / EXL3

Quantized formats for the ExLlama runtimes on NVIDIA GPUs. EXL2 mixes 2- to 8-bit pieces to hit any average bits per weight; EXL3, from ExLlamaV3, is based on QTIP. Usually served through TabbyAPI.

You'll see it as: -exl2 · -exl3 · 4.65bpw

Go deeper: GGUF vs EXL vs AWQ · Related: Quantization, Bits per weight (bpw)

Flash attention

An attention algorithm that computes results in tiles without writing the full attention matrix to memory. In llama.cpp it reduces the compute buffer, speeds up long contexts and is required to quantize the V cache.

You'll see it as: -fa · --flash-attn on

Go deeper: KV cache · Related: KV cache

GGUF

The single-file model format used by llama.cpp and the tools built on it (Ollama, LM Studio, llamafile). It stores the quantized tensors plus metadata such as the architecture, tokenizer and quant type, so a runtime can load it without extra config files.

You'll see it as: model-Q4_K_M.gguf · -m model.gguf

Go deeper: Quantization · Related: Quantization, K-quants

Grouped-query attention (GQA)

An attention design where several query heads share one key/value head. Llama 3.1 8B has 32 query heads but 8 KV heads, which cuts the KV cache to a quarter of what full multi-head attention would need.

You'll see it as: num_key_value_heads · n_head_kv

Go deeper: KV cache · Related: KV cache, Multi-head latent attention (MLA)

Importance matrix (imatrix)

A calibration file that records which weights matter most on sample text. Quantizing with an imatrix spends precision where it counts, which noticeably improves quality at 3 and 4 bits for the same file size.

You'll see it as: --imatrix · i1-Q4_K_M · IQ4_XS

Go deeper: Quantization · Related: K-quants, Quantization, KL-divergence

K-quants

llama.cpp's family of block quantization schemes (Q3_K, Q4_K, Q5_K, Q6_K). Weights are grouped into super-blocks with their own scales, and the _S/_M/_L suffix says how many sensitive tensors are kept at higher precision.

You'll see it as: Q4_K_M · Q5_K_S · Q6_K

Go deeper: Quantization · Related: Quantization, Bits per weight (bpw), Importance matrix (imatrix)

KL-divergence

A measure of how far a quantized model's next-token probabilities drift from the full-precision model's on the same text. Zero means identical. It is a more direct test of quantization damage than perplexity, and is how The Local Bench reports quant fidelity.

You'll see it as: --kl-divergence · KLD 0.017

Go deeper: Quantization · Related: Quantization, Perplexity

KV cache

The memory where a transformer keeps the attention keys and values for every token it has already read, so it doesn't recompute them. It grows linearly with context length and with the number of concurrent users, and on long contexts it can rival the weights in size.

You'll see it as: --ctx-size · -ctk q8_0 · --cache-type-k

Go deeper: KV cache · Related: Context window, Grouped-query attention (GQA), Sliding-window attention, Flash attention

MCP (Model Context Protocol)

An open protocol, using JSON-RPC 2.0, that lets AI apps connect to servers offering tools, resources and prompts. Any MCP-capable app can then use any MCP server, local or remote.

You'll see it as: MCP server · mcpServers · stdio

Go deeper: Tool calling and MCP · Related: Tool calling

Memory bandwidth

How many gigabytes per second the processor can read from its memory. For single-user text generation it is usually the speed limit, because every new token requires reading all active weights once. Ceiling tok/s ≈ bandwidth ÷ gigabytes read per token.

You'll see it as: GB/s · TB/s · memory bus

Go deeper: Memory bandwidth · Related: Decode, Unified memory, VRAM, Mixture of experts (MoE)

Mixture of experts (MoE)

A model split into many expert sub-networks, of which only a few run for each token. You must hold all of them in memory, but only the active ones are read per token, so a 30B MoE with 3B active decodes about as fast as a 3B dense model.

You'll see it as: 30B-A3B · num_experts · experts_per_token

Go deeper: Mixture of experts · Related: Memory bandwidth, Decode

Multi-head latent attention (MLA)

DeepSeek's attention design, which compresses keys and values into a small shared latent vector per token. It shrinks the KV cache far more than GQA, so memory per token depends on the latent rank rather than the number of heads.

You'll see it as: kv_lora_rank · qk_rope_head_dim

Go deeper: KV cache · Related: Grouped-query attention (GQA), KV cache

Offloading

Running some layers on the GPU and the rest on the CPU when a model doesn't fit in VRAM. It works, but each token's time is the sum of both parts, so even a small CPU share can halve the speed.

You'll see it as: -ngl 40 · --n-gpu-layers · num_gpu

Go deeper: Memory bandwidth · Related: VRAM, Memory bandwidth

Perplexity

How surprised a model is by a test text, on average; lower is better. Useful for comparing quants of the same model, but not across models with different tokenizers, and it can hide errors that KL-divergence catches.

You'll see it as: PPL · llama-perplexity

Related: KL-divergence, Quantization

Prefill

The phase where the model reads your whole prompt in parallel before replying. It is limited by compute (TFLOPS) rather than bandwidth, and it sets how long you wait before the first token on long prompts. llama-bench reports it as pp512.

You'll see it as: pp512 · prompt eval rate

Go deeper: Memory bandwidth · Related: Decode, Time to first token (TTFT)

Quantization

Storing a model's weights with fewer bits per number, for example 4 instead of 16. It shrinks the file and the memory needed roughly in proportion, and speeds up generation on bandwidth-limited hardware, at the cost of a small, measurable loss of accuracy.

You'll see it as: Q4_K_M · Q8_0 · MXFP4 · --quantize

Go deeper: Quantization · Related: Bits per weight (bpw), GGUF, K-quants, KL-divergence

RAG

Retrieval-augmented generation. Your documents are split into chunks and embedded; the chunks closest to a question are pasted into the prompt so the model answers from them. Nothing is retrained.

You'll see it as: knowledge base · #documents · top-k

Go deeper: RAG and embeddings · Related: Embeddings, Context window

RoPE scaling (YaRN)

Techniques that stretch a model's positional encoding to run beyond its trained context, such as Qwen3's 32K native window extended to 128K with YaRN. Quality can drop on short prompts if scaling is left on unnecessarily.

You'll see it as: --rope-scaling yarn · rope_scaling

Go deeper: Context length and RoPE · Related: Context window

Sliding-window attention

Layers that only attend to the most recent N tokens instead of the whole context. Their KV cache stops growing at N tokens. Gemma 3 uses a 1,024-token window on five of every six layers; gpt-oss uses 128 tokens on half of its layers.

You'll see it as: sliding_window · SWA · --swa-full

Go deeper: KV cache · Related: KV cache, Context window

Speculative decoding

A speed-up where a small draft model guesses several tokens and the large model verifies them in one pass. Accepted guesses come almost free, and the output is identical to the large model's. Typical gains are 1.5–3× on predictable text.

You'll see it as: -md · --spec-draft-n-max · draft model · EAGLE

Go deeper: Speculative decoding · Related: Decode, Memory bandwidth

Time to first token (TTFT)

The delay between sending a prompt and seeing the first token of the reply. Roughly prompt tokens ÷ prefill speed. It matters most for RAG and coding agents, which send prompts of 8k–16k tokens.

You'll see it as: TTFT · first token latency

Go deeper: Memory bandwidth · Related: Prefill, Decode

Tool calling

A model's ability to answer with a structured request to run a function, such as a search or a calculator, instead of plain text. The app runs the function and feeds the result back. It needs a model and chat template trained for it.

You'll see it as: function calling · tools · tool_calls · --jinja

Go deeper: Tool calling and MCP · Related: MCP (Model Context Protocol)

Unified memory

A single pool of memory shared by CPU and GPU, as on Apple Silicon, AMD Strix Halo and NVIDIA DGX Spark. It offers large capacity (up to 512 GB) but the OS reserves part of it; we assume 75% is usable by the GPU by default.

You'll see it as: 128 GB unified · iogpu.wired_limit_mb

Go deeper: Memory bandwidth · Related: VRAM, Memory bandwidth

VRAM

The memory on a discrete graphics card. A model runs at full speed only if its weights, KV cache and runtime overhead all fit in VRAM; whatever spills over to system RAM runs at system-RAM speed.

You'll see it as: 24 GB · -ngl 99 · out of memory

Go deeper: Memory bandwidth · Related: Unified memory, Offloading, Memory bandwidth