Skip to main contentThe Local Bench

Explainer · REV 2026.10 · reviewed 2026-10-13 · 7 min

What is speculative decoding, and is it a free speed-up?

How a small "draft" model can make a big one type 2–3× faster with identical output, why the speed-up depends on what you're writing, and how to switch it on in llama.cpp and vLLM.

Discrete graphics card

The 50-word answer

Speculative decoding lets a small, fast draft model guess the next few tokens, then has the large model check all the guesses in one pass. Accepted guesses come out almost for free, so the output is identical to the large model's own, but typically 1.5–3× faster when the draft guesses well.

The plain version

When one person chats with a local model, the graphics card spends most of each step waiting for weights to arrive from memory, not doing maths (see memory bandwidth). Reading those weights once and using them to check five tokens costs almost the same as using them for one. Speculative decoding exploits that spare capacity.

How it works. A small draft model from the same family (say, Qwen3-0.6B next to Qwen3-32B) quickly writes a guess of the next few tokens. The big target model then checks all of them in a single pass. It keeps every guess up to the first one it disagrees with, swaps in its own token there, and the round starts again.

One round of speculative decodingA small draft model proposes five tokens: The, cat, sat, on, a. The large target model checks all five in one pass, accepts the first three, rejects on, and substitutes its own token, the. Four tokens are produced from one pass of the large model.Draft model proposesThecatsatonaTarget checks in 1 pass✓✓✓✗✗Output this roundThecatsatthe4 tokens, 1 big pass
Fig. 1 · Three drafted tokens are accepted, the fourth is rejected and replaced by the target model’s own choice, so this round yields four tokens for the cost of one large-model pass plus five cheap draft steps. Illustrative, not measured.

An everyday analogy. A junior writer drafts the next sentence, and the editor reads it in one go, keeping the words they'd have written anyway and correcting the first one they wouldn't. If the junior is good, the editor gets through pages much faster. The finished text is still the editor's.

The output is the same as the big model on its own; only the speed changes. The original papers prove this for sampling, not just greedy decoding. Leviathan et al. report 2–3× on T5-XXL "with identical outputs". DeepMind measured 2–2.5× on the 70B Chinchilla. EAGLE, which drafts from the target's own internal features instead of a separate model, reports 2.7–3.5× on LLaMA2-Chat 70B.

Why your mileage varies. The gain depends on the acceptance rate: how often the draft guesses right. Code, structured output, and text that repeats your prompt are easy to guess, so speed-ups are large. Creative writing at high temperature is hard to guess, and the gain can shrink to nothing. A draft model also costs memory and some compute, so on a GPU that's already full, it can make things worse.

Picking a draft. It must share the target's tokenizer (the same family is the simple rule). It should be 10–50× smaller. And it should be trained on similar data: a chat-tuned draft for a chat-tuned target.

  • Why it's lossless

    For each drafted token x with draft probability q(x) and target probability p(x), accept with probability min(1, p(x)/q(x)). On rejection, sample from the normalised residual max(0, p − q). The result is distributed exactly as p. In llama.cpp that's "within hardware numerics": floating-point differences between batched and single-token passes can still change a token now and then.

    Expected speed-up

    With acceptance rate α per token and k drafted tokens, the expected tokens per target pass is (1 − α^(k+1)) / (1 − α). At α = 0.8 and k = 4, that's about 3.4 tokens per pass, before subtracting the draft's own cost.

    llama.cpp

    llama-server -m Qwen3-32B-Q4_K_M.gguf -md Qwen3-0.6B-Q8_0.gguf \
  -ngl 99 -ngld 99 --spec-draft-n-max 8 -c 16384 --port 8080

    -md (--spec-draft-model) loads the draft, -ngld puts its layers on the GPU, and --spec-draft-n-max sets how many tokens to draft (default 3). -hfd downloads a draft from Hugging Face the same way -hf does for the main model. The older --draft-max flag has been removed. Recent builds also list a --spec-type option (default none), with draft-model, EAGLE-3, MTP and n-gram variants; run llama-server --help on your build to see which apply.

    Draft-free variants

    • n-gram / prompt lookup: guess by copying spans from the prompt. Free, and excellent for editing code or summarising.
    • EAGLE and MTP heads: a small extra head trained on the target model's hidden states. Higher acceptance than a separate draft, but it must be trained for that exact model.

    Pitfalls

    • Batching kills it. With many concurrent users the GPU is busy with real work, so there's no spare capacity to spend on guesses. Servers often switch it off under load.
    • Different tokenizers fail. A Llama draft can't speculate for a Qwen target.
    • Measure acceptance, not vibes. Start llama-server with --metrics and compare llamacpp:spec_decode_num_accepted_tokens_total with llamacpp:spec_decode_num_draft_tokens_total on /metrics. Below about 50% acceptance, try a smaller k or turn it off.

Sources

  1. Leviathan, Kalman & Matias, "Fast Inference from Transformers via Speculative Decoding" (arXiv 2211.17192)
  2. Chen et al., "Accelerating LLM Decoding with Speculative Sampling" (arXiv 2302.01318)
  3. Li et al., "EAGLE" (arXiv 2401.15077)
  4. llama.cpp server README: --spec-* options
  5. vLLM docs: speculative decoding

Other explainers: Context length and RoPE · KV cache · Memory bandwidth · Mixture of experts · GGUF vs EXL vs AWQ · Quantization · RAG and embeddings · Tool calling and MCP