About The Local Bench

How every number is labelled

Every value in our data files carries a source and one of five flags, and the flag is shown next to the number on the page:

• Verified checked against the primary source. Model architecture (layers, heads, head dimension, context, sliding window, parameter count) is read from each model's config.json and the Hugging Face API; GGUF file sizes are summed from the repository's file listing.
• Sourced taken from a cited source (a vendor page, a price tracker, the model card) but not independently checked.
• Estimate calculated with the formulas below. Not a measurement.
• Not yet measured no public, citable score exists for that benchmark yet. Used in the special-purpose model rankings instead of a guess.
• Sample placeholder data for layout. In this preview, KL-divergence values, the mockup's Gemma 3 27B speed figures, Qwen3 8B/14B quality and some prices and build parts are still sample data.

Dates and review intervals

Prices are due for review 30 days after their recorded check. Specifications and benchmark collections are due after 90 days. An overdue record remains available with its date and evidence label; it is not a current price promise. Undated or future-dated records have unknown freshness. A runtime, model or quant change needs a new test; a recent metadata check does not refresh an old benchmark.

Measured performance means a cited benchmark with recorded model, hardware, quant, runtime and test conditions. Verified means metadata checked against its primary source; it does not mean we ran the model. Measured-only rankings require a matching recorded context. Missing drivers, build hashes, power limits or cache settings remain unknown, and third-party results are not independently reproduced.

Methodology

Speed (planned bench)

Runtimes: llama.cpp (CUDA, Metal, Vulkan, ROCm) as the baseline, MLX on Apple Silicon, vLLM optionally for serving. Tool: llama-bench -m model.gguf -p 512 -n 128 -pg 512,128 -r 5 -o json, which matches the community threads so our numbers are comparable. We add long-context points at 4k, 16k and 32k depth, record build hash, driver, power limit, flash-attention and KV-cache settings, and measure wall power for tokens per joule. A run is accepted when the coefficient of variation across repeats is under 5%; we report the median and the min–max range.

Quality

Quant fidelity is KL-divergence and top-token agreement against the BF16 reference with llama-perplexity --kl-divergence on a fixed corpus. Capability scores come from the LMArena leaderboard dataset (CC-BY-4.0). Speed and quality are never merged into one score.

Estimates (where we haven't measured)

W   = GGUF file bytes / 1e9            (fallback: params × bpw / 8)
KV  = 2 × layers × kv_heads × head_dim × ctx × users × bytes/elem / 1e9
      sliding-window layers use min(ctx, window)
O   = min(1.5, 0.5 + ctx / 131072)  [EST]
M   = W + KV + O
U   = VRAM − 0.5 (discrete) | RAM × 0.75 (unified; 0.90 raised) | RAM × 0.75 (CPU)
fit when M ≤ 0.9 × U; tight when M ≤ U; else offload or no fit
tps = η × bandwidth / (W × active/total + KV)     η: 0.60 GPU, 0.45 unified, 0.50 CPU
offload: tps = 1 / (B_gpu/(η·BW_gpu) + B_cpu/(η_c·BW_ram))
prefill = 0.35 × FLOPS / (2 × active params); TTFT = prompt tokens / prefill
E_month = (W_load × h_load + W_idle × h_idle) × 30 / 1000 × $/kWh
break-even = hardware / (plan − E_month)

η is the share of theoretical bandwidth a runtime achieves. A LocalScore run (LLaMA-2 6.7B Q4_0 on an M4 Max, 62.5 tok/s) implies η ≈ 0.43, below the 0.65–0.70 that other calculators assume. Published llama.cpp runs on each hardware page (Llama 2 7B Q4_0 calibration test) give a per-device check. We will fit η per device class, runtime and model-size band from our own runs; until then every estimate carries a ±25% band.

Open data and licence

Data, charts and API output are licensed CC BY 4.0. Cite as: The Local Bench (thelocalbench.com), data as of YYYY-MM-DD, CC BY 4.0. Third-party data keeps its own licence; each model keeps its own licence, shown on its datasheet. See open data, the JSON API and llms.txt.

Independence

No paid placement in rankings. Sponsored content and review units will be labelled. Affiliate links, when they arrive, will never change a verdict or a rank.

Corrections policy

No silent edits to numbers. Corrections state what changed, why, and who reported it (credited with permission). Changes are recorded on the affected page.

Report a correction to vj@andorlabs.ca. Include the page URL, the value or instruction in question, and a source or reproduction steps. Please omit private prompts and credentials. Email opens in your mail app; this site does not collect or store submissions. Corrections log: none yet.

Special-purpose models and category rankings

Each of the 102 special-purpose models was checked against the Hugging Face API on 2026-10-13: the repository exists and is public (some are gated behind a licence click-through), and the parameter count (safetensors total) and licence come from the API. Each category is ranked by one public benchmark, named on the category page. We copy the published number with its source, never average different benchmarks, and mark models with no comparable public score as “not yet measured”. Vendor-reported scores use each vendor's own settings. The GIFT-Eval time-series figures are our own geometric-mean calculation from the published per-dataset results files (MASE relative to seasonal naive), the method the GIFT-Eval leaderboard describes.

Maker logos and trademarks

Small monochrome maker marks used only to identify who made a model or device. Not an endorsement. All marks are trademarks of their respective owners. We show them small, in one colour, next to the maker's name. Sources: