# Why is memory bandwidth the speed limit for local AI? > Each generated token requires reading every active weight from memory once, so tokens per second can't exceed bandwidth divided by gigabytes read per token. In October 2026, a used RTX 3090 (936 GB/s) running Gemma 3 27B at Q4_K_M (16.6 GB) tops out near 57 tok/s; real runs reach about 60% of that. Canonical: https://thelocalbench.com/learn/memory-bandwidth · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0 Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder Reviewed 2026-10-12 · REV 2026.10 ## The plain version When you watch a local model type, each new word piece appears only after the processor has done something surprisingly brute-force: read **every active weight in the model** from memory, once. For a 27-billion-parameter model at 4 bits, that's about 16.6 GB of reading for every single token. Modern chips can do the arithmetic on those weights far faster than memory can deliver them. So for one person chatting, the processor mostly sits waiting for data. The speed limit isn't how fast the chip computes; it's how fast memory can feed it. That rate is **memory bandwidth**, measured in gigabytes per second (GB/s). **An everyday analogy.** Picture filling a bathtub through a pipe. The size of the tub is the model: you must move all of it for every token. The width of the pipe is the bandwidth. A bigger pump (more compute) doesn't help if the pipe is narrow. Divide the pipe's flow by the tub's size and you know how many tubs per second you can fill: that's your tokens per second. **The rule of thumb.** Maximum tokens per second ≈ bandwidth ÷ gigabytes read per token. Real software reaches roughly 45–70% of that ceiling, so multiply by about 0.6 for a graphics card and about 0.45 for unified-memory machines until we've measured your exact setup. **Capacity and speed are different things.** How much memory you have decides *what fits*. How fast it is decides *how fast it runs*. That's the whole story of today's home-AI hardware: - A **used RTX 3090** has only 24 GB, but 936 GB/s. Fast, if the model fits. - A **128 GB Strix Halo mini PC** holds far larger models, but at about 256 GB/s. Big, but slower per gigabyte. - A **Mac Studio M5 Ultra** has both: up to 512 GB at 1.2 TB/s, at a price. - A **CPU-only desktop** with dual-channel DDR5 manages about 90 GB/s: fine for small or MoE models, painful for big dense ones. **Why mixture-of-experts models change the picture.** In an MoE model only a few "experts" run for each token, so only their weights are read. gpt-oss-120b is a 63 GB file, but each token reads only about 2.8 GB. On a 256 GB/s mini PC that's a ceiling around 90 tok/s, which is why 128 GB unified-memory boxes and MoE models are such a good match, and why a dense 70B model on the same box crawls. > **Why it matters for your rig** > Llama 3.3 70B at Q4_K_M (42.5 GB, dense) on a Strix Halo 128 GB: ceiling 256 ÷ 42.5 ≈ **6 tok/s**, so expect about 3. On a Mac Studio M5 Ultra: 1,200 ÷ 42.5 ≈ 28, so expect about 13. gpt-oss-120b on the same Strix Halo: ceiling ≈ 92, expect about 40. Same box, a 14× difference, all from bytes read per token. *[Diagram: decode ceiling by device — see the HTML page]* **What bandwidth doesn't control.** Reading your prompt before the reply starts ("prefill") processes thousands of tokens in parallel, so it's limited by compute instead. That's where a 4090 pulls ahead of a 3090, and where Macs and mini PCs feel slower on long documents even when their typing speed is fine. ## The technical layer ### The estimate we use Single-stream decode is memory-bound (brief §4.3 and §7.3 formula 6): ``` bytes_tok = W_active + KV_read(ctx) W_active = W × (active_params / total_params) # MoE; = W for dense tps ≈ η × BW / bytes_tok ``` η is the share of theoretical bandwidth the runtime achieves. Our defaults are 0.60 (discrete GPU), 0.45 (unified memory) and 0.50 (CPU), labelled *estimate* until calibrated. canirun.ai uses a flat 0.70 / 0.65. Measured data points lower: LocalScore's LLaMA-2 6.7B Q4_0 run on an M4 Max 40-core GPU reached 62.5 tok/s. With 546 GB/s and about 3.77 GB of weights, the ceiling is ≈145 tok/s, so η ≈ 0.43. We'll fit η per device class × runtime × model-size band from our own runs. ### Offloading When the model doesn't fit, layers on the CPU are read at system-RAM bandwidth. Time per token is additive: ``` t_token = B_gpu / (η_g · BW_gpu) + B_cpu / (η_c · BW_ram) tps = 1 / t_token ``` With a 3090 (936 GB/s) and DDR5 (≈90 GB/s), putting just 20% of a model's bytes on the CPU roughly halves the speed: the slow part dominates. ### Multi-GPU Splitting layers across two cards (`--split-mode layer`, llama.cpp's default) adds **capacity, not bandwidth**: for one stream, each token still passes through each card in turn. Tensor parallelism (`-sm row`, or vLLM's `--tensor-parallel-size 2`) can use both cards' bandwidth at once, at the cost of PCIe traffic; gains on consumer boards without NVLink are typically modest. The configurator models layer split only. ### Prefill and time to first token ``` pp_tps ≈ η_c × FLOPS / (2 × active_params) TTFT ≈ prompt_tokens / pp_tps ``` Use dense FP16 tensor throughput for FLOPS. An 8k-token RAG prompt is where this matters: a Strix Halo can take several seconds before the first token on a 27B model where a 4090 takes about one. ### Measure it yourself ``` llama-bench -m model.gguf -p 512 -n 128 -r 5 -fa 1 -o json ``` `tg128` is decode (bandwidth-bound), `pp512` is prefill (compute-bound). Divide your model's file size by the tg128 result's seconds-per-token to see your effective GB/s, then divide by the spec bandwidth to get your η. ### Pitfalls - **Spec bandwidth is a peak.** Memory clocks under power limits, or a GPU in a PCIe x4 slot for multi-GPU, change what you get. - **The KV cache counts.** At long context, bytes per token grow and speed drops (see [KV cache](/learn/kv-cache)). - **Batching changes everything.** With many concurrent users, weights are read once per *batch*, and serving becomes compute-bound. These formulas are for one person at a keyboard. ## Sources - [Concept brief §4.3: estimation method and η calibration](/about#methodology) - [LocalScore: LLaMA-2 6.7B Q4_0 on Apple M4 Max (62.5 tok/s)](https://www.localscore.ai/model/1248) - [canirun.ai: method and efficiency assumptions](https://canirun.ai/why/) - [llama.cpp Apple Silicon performance thread (#4167)](https://github.com/ggml-org/llama.cpp/discussions/4167) - [llama.cpp CUDA performance thread (#15013)](https://github.com/ggml-org/llama.cpp/discussions/15013) Try it: https://thelocalbench.com/configurator?mode=hardware&hw=ryzen-ai-max-395-128g