The plain version
When you watch a local model type, each new word piece appears only after the processor has done something surprisingly brute-force: read every active weight in the model from memory, once. For a 27-billion-parameter model at 4 bits, that's about 16.6 GB of reading for every single token.
Modern chips can do the arithmetic on those weights far faster than memory can deliver them. So for one person chatting, the processor mostly sits waiting for data. The speed limit isn't how fast the chip computes; it's how fast memory can feed it. That rate is memory bandwidth, measured in gigabytes per second (GB/s).
An everyday analogy. Picture filling a bathtub through a pipe. The size of the tub is the model: you must move all of it for every token. The width of the pipe is the bandwidth. A bigger pump (more compute) doesn't help if the pipe is narrow. Divide the pipe's flow by the tub's size and you know how many tubs per second you can fill: that's your tokens per second.
The rule of thumb. Maximum tokens per second ≈ bandwidth ÷ gigabytes read per token. Real software reaches roughly 45–70% of that ceiling, so multiply by about 0.6 for a graphics card and about 0.45 for unified-memory machines until we've measured your exact setup.
Capacity and speed are different things. How much memory you have decides what fits. How fast it is decides how fast it runs. That's the whole story of today's home-AI hardware:
- A used RTX 3090 has only 24 GB, but 936 GB/s. Fast, if the model fits.
- A 128 GB Strix Halo mini PC holds far larger models, but at about 256 GB/s. Big, but slower per gigabyte.
- A Mac Studio M5 Ultra has both: up to 512 GB at 1.2 TB/s, at a price.
- A CPU-only desktop with dual-channel DDR5 manages about 90 GB/s: fine for small or MoE models, painful for big dense ones.
Why mixture-of-experts models change the picture. In an MoE model only a few "experts" run for each token, so only their weights are read. gpt-oss-120b is a 63 GB file, but each token reads only about 2.8 GB. On a 256 GB/s mini PC that's a ceiling around 90 tok/s, which is why 128 GB unified-memory boxes and MoE models are such a good match, and why a dense 70B model on the same box crawls.
What bandwidth doesn't control. Reading your prompt before the reply starts ("prefill") processes thousands of tokens in parallel, so it's limited by compute instead. That's where a 4090 pulls ahead of a 3090, and where Macs and mini PCs feel slower on long documents even when their typing speed is fine.
The estimate we use
Single-stream decode is memory-bound (brief §4.3 and §7.3 formula 6):
bytes_tok = W_active + KV_read(ctx) W_active = W × (active_params / total_params) # MoE; = W for dense tps ≈ η × BW / bytes_tokη is the share of theoretical bandwidth the runtime achieves. Our defaults are 0.60 (discrete GPU), 0.45 (unified memory) and 0.50 (CPU), labelled estimate until calibrated. canirun.ai uses a flat 0.70 / 0.65. Measured data points lower: LocalScore's LLaMA-2 6.7B Q4_0 run on an M4 Max 40-core GPU reached 62.5 tok/s. With 546 GB/s and about 3.77 GB of weights, the ceiling is ≈145 tok/s, so η ≈ 0.43. We'll fit η per device class × runtime × model-size band from our own runs.
Offloading
When the model doesn't fit, layers on the CPU are read at system-RAM bandwidth. Time per token is additive:
t_token = B_gpu / (η_g · BW_gpu) + B_cpu / (η_c · BW_ram) tps = 1 / t_tokenWith a 3090 (936 GB/s) and DDR5 (≈90 GB/s), putting just 20% of a model's bytes on the CPU roughly halves the speed: the slow part dominates.
Multi-GPU
Splitting layers across two cards (
--split-mode layer, llama.cpp's default) adds capacity, not bandwidth: for one stream, each token still passes through each card in turn. Tensor parallelism (-sm row, or vLLM's--tensor-parallel-size 2) can use both cards' bandwidth at once, at the cost of PCIe traffic; gains on consumer boards without NVLink are typically modest. The configurator models layer split only.Prefill and time to first token
pp_tps ≈ η_c × FLOPS / (2 × active_params) TTFT ≈ prompt_tokens / pp_tpsUse dense FP16 tensor throughput for FLOPS. An 8k-token RAG prompt is where this matters: a Strix Halo can take several seconds before the first token on a 27B model where a 4090 takes about one.
Measure it yourself
llama-bench -m model.gguf -p 512 -n 128 -r 5 -fa 1 -o jsontg128is decode (bandwidth-bound),pp512is prefill (compute-bound). Divide your model's file size by the tg128 result's seconds-per-token to see your effective GB/s, then divide by the spec bandwidth to get your η.Pitfalls
- Spec bandwidth is a peak. Memory clocks under power limits, or a GPU in a PCIe x4 slot for multi-GPU, change what you get.
- The KV cache counts. At long context, bytes per token grow and speed drops (see KV cache).
- Batching changes everything. With many concurrent users, weights are read once per batch, and serving becomes compute-bound. These formulas are for one person at a keyboard.