# What is a mixture-of-experts model, and why does "30B-A3B" run so fast?

> A mixture-of-experts (MoE) model splits each layer into many expert sub-networks and runs only a few per token. You must store every expert, but each token reads only the active ones, so Qwen3-30B-A3B (30.5B total, 3.3B active) needs memory like a 30B model yet decodes at speeds closer to a 3B one.

Canonical: https://thelocalbench.com/learn/mixture-of-experts · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0

Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder

Reviewed 2026-10-13 · REV 2026.10

## The plain version

A normal ("dense") language model uses all of its weights for every word piece it writes. A **mixture-of-experts** model doesn't. Inside each layer it keeps a crowd of smaller sub-networks, the *experts*, plus a tiny *router* that looks at the current token and picks a handful of them to do the work. The rest sit idle for that token.

**An everyday analogy.** Think of a large hospital. It employs hundreds of specialists, and the building has to be big enough for all of them. But any one patient sees only two or three. The hospital's size is the model's **total parameters**; the doctors you actually see are its **active parameters**.

That's what names like **Qwen3-30B-A3B** mean: 30 billion parameters in total, about 3 billion active per token. Its model card lists 128 experts per layer with 8 chosen for each token.

**Why that matters on your hardware.** In the [memory bandwidth explainer](/learn/memory-bandwidth) we showed that typing speed is roughly *bandwidth ÷ bytes read per token*. An MoE model reads only its active experts, so memory capacity is set by the total size, but speed is set by the active size.

<MoEDiagram />

**The numbers.** Qwen3-30B-A3B at Q4_K_M is an 18.6 GB file, roughly the same as dense Qwen3-32B (19.8 GB). But only about 11% of the weights are active, so each token reads about 2 GB instead of about 20 GB. On a 256 GB/s Strix Halo mini PC, that's a ceiling around 125 tok/s versus about 13 for the dense model. Real runs land well below the ceiling, but the order-of-magnitude gap holds.

**The big one.** gpt-oss-120b has 117 billion parameters but only 5.1 billion active. OpenAI ships it in MXFP4 so it fits on one 80 GB GPU, and it also fits comfortably in a 128 GB unified-memory machine. That's the reason MoE models and big-but-slow-memory boxes are such a natural pair.

> **Why it matters for your rig**
>

**128 GB mini PC or Mac:** pick MoE. You have the room for all the experts and not much bandwidth, which is the exact trade MoE makes. **24 GB GPU:** a 30B-A3B fits at 4 bits with room for context and flies. A 120B MoE doesn't fit, but llama.cpp can keep expert weights in system RAM (`--n-cpu-moe`) and the result is often usable. **CPU only:** MoE is the only way large models feel interactive.



**When MoE isn't the answer.** Quality per gigabyte is lower: a 30B-A3B usually behaves more like a strong mid-size dense model than like a 30B dense one. Reading your prompt ("prefill") still costs compute in proportion to the active parameters, which is fine. But fine-tuning, and serving many users at once, are more complicated. If your hardware has fast memory but little of it, a dense model that fits is often the better pick.

## The technical layer

### Routing in one paragraph

Each MoE layer replaces the feed-forward block with *E* experts. A learned router scores all *E* for the current hidden state and keeps the top *k*. Their outputs are combined, weighted by the router scores. Mixtral 8x7B uses *E* = 8, *k* = 2 (47B total, 13B active). Qwen3-30B-A3B uses *E* = 128, *k* = 8. Attention layers, embeddings and the router itself are shared and always active. That's why active parameters are a bit more than *k*/*E* of the total.

### The estimate we use

```
W_active  = W × (active_params / total_params)
tps       ≈ η × BW / (W_active + KV_read)
```

For Qwen3-30B-A3B at Q4_K_M: 18.56 GB × (3.3 / 30.53) ≈ 2.0 GB per token.

### Splitting experts across GPU and CPU

```
llama-server -m model.gguf -ngl 999 --n-cpu-moe 20 -c 16384
```

`-ngl` puts all layers on the GPU, then `--n-cpu-moe N` moves the expert weights of the first *N* layers back to system RAM (`--cpu-moe` moves all of them). Attention and the KV cache stay on the GPU, where they matter most. Raise *N* until the model fits. Each step costs speed, because those expert reads now run at RAM bandwidth.

### Pitfalls

- **Routing isn't uniform.** Different tokens pick different experts, so you can't predict which experts to keep on the GPU. All of them must be loaded somewhere.
- **"Active" excludes the KV cache.** Long contexts still cost memory and bandwidth like any other model (see [KV cache](/learn/kv-cache)).
- **Batching erodes the advantage.** With many concurrent users, different tokens hit different experts, and more of the model is read per step.

## Sources

- [Jiang et al., "Mixtral of Experts" (arXiv 2401.04088)](https://arxiv.org/abs/2401.04088)
- [Fedus et al., "Switch Transformers" (arXiv 2101.03961)](https://arxiv.org/abs/2101.03961)
- [Qwen3-30B-A3B model card (128 experts, 8 active)](https://huggingface.co/Qwen/Qwen3-30B-A3B)
- [gpt-oss-120b model card (117B total, 5.1B active)](https://huggingface.co/openai/gpt-oss-120b)
- [llama.cpp server README: --cpu-moe and --n-cpu-moe](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)

Try it: https://thelocalbench.com/configurator?mode=hardware&hw=ryzen-ai-max-395-128g