Skip to main contentThe Local Bench

Explainer · REV 2026.10 · reviewed 2026-10-13 · 8 min

What is a mixture-of-experts model, and why does "30B-A3B" run so fast?

Why a model can be 117 billion parameters and still type quickly on a mini PC, what "active parameters" means for your memory and your speed, and when MoE is the wrong choice.

Mini PC

The 50-word answer

A mixture-of-experts (MoE) model splits each layer into many expert sub-networks and runs only a few per token. You must store every expert, but each token reads only the active ones, so Qwen3-30B-A3B (30.5B total, 3.3B active) needs memory like a 30B model yet decodes at speeds closer to a 3B one.

The plain version

A normal ("dense") language model uses all of its weights for every word piece it writes. A mixture-of-experts model doesn't. Inside each layer it keeps a crowd of smaller sub-networks, the experts, plus a tiny router that looks at the current token and picks a handful of them to do the work. The rest sit idle for that token.

An everyday analogy. Think of a large hospital. It employs hundreds of specialists, and the building has to be big enough for all of them. But any one patient sees only two or three. The hospital's size is the model's total parameters; the doctors you actually see are its active parameters.

That's what names like Qwen3-30B-A3B mean: 30 billion parameters in total, about 3 billion active per token. Its model card lists 128 experts per layer with 8 chosen for each token.

Why that matters on your hardware. In the memory bandwidth explainer we showed that typing speed is roughly bandwidth ÷ bytes read per token. An MoE model reads only its active experts, so memory capacity is set by the total size, but speed is set by the active size.

Total versus active parameters per tokenGemma 3 27B IT: 27.4B total, 27.4B active; Qwen3 32B: 32.8B total, 32.8B active; Qwen3 30B-A3B: 30.5B total, 3.3B active; gpt-oss-20b: 20.9B total, 3.6B active; gpt-oss-120b: 116.8B total, 5.1B active.Gemma 3 27B IT27.4B of 27.4BQwen3 32B32.8B of 32.8BQwen3 30B-A3B3.3B of 30.5Bgpt-oss-20b3.6B of 20.9Bgpt-oss-120b5.1B of 116.8B
Fig. 1 · Grey is what you must hold in memory; the solid bar is what is read for each token. Dense models (dark) read everything; MoE models (blue) read a small slice. Parameter counts from each model card via our datasheets.

The numbers. Qwen3-30B-A3B at Q4_K_M is an 18.6 GB file, roughly the same as dense Qwen3-32B (19.8 GB). But only about 11% of the weights are active, so each token reads about 2 GB instead of about 20 GB. On a 256 GB/s Strix Halo mini PC, that's a ceiling around 125 tok/s versus about 13 for the dense model. Real runs land well below the ceiling, but the order-of-magnitude gap holds.

The big one. gpt-oss-120b has 117 billion parameters but only 5.1 billion active. OpenAI ships it in MXFP4 so it fits on one 80 GB GPU, and it also fits comfortably in a 128 GB unified-memory machine. That's the reason MoE models and big-but-slow-memory boxes are such a natural pair.

When MoE isn't the answer. Quality per gigabyte is lower: a 30B-A3B usually behaves more like a strong mid-size dense model than like a 30B dense one. Reading your prompt ("prefill") still costs compute in proportion to the active parameters, which is fine. But fine-tuning, and serving many users at once, are more complicated. If your hardware has fast memory but little of it, a dense model that fits is often the better pick.

  • Routing in one paragraph

    Each MoE layer replaces the feed-forward block with E experts. A learned router scores all E for the current hidden state and keeps the top k. Their outputs are combined, weighted by the router scores. Mixtral 8x7B uses E = 8, k = 2 (47B total, 13B active). Qwen3-30B-A3B uses E = 128, k = 8. Attention layers, embeddings and the router itself are shared and always active. That's why active parameters are a bit more than k/E of the total.

    The estimate we use

    W_active  = W × (active_params / total_params)
tps       ≈ η × BW / (W_active + KV_read)

    For Qwen3-30B-A3B at Q4_K_M: 18.56 GB × (3.3 / 30.53) ≈ 2.0 GB per token.

    Splitting experts across GPU and CPU

    llama-server -m model.gguf -ngl 999 --n-cpu-moe 20 -c 16384

    -ngl puts all layers on the GPU, then --n-cpu-moe N moves the expert weights of the first N layers back to system RAM (--cpu-moe moves all of them). Attention and the KV cache stay on the GPU, where they matter most. Raise N until the model fits. Each step costs speed, because those expert reads now run at RAM bandwidth.

    Pitfalls

    • Routing isn't uniform. Different tokens pick different experts, so you can't predict which experts to keep on the GPU. All of them must be loaded somewhere.
    • "Active" excludes the KV cache. Long contexts still cost memory and bandwidth like any other model (see KV cache).
    • Batching erodes the advantage. With many concurrent users, different tokens hit different experts, and more of the model is read per step.

Sources

  1. Jiang et al., "Mixtral of Experts" (arXiv 2401.04088)
  2. Fedus et al., "Switch Transformers" (arXiv 2101.03961)
  3. Qwen3-30B-A3B model card (128 experts, 8 active)
  4. gpt-oss-120b model card (117B total, 5.1B active)
  5. llama.cpp server README: --cpu-moe and --n-cpu-moe

Other explainers: Context length and RoPE · KV cache · Memory bandwidth · GGUF vs EXL vs AWQ · Quantization · RAG and embeddings · Speculative decoding · Tool calling and MCP