# GGUF, EXL2/EXL3, AWQ or GPTQ — which quantized file should you download?

> Choose by runtime, not by name. GGUF is for llama.cpp, Ollama and LM Studio, and runs on CPUs, Macs and any GPU. EXL2 and EXL3 are for ExLlama on NVIDIA GPUs and allow any average bit rate. AWQ and GPTQ are 4-bit GPU formats for vLLM-style servers. At similar bits per weight, quality differences are small.

Canonical: https://thelocalbench.com/learn/quantization-formats · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0

Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder

Reviewed 2026-10-13 · REV 2026.10

## The plain version

The [quantization explainer](/learn/quantization) covers *why* you shrink a model. This one is about the **containers** those shrunk models come in. Open any popular model on Hugging Face and you'll find the same weights packaged five ways: `-GGUF`, `-exl2`, `-exl3`, `-AWQ`, `-GPTQ`. They aren't interchangeable. Each format belongs to a family of runtimes, so pick your software first and the format follows.

**An everyday analogy.** It's like video files. The same film can come as MP4, MKV or a Blu-ray disc image. Quality depends far more on the bit rate than on the container, but your player decides which ones will open at all.

| Format | Made for | Runs on | Bit rates | Pick it when |
| --- | --- | --- | --- | --- |
| GGUF | llama.cpp, Ollama, LM Studio, Jan | CPU, Apple, NVIDIA, AMD, Intel; mixed CPU+GPU | Fixed presets (Q2_K … Q8_0) | Almost always, and always on a Mac, a mini PC, or a GPU that's too small |
| EXL2 / EXL3 | ExLlamaV2/V3 via TabbyAPI | NVIDIA GPUs (CUDA) | Any average, about 2–8 bpw | Everything fits in VRAM and you want the best quality per gigabyte |
| AWQ | vLLM, SGLang, Transformers | NVIDIA and other GPUs | Usually 4-bit | Serving several users on a GPU server |
| GPTQ | vLLM, Transformers, ExLlama | NVIDIA and other GPUs | 2–8 bit, usually 4 | As AWQ; older but widely available |

**GGUF** is the default for home use. It's a single file holding the weights *and* everything needed to run them (the tokenizer, the chat template, and the architecture details). It loads fast and runs on almost anything, and llama.cpp can split it between your graphics card and system RAM when it doesn't fit. Its `Q4_K_M`-style names are presets of the k-quant family.

**EXL2 and EXL3** come from the ExLlama project and target NVIDIA GPUs only. Their trick is **any bit rate you like**: EXL2 mixes 2- to 8-bit pieces inside one model to hit an exact average, such as 4.65 bits per weight, so you can fill a 24 GB card precisely. ExLlamaV2 is now archived. Development continues in ExLlamaV3, whose EXL3 format is based on the QTIP method.

**AWQ and GPTQ** are 4-bit-centric GPU formats from research papers. GPTQ uses approximate second-order information to quantize one layer at a time. AWQ notices that protecting about 1% of "salient" weights, found by looking at the activations, removes most of the error. Both are first-class in vLLM and SGLang, the servers built for many simultaneous users.

> **Why it matters for your rig**
>

**Mac, Strix Halo or CPU:** GGUF, full stop; the others won't run. **One 24 GB NVIDIA card, single user, model fits:** GGUF Q4_K_M is the easy choice; EXL3 at about 4–5 bpw can be a little sharper or faster per gigabyte. **Serving a household or a team from a GPU box:** AWQ or GPTQ in vLLM, for batching and paged attention. **Model too big for VRAM:** only GGUF can spill gracefully into system RAM.



**What about MXFP4, FP8 and NVFP4?** These are *number formats* rather than file families. gpt-oss ships its MoE weights in MXFP4, a 4-bit floating-point format with a shared scale per small block. GGUF, vLLM and others can all carry it. FP8 and NVFP4 need recent NVIDIA hardware to get the speed benefit.

## The technical layer

### Comparing quality across formats

Bits per weight (bpw) is the fair unit, not the name. Q4_K_M averages about 4.8–4.9 bpw on most models, so compare it with EXL3 at 4.8, not 4.0. The best single metric is KL divergence against the full-precision model on your own kind of text. Perplexity alone hides tail errors.

### Calibration data

GPTQ, AWQ and EXL2/EXL3 all use a calibration set to decide where precision matters. GGUF k-quants work without one, but an importance matrix (`--imatrix`) brings calibration to GGUF too. Calibration on mostly English text can cost more in other languages or in code; check what the uploader used.

### Running each one

```
# GGUF
llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M --port 8080
# AWQ / GPTQ in vLLM (the quantization method is read from the model config)
vllm serve <org>/<model>-AWQ
```

EXL2/EXL3 run through TabbyAPI, which the ExLlama README recommends as the official server. Its configuration lives in a YAML file rather than in command-line flags.

### Pitfalls

- **Old GGUFs break chat templates.** Re-download if tool calling or "thinking" misbehaves; newer files embed fixed templates.
- **Same name, different recipe.** Two "Q4_K_M" uploads with and without an imatrix differ measurably.
- **4-bit AWQ is not 4.0 bpw.** Group scales add about 0.15–0.25 bpw, so check the actual file size.

## Sources

- [ggml docs: GGUF file format specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
- [ExLlamaV2 README: EXL2 format (archived; continues as ExLlamaV3)](https://github.com/turboderp-org/exllamav2)
- [ExLlamaV3 README: EXL3, based on QTIP](https://github.com/turboderp-org/exllamav3)
- [Lin et al., "AWQ: Activation-aware Weight Quantization" (arXiv 2306.00978)](https://arxiv.org/abs/2306.00978)
- [Frantar et al., "GPTQ" (arXiv 2210.17323)](https://arxiv.org/abs/2210.17323)
- [vLLM docs: supported quantization methods](https://docs.vllm.ai/en/latest/features/quantization/index.html)

Try it: https://thelocalbench.com/configurator?mode=hardware&hw=rtx-3090-24g-used