The plain version
The quantization explainer covers why you shrink a model. This one is about the containers those shrunk models come in. Open any popular model on Hugging Face and you'll find the same weights packaged five ways: -GGUF, -exl2, -exl3, -AWQ, -GPTQ. They aren't interchangeable. Each format belongs to a family of runtimes, so pick your software first and the format follows.
An everyday analogy. It's like video files. The same film can come as MP4, MKV or a Blu-ray disc image. Quality depends far more on the bit rate than on the container, but your player decides which ones will open at all.
Format | Made for | Runs on | Bit rates | Pick it when |
|---|---|---|---|---|
| GGUF | llama.cpp, Ollama, LM Studio, Jan | CPU, Apple, NVIDIA, AMD, Intel; mixed CPU+GPU | Fixed presets (Q2_K … Q8_0) | Almost always, and always on a Mac, a mini PC, or a GPU that's too small |
| EXL2 / EXL3 | ExLlamaV2/V3 via TabbyAPI | NVIDIA GPUs (CUDA) | Any average, about 2–8 bpw | Everything fits in VRAM and you want the best quality per gigabyte |
| AWQ | vLLM, SGLang, Transformers | NVIDIA and other GPUs | Usually 4-bit | Serving several users on a GPU server |
| GPTQ | vLLM, Transformers, ExLlama | NVIDIA and other GPUs | 2–8 bit, usually 4 | As AWQ; older but widely available |
GGUF is the default for home use. It's a single file holding the weights and everything needed to run them (the tokenizer, the chat template, and the architecture details). It loads fast and runs on almost anything, and llama.cpp can split it between your graphics card and system RAM when it doesn't fit. Its Q4_K_M-style names are presets of the k-quant family.
EXL2 and EXL3 come from the ExLlama project and target NVIDIA GPUs only. Their trick is any bit rate you like: EXL2 mixes 2- to 8-bit pieces inside one model to hit an exact average, such as 4.65 bits per weight, so you can fill a 24 GB card precisely. ExLlamaV2 is now archived. Development continues in ExLlamaV3, whose EXL3 format is based on the QTIP method.
AWQ and GPTQ are 4-bit-centric GPU formats from research papers. GPTQ uses approximate second-order information to quantize one layer at a time. AWQ notices that protecting about 1% of "salient" weights, found by looking at the activations, removes most of the error. Both are first-class in vLLM and SGLang, the servers built for many simultaneous users.
What about MXFP4, FP8 and NVFP4? These are number formats rather than file families. gpt-oss ships its MoE weights in MXFP4, a 4-bit floating-point format with a shared scale per small block. GGUF, vLLM and others can all carry it. FP8 and NVFP4 need recent NVIDIA hardware to get the speed benefit.
Comparing quality across formats
Bits per weight (bpw) is the fair unit, not the name. Q4_K_M averages about 4.8–4.9 bpw on most models, so compare it with EXL3 at 4.8, not 4.0. The best single metric is KL divergence against the full-precision model on your own kind of text. Perplexity alone hides tail errors.
Calibration data
GPTQ, AWQ and EXL2/EXL3 all use a calibration set to decide where precision matters. GGUF k-quants work without one, but an importance matrix (
--imatrix) brings calibration to GGUF too. Calibration on mostly English text can cost more in other languages or in code; check what the uploader used.Running each one
# GGUF llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M --port 8080 # AWQ / GPTQ in vLLM (the quantization method is read from the model config) vllm serve <org>/<model>-AWQEXL2/EXL3 run through TabbyAPI, which the ExLlama README recommends as the official server. Its configuration lives in a YAML file rather than in command-line flags.
Pitfalls
- Old GGUFs break chat templates. Re-download if tool calling or "thinking" misbehaves; newer files embed fixed templates.
- Same name, different recipe. Two "Q4_K_M" uploads with and without an imatrix differ measurably.
- 4-bit AWQ is not 4.0 bpw. Group scales add about 0.15–0.25 bpw, so check the actual file size.