# Build llama.cpp and run a model with GPU offload

> Clone and build llama.cpp with -DGGML_CUDA=ON, start llama-server with -hf and -ngl, check /health and the OpenAI-compatible chat endpoint, then tune -ngl, -c and -fa to fit your VRAM.

Canonical: https://thelocalbench.com/how-to/llama-cpp-gpu-offload · The Local Bench (thelocalbench.com), data as of 2026-10-13, CC BY 4.0

- Difficulty: Intermediate
- Time: about 45 minutes
- Cost: $0 (free software; an NVIDIA GPU for the offload steps)
- Hardware: NVIDIA GPU with 8 GB+ VRAM for the CUDA path; Any x86-64 or ARM CPU for the CPU-only path
- OS: Linux, Windows
- Prerequisites: git, CMake and a C++ compiler; NVIDIA driver and CUDA Toolkit (GPU path); About 10 GB of free disk space
- Parts: llama.cpp (built from source); CUDA Toolkit (GPU path only); Gemma 3 4B GGUF (ggml-org) (test model, about 2.5 GB at Q4_K_M)
- Status: Partly tested. We built llama.cpp and ran the server on a CPU-only machine on 2026-10-13. The CUDA build and every -ngl offload step are copied from the official build docs and server README but have not yet been run on our GPU bench.
- Tested on: Ubuntu (Debian-based) x86-64, 8 vCPU Xeon, no GPU, 2026-10-13: CPU build at commit 45451c3 (2026-10-13), llama-server with -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M, /health and /v1/chat/completions
- Last tested: 2026-10-13
- Reviewed: 2026-10-13

## Not yet tested by us

- CUDA build (-DGGML_CUDA=ON)
- Layer offload with -ngl on a real GPU
- Windows build

## Step 1: Get the source

```sh
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
```

> **Tip: Just want binaries?**
>
The llama.cpp README also lists prebuilt releases and package-manager installs. Building yourself gets you the newest CUDA kernels and works with any CUDA version you have installed.




## Step 2: Build with CUDA

```sh
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
```

For a CPU-only build, leave out `-DGGML_CUDA=ON`. The binaries end up in `build/bin/`. Add `-j 8` (or your core count) to the second command to build faster.

> **Warning: CUDA not found**
>
If CMake says it can't find CUDA, install the CUDA Toolkit (not just the driver) and make sure `nvcc --version` works in the same terminal.




## Step 3: Start the server with a model from Hugging Face

`-hf` downloads a GGUF and caches it. `-ngl` sets how many layers go to the GPU.

```sh
./build/bin/llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M -ngl 99 -c 8192 --port 8080
```

```text
# Expected output
… llama_server: model loaded
… llama_server: listening on http://127.0.0.1:8080
```

> **Tip: Ports and defaults**
>
Current builds listen on 127.0.0.1 port 9931 unless you pass `--port`. `-ngl` defaults to `auto`, which fits as many layers as it can, so passing a number is only needed when you want to control the split yourself.




## Step 4: Verify it works

```sh
curl http://localhost:8080/health
```

```text
# Expected output
{"status":"ok"}
```

Then send a chat request to the OpenAI-compatible endpoint:

```sh
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "Say hello in five words."}]}'
```

The reply is JSON with the answer in `choices[0].message.content` and speed in `timings.predicted_per_second`. You can also open http://localhost:8080 for the built-in web chat.



## Step 5: Fit a bigger model with partial offload

When a model is larger than your VRAM, lower `-ngl` so some layers stay in system RAM. Each layer kept on the CPU costs speed, because it's read at RAM bandwidth.

```sh
./build/bin/llama-server -m /path/to/model-Q4_K_M.gguf -ngl 30 -c 8192 -fa on --port 8080
```

Watch the load log for how many layers were offloaded, and `nvidia-smi` for memory. Raise `-ngl` until VRAM is nearly full, leaving about 1 GB of headroom. For MoE models, keep `-ngl 99` and use `--n-cpu-moe N` instead, which moves only expert weights to the CPU.





## Troubleshooting

### Out of memory when loading

Lower `-ngl` by 5 at a time, or reduce `-c`: the KV cache grows with context. `-fa on` plus `-ctk q8_0 -ctv q8_0` roughly halves KV-cache memory.



### It runs, but on the CPU

The binary was built without CUDA. Rebuild with `-DGGML_CUDA=ON` into a fresh `build` directory, and check that the log lists your GPU at start-up.



### -hf download fails

Check the repository and quant tag exist on Hugging Face. Gated models need a token, set with the `HF_TOKEN` environment variable. Or download the `.gguf` yourself and use `-m`.



### Other apps can't connect

By default the server listens only on 127.0.0.1. To serve your network, add `--host 0.0.0.0` and set `--api-key` so not everyone on the network can use it.

## Sources

- [llama.cpp build documentation (CPU and CUDA)](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)
- [llama.cpp server README (flags, /health, OpenAI-compatible API)](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- [ggml-org/gemma-3-4b-it-GGUF on Hugging Face](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF)