Build llama.cpp and run a model with GPU offload

Get the source
git clone https://github.com/ggml-org/llama.cpp cd llama.cpp
Just want binaries?
The llama.cpp README also lists prebuilt releases and package-manager installs. Building yourself gets you the newest CUDA kernels and works with any CUDA version you have installed.
Build with CUDA
cmake -B build -DGGML_CUDA=ON cmake --build build --config Release
For a CPU-only build, leave out -DGGML_CUDA=ON. The binaries end up in build/bin/. Add -j 8 (or your core count) to the second command to build faster.
CUDA not found
If CMake says it can't find CUDA, install the CUDA Toolkit (not just the driver) and make sure nvcc --version works in the same terminal.
Start the server with a model from Hugging Face
-hf downloads a GGUF and caches it. -ngl sets how many layers go to the GPU.
./build/bin/llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M -ngl 99 -c 8192 --port 8080
Expected output
… llama_server: model loaded … llama_server: listening on http://127.0.0.1:8080
Ports and defaults
Current builds listen on 127.0.0.1 port 9931 unless you pass --port. -ngl defaults to auto, which fits as many layers as it can, so passing a number is only needed when you want to control the split yourself.
Verify it works
Check
curl http://localhost:8080/health
Expected output
{"status":"ok"}
Then send a chat request to the OpenAI-compatible endpoint:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "Say hello in five words."}]}'
The reply is JSON with the answer in choices[0].message.content and speed in timings.predicted_per_second. You can also open http://localhost:8080 for the built-in web chat.
Fit a bigger model with partial offload
When a model is larger than your VRAM, lower -ngl so some layers stay in system RAM. Each layer kept on the CPU costs speed, because it's read at RAM bandwidth.
./build/bin/llama-server -m /path/to/model-Q4_K_M.gguf -ngl 30 -c 8192 -fa on --port 8080
Watch the load log for how many layers were offloaded, and nvidia-smi for memory. Raise -ngl until VRAM is nearly full, leaving about 1 GB of headroom. For MoE models, keep -ngl 99 and use --n-cpu-moe N instead, which moves only expert weights to the CPU.

Troubleshooting

Lower -ngl by 5 at a time, or reduce -c: the KV cache grows with context. -fa on plus -ctk q8_0 -ctv q8_0 roughly halves KV-cache memory.
The binary was built without CUDA. Rebuild with -DGGML_CUDA=ON into a fresh build directory, and check that the log lists your GPU at start-up.
Check the repository and quant tag exist on Hugging Face. Gated models need a token, set with the HF_TOKEN environment variable. Or download the .gguf yourself and use -m.
By default the server listens only on 127.0.0.1. To serve your network, add --host 0.0.0.0 and set --api-key so not everyone on the network can use it.