{"as_of":"2026-10-12","license":"CC-BY-4.0","license_url":"https://creativecommons.org/licenses/by/4.0/","cite_as":"The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0","source_url":"https://thelocalbench.com","guide":{"slug":"llama-cpp-gpu-offload","title":"Build llama.cpp and run a model with GPU offload","short":"llama.cpp with GPU offload","dek":"Compile llama.cpp from source with CUDA, download a GGUF straight from Hugging Face, and control how many layers go to the graphics card so models bigger than your VRAM still run.","summary":"Clone and build llama.cpp with -DGGML_CUDA=ON, start llama-server with -hf and -ngl, check /health and the OpenAI-compatible chat endpoint, then tune -ngl, -c and -fa to fit your VRAM.","difficulty":"Intermediate","time_minutes":45,"cost":{"usd":0,"note":"free software; an NVIDIA GPU for the offload steps"},"hardware":["NVIDIA GPU with 8 GB+ VRAM for the CUDA path","Any x86-64 or ARM CPU for the CPU-only path"],"os":["Linux","Windows"],"prerequisites":["git, CMake and a C++ compiler","NVIDIA driver and CUDA Toolkit (GPU path)","About 10 GB of free disk space"],"parts":[{"name":"llama.cpp","note":"built from source","href":"https://github.com/ggml-org/llama.cpp"},{"name":"CUDA Toolkit","note":"GPU path only","href":"https://developer.nvidia.com/cuda-downloads"},{"name":"Gemma 3 4B GGUF (ggml-org)","note":"test model, about 2.5 GB at Q4_K_M","href":"https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF"}],"tested_on":[{"env":"Ubuntu (Debian-based) x86-64, 8 vCPU Xeon, no GPU","date":"2026-10-13","scope":"CPU build at commit 45451c3 (2026-10-13), llama-server with -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M, /health and /v1/chat/completions"}],"last_tested":"2026-10-13","flag":"partly-tested","flag_note":"We built llama.cpp and ran the server on a CPU-only machine on 2026-10-13. The CUDA build and every -ngl offload step are copied from the official build docs and server README but have not yet been run on our GPU bench.","untested":["CUDA build (-DGGML_CUDA=ON)","Layer offload with -ngl on a real GPU","Windows build"],"reviewed":"2026-10-13","tools":["llama-cpp"],"explainers":["memory-bandwidth","kv-cache","mixture-of-experts","context-length"],"sources":[{"label":"llama.cpp build documentation (CPU and CUDA)","url":"https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md"},{"label":"llama.cpp server README (flags, /health, OpenAI-compatible API)","url":"https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md"},{"label":"ggml-org/gemma-3-4b-it-GGUF on Hugging Face","url":"https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF"}],"steps":[{"n":1,"title":"Get the source"},{"n":2,"title":"Build with CUDA"},{"n":3,"title":"Start the server with a model from Hugging Face"},{"n":4,"title":"Verify it works"},{"n":5,"title":"Fit a bigger model with partial offload"}],"url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload","markdown":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload.md","json":"https://thelocalbench.com/api/v1/how-to/llama-cpp-gpu-offload.json"},"json_ld":{"@context":"https://schema.org","@type":"HowTo","name":"Build llama.cpp and run a model with GPU offload","description":"Clone and build llama.cpp with -DGGML_CUDA=ON, start llama-server with -hf and -ngl, check /health and the OpenAI-compatible chat endpoint, then tune -ngl, -c and -fa to fit your VRAM.","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload","totalTime":"PT45M","dateModified":"2026-10-13","estimatedCost":{"@type":"MonetaryAmount","currency":"USD","value":0},"supply":[{"@type":"HowToSupply","name":"llama.cpp"},{"@type":"HowToSupply","name":"CUDA Toolkit"},{"@type":"HowToSupply","name":"Gemma 3 4B GGUF (ggml-org)"}],"tool":[{"@type":"HowToTool","name":"git, CMake and a C++ compiler"},{"@type":"HowToTool","name":"NVIDIA driver and CUDA Toolkit (GPU path)"},{"@type":"HowToTool","name":"About 10 GB of free disk space"}],"step":[{"@type":"HowToStep","position":1,"name":"Get the source","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload#step-1"},{"@type":"HowToStep","position":2,"name":"Build with CUDA","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload#step-2"},{"@type":"HowToStep","position":3,"name":"Start the server with a model from Hugging Face","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload#step-3"},{"@type":"HowToStep","position":4,"name":"Verify it works","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload#step-4"},{"@type":"HowToStep","position":5,"name":"Fit a bigger model with partial offload","url":"https://thelocalbench.com/how-to/llama-cpp-gpu-offload#step-5"}]}}