Run your first local model with Ollama and Open WebUI

Install Ollama
On Linux, run the official install script:
curl -fsSL https://ollama.com/install.sh | sh
On Windows, in PowerShell:
irm https://ollama.com/install.ps1 | iex
On macOS, download the app from ollama.com/download, drag it to Applications, and open it once.
Read before you pipe to a shell
The Linux and Windows one-liners download and run a script. If you'd rather check it first, open the URL in a browser, or use the manual install instructions on the download page.
Download and chat with a first model
Pick a size that fits your memory. Gemma 3 4B runs on most laptops; 12B wants about 16 GB of RAM or an 8 GB+ graphics card.
ollama run gemma3:4b
The first run downloads the model, then opens a chat prompt. Type a question and press Enter; type /bye to quit.
Not sure which size?
Gemma 3 4B and 12B are not yet in our fit catalogue, so the calculator cannot validate these examples. The memory figures above are starting points, not measured fit guarantees.
The configurator estimates memory fit and speed for models in our catalogue. For a small setup we have run ourselves, try the CPU generation demo.
Check the local API is running
Ollama serves an API on port 11434. Other apps, including Open WebUI, talk to it there.
curl http://localhost:11434/api/chat -d '{"model": "gemma3:4b", "messages": [{"role": "user", "content": "Say hello in five words."}], "stream": false}'
You should get back a JSON object with a message containing the reply.
Run Open WebUI in Docker
With Docker running, start Open WebUI and point it at Ollama on your machine:
docker run -d -p 127.0.0.1:3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
Open http://localhost:3000. The first account you create becomes the administrator.
NVIDIA GPU
The Open WebUI docs also offer a :cuda image, started with --gpus all, which speeds up its built-in features such as document embedding. Ollama itself already uses your GPU without it.
Verify it works
Check
In Open WebUI, choose gemma3:4b from the model menu at the top and ask a question. Then confirm it ran locally:
ollama ps
Expected output
NAME ID SIZE PROCESSOR CONTEXT UNTIL gemma3:4b … … 100% GPU 4096 4 minutes from now
The exact columns vary by version. PROCESSOR shows whether the model is on your GPU, CPU or split between them.

Troubleshooting

The container can't reach Ollama. Check that curl http://localhost:11434 replies "Ollama is running" on the host, and that you included --add-host=host.docker.internal:host-gateway. On Linux, Ollama listens on 127.0.0.1 by default; see the Ollama FAQ for setting OLLAMA_HOST if the container still can't connect.
Run ollama ps. If PROCESSOR shows a CPU/GPU split, the model doesn't fit in VRAM; try a smaller tag, such as gemma3:1b or gemma3:4b.
Ollama sets the context length from your VRAM, and below 24 GiB that's 4K tokens. Raise it in the Ollama app's settings, or start the server with OLLAMA_CONTEXT_LENGTH=8192 ollama serve. Longer context uses more memory.
Change the left side of -p 127.0.0.1:3000:8080, for example -p 127.0.0.1:3001:8080, and open localhost:3001 instead.