# Set up private document search (RAG) with Ollama and Open WebUI

> Pull a local embedding model and a tool-capable chat model in Ollama, raise Ollama's context length, point Open WebUI's document embedding at Ollama, create a knowledge base, upload files, and query it with

Canonical: https://thelocalbench.com/how-to/private-document-search-rag · The Local Bench (thelocalbench.com), data as of 2026-10-13, CC BY 4.0

- Difficulty: Intermediate
- Time: about 40 minutes
- Cost: $0 (free software; a GPU with 12 GB+ VRAM or 32 GB+ unified memory recommended)
- Hardware: GPU with 12 GB+ VRAM, or a Mac / mini PC with 32 GB+ unified memory; CPU-only works for small collections, slowly
- OS: Linux, macOS, Windows
- Prerequisites: Ollama and Open WebUI installed (see "Run your first local model"); Some text-based PDFs, Word files or notes to test with
- Parts: Ollama (runs the chat and embedding models); Open WebUI (knowledge bases and chat); nomic-embed-text (embedding model, 274 MB); Qwen3 8B or 14B (chat model with native tool calling)
- Status: Sourced, not yet tested. Commands and menu paths come from the Ollama and Open WebUI documentation (checked 2026-10-13). We have not yet run this guide end to end. Open WebUI menu labels change between releases, so expect small differences.
- Tested on: not yet tested
- Last tested: never
- Reviewed: 2026-10-13

## Not yet tested by us

- The whole guide end to end
- The exact labels of the Embedding Model Engine and Embedding Model fields in Settings > Admin > Documents
- Retrieval quality with nomic-embed-text on non-English documents

## Step 1: Pull an embedding model and a chat model

```sh
ollama pull nomic-embed-text
ollama pull qwen3:8b
```

The embedding model turns text into vectors for search. The chat model reads the passages it finds. Use `qwen3:14b` if you have 16 GB+ of VRAM.



## Step 2: Check the embedding model answers

```sh
curl http://localhost:11434/api/embed -d '{"model": "nomic-embed-text", "input": "Why is the sky blue?"}'
```

```text
# Expected output
{"model":"nomic-embed-text","embeddings":[[0.0107, -0.0012, …]], …}
```

A long list of numbers means it works. The values themselves will differ.



## Step 3: Give the chat model enough context

Retrieved passages have to fit in the prompt. Ollama's default context is 4K tokens on GPUs under 24 GiB, which is too small for document chat. Restart Ollama with a larger value:

```sh
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
```

On macOS and Windows you can set the context length in the Ollama app's settings instead.

> **Warning: More context uses more memory**
>
Every extra token adds to the KV cache. If the model starts spilling to the CPU (check `ollama ps`), use 8192 or a smaller model.




## Step 4: Point Open WebUI's document embedding at Ollama

In Open WebUI, open **Settings > Admin > Documents**. Set the embedding engine to **Ollama** and the embedding model to `nomic-embed-text`, then save.

> **Tip: Changed embedding model later?**
>
Open WebUI's docs say to reindex existing documents from the same Documents page after switching embedding models, because vectors from different models can't be compared.




## Step 5: Create a knowledge base and add files

Go to **Workspace > Knowledge**, click **Create**, give it a name and description, and upload your files. Wait for processing to finish before asking about them. Start with a few text-based PDFs: scanned images without a text layer won't be searchable.



## Step 6: Verify it works

Start a new chat with `qwen3:8b`, type `#`, and pick your knowledge base. Then ask something whose answer is in one of the files, and ask for a quote:

```text
Using the attached documents, what is the notice period in the lease? Quote the sentence and name the file.
```

A good answer quotes the passage and shows citations you can click. If it answers confidently without a citation, it's guessing; see troubleshooting below.





## Troubleshooting

### The model says it can't see the documents

The Open WebUI troubleshooting guide lists this as the most common problem. Check that the upload finished processing, that you selected the knowledge base with `#`, and that the Documents settings saved. Re-save the embedding model even if it's already selected; that forces a recheck.



### Answers use only part of the document

The context is too small, so retrieved passages are cut off. Raise `OLLAMA_CONTEXT_LENGTH` (step 3) and check the allocated context with `ollama ps`.



### Retrieval finds the wrong passages

Ask with words that appear in the document, try smaller files, or try a different embedding model. The MTEB leaderboard ranks embedding models by retrieval quality and language.



### Uploads fail or are very slow

Embedding runs on whatever hardware Ollama uses. On a CPU-only machine, large PDFs can take minutes. Check the Open WebUI logs with `docker logs open-webui`.

## Sources

- [Open WebUI docs: Knowledge bases and document chat](https://docs.openwebui.com/features/workspace/knowledge/)
- [Open WebUI docs: RAG troubleshooting](https://docs.openwebui.com/troubleshooting/rag/)
- [Ollama blog: embedding models](https://ollama.com/blog/embedding-models)
- [Ollama API: POST /api/embed](https://github.com/ollama/ollama/blob/main/docs/api.md)
- [Ollama docs: context length](https://docs.ollama.com/context-length)