Skip to main contentThe Local Bench

How-to · Intermediate · about 40 min · reviewed 2026-10-13

Set up private document search (RAG) with Ollama and Open WebUI

Ask questions about your own PDFs and notes with a model that never leaves your machine. Uses a local embedding model, an Open WebUI knowledge base, and a chat model with enough context to read what it finds.

warning icon
Sourced, not yet tested.
Commands and menu paths come from the Ollama and Open WebUI documentation (checked 2026-10-13). We have not yet run this guide end to end. Open WebUI menu labels change between releases, so expect small differences.

Before you start

GPU with 12 GB+ VRAM, or a Mac / mini PC with 32 GB+ unified memory; CPU-only works for small collections, slowly. Linux, macOS, Windows.

  • Ollama and Open WebUI installed (see "Run your first local model")
  • Some text-based PDFs, Word files or notes to test with
Start guide

  • Software and cost

    $0 · free software; a GPU with 12 GB+ VRAM or 32 GB+ unified memory recommended

    Tested on

    Last tested: not yet

    Not yet run on our bench.

    Not yet tested by us

    • The whole guide end to end
    • The exact labels of the Embedding Model Engine and Embedding Model fields in Settings > Admin > Documents
    • Retrieval quality with nomic-embed-text on non-English documents

  1. Pull an embedding model and a chat model

    ollama pull nomic-embed-text
ollama pull qwen3:8b

    The embedding model turns text into vectors for search. The chat model reads the passages it finds. Use qwen3:14b if you have 16 GB+ of VRAM.

  2. Check the embedding model answers

    curl http://localhost:11434/api/embed -d '{"model": "nomic-embed-text", "input": "Why is the sky blue?"}'
    Expected output
    {"model":"nomic-embed-text","embeddings":[[0.0107, -0.0012, …]], …}

    A long list of numbers means it works. The values themselves will differ.

  3. Give the chat model enough context

    Retrieved passages have to fit in the prompt. Ollama's default context is 4K tokens on GPUs under 24 GiB, which is too small for document chat. Restart Ollama with a larger value:

    OLLAMA_CONTEXT_LENGTH=16384 ollama serve

    On macOS and Windows you can set the context length in the Ollama app's settings instead.

    warning icon
    More context uses more memory

    Every extra token adds to the KV cache. If the model starts spilling to the CPU (check ollama ps), use 8192 or a smaller model.

  4. Point Open WebUI's document embedding at Ollama

    In Open WebUI, open Settings > Admin > Documents. Set the embedding engine to Ollama and the embedding model to nomic-embed-text, then save.

    info icon
    Changed embedding model later?

    Open WebUI's docs say to reindex existing documents from the same Documents page after switching embedding models, because vectors from different models can't be compared.

  5. Create a knowledge base and add files

    Go to Workspace > Knowledge, click Create, give it a name and description, and upload your files. Wait for processing to finish before asking about them. Start with a few text-based PDFs: scanned images without a text layer won't be searchable.

  6. Verify it works
    Check

    Start a new chat with qwen3:8b, type #, and pick your knowledge base. Then ask something whose answer is in one of the files, and ask for a quote:

    Using the attached documents, what is the notice period in the lease? Quote the sentence and name the file.

    A good answer quotes the passage and shows citations you can click. If it answers confidently without a citation, it's guessing; see troubleshooting below.

Troubleshooting

  • The Open WebUI troubleshooting guide lists this as the most common problem. Check that the upload finished processing, that you selected the knowledge base with #, and that the Documents settings saved. Re-save the embedding model even if it's already selected; that forces a recheck.

  • The context is too small, so retrieved passages are cut off. Raise OLLAMA_CONTEXT_LENGTH (step 3) and check the allocated context with ollama ps.

  • Ask with words that appear in the document, try smaller files, or try a different embedding model. The MTEB leaderboard ranks embedding models by retrieval quality and language.

  • Embedding runs on whatever hardware Ollama uses. On a CPU-only machine, large PDFs can take minutes. Check the Open WebUI logs with docker logs open-webui.

Sources

  1. Open WebUI docs: Knowledge bases and document chat
  2. Open WebUI docs: RAG troubleshooting
  3. Ollama blog: embedding models
  4. Ollama API: POST /api/embed
  5. Ollama docs: context length

Markdown: /how-to/private-document-search-rag.md · JSON: /api/v1/how-to/private-document-search-rag.json