Skip to main contentThe Local Bench

Use case 2 of 13

Offline document search with RAG

Ask questions of your own PDFs, notes and manuals, with citations, on a machine that never phones home.

A neat stack of paper documents with a blue tab beside a black mini PC
Image: generated with Grok Imagine

The real example

Open WebUI knowledge bases

Open WebUI, a self-hosted chat front end, ships retrieval-augmented generation (RAG) built in: upload documents to a knowledge base, they are chunked and embedded, and answers cite the passages used. Its documentation describes an optional hybrid search that combines BM25 keyword matching with embedding search, followed by re-ranking with a cross-encoder and a relevance threshold. It also warns that changing the embedding model means re-indexing everything, because vectors from different models are not compatible.

Sources: Open WebUI: RAG · Qwen3-Embedding model card (MMTEB results)

Why run it locally

Contracts, medical letters and company wikis are exactly the documents you should not upload. Embedding a few thousand pages is a one-off job any recent GPU handles.

Recommended stack

Three tiers built from our builds. Weights-in-memory is our arithmetic (parameters × bits), checked against each device's usable memory; it excludes KV cache and runtime overhead. Runtime compatibility is not validated.

Pro

Dual-GPU 48 GB

$4,044Sample

48 GB on 2× RTX 3090 (used) 48 GB

Weights ≈ 25.5 GB of 47.0 GB usableEstimate

Excludes KV cache and runtime overhead. Runtime compatibility is not validated.

The 8B embedder leads MMTEB in our table, and 48 GB leaves room for a 32B answerer with long context.

Runtime: Open WebUI + vLLM or llama.cpp

Setup outline

  1. Run Open WebUI and a local model server (Ollama or llama.cpp) with Docker.
  2. In the admin settings, set the embedding model to a local one and turn on hybrid search with a reranker.
  3. Create a knowledge base and upload documents. Scanned PDFs need OCR first (see the receipt-scanning use case).
  4. Attach the knowledge base to a model and ask questions. Check that citations point to the right passages.
  5. Decide the chunk size once. Changing chunking or the embedder later means re-indexing.

Limits

  • RAG retrieves passages; it does not read every document for each question. Summaries of a whole archive need a different approach.
  • Tables and scanned pages retrieve poorly unless converted to clean text first.
  • Answers are only as current as the last re-index.

Next: A local coding copilot

Build prices come from our build datasheets: GPUs and devices are sourced retail or sold prices; other parts are sample prices until sourced, and the flag on each total says which. Model links go to our datasheets, where each parameter count and licence was checked against the Hugging Face API.