Skip to main contentThe Local Bench

Explainer · REV 2026.10 · reviewed 2026-10-13 · 9 min

What are RAG and embeddings, and how does a local model "read" your documents?

How retrieval-augmented generation lets a small local model answer from your own files without retraining, what an embedding actually is, and the four settings that decide whether it works.

Discrete graphics card

The 50-word answer

Retrieval-augmented generation (RAG) turns your documents into embeddings (lists of numbers that capture meaning), finds the passages closest to your question, and pastes them into the model's prompt. The model never learns your files; it reads the few relevant pieces each time. Retrieval quality, not model size, usually decides the result.

The plain version

A language model knows only what was in its training data. It has never seen your tax folder, your lab notebook or your company wiki. You could fine-tune it on them, but that's slow, expensive, and bad at precise facts. Retrieval-augmented generation takes the practical route instead: find the right paragraphs first, then hand them to the model along with your question.

An everyday analogy. An open-book exam. The student (the model) doesn't memorise the textbook; they get a few marked pages to read before answering. A good index makes all the difference. If you bookmark the wrong pages, even a brilliant student gets it wrong.

Retrieval-augmented generation pipelineYour files (PDF, notes, mail), then Chunk (~500 tokens each), then Embed (text → vector), then Vector index (stored locally), then Retrieve (top-k nearest), then LLM answers (with citations). The question is embedded with the same model and compared against the index at the retrieve step.Your filesPDF, notes, mailChunk~500 tokens eachEmbedtext → vectorVector indexstored locallyRetrievetop-k nearestLLM answerswith citationsIndex once (steps 1–4) · Answer every question (steps 5–6)
Fig. 1 · Everything left of “Retrieve” happens once, when you add documents. The highlighted step is where most quality problems start: if the right chunk isn’t retrieved, the model can’t use it.

What's an embedding? An embedding model is a small neural network that turns a piece of text into a list of numbers, typically a few hundred to a few thousand of them. Texts with similar meaning end up with similar lists, even if they share no words: "car insurance renewal" lands near "the policy on the Honda expires in May". Searching means turning your question into the same kind of list and finding the stored passages whose lists are closest.

Embedding models are tiny compared with chat models. nomic-embed-text, a common default in Ollama, is a 274 MB download, so indexing runs comfortably on a CPU.

The four settings that matter most:

  1. Chunk size. Documents are cut into pieces before embedding. Too small and a chunk loses context; too big and it blurs several topics together. A few hundred tokens with some overlap is the usual starting point.
  2. How many chunks you retrieve (top-k). More chunks raise the chance the answer is included, but they also fill the prompt and slow the reply.
  3. The model's context window. Every retrieved chunk has to fit. Ollama sets its default context length from your VRAM, and below 24 GiB that's 4K tokens. Open WebUI's own troubleshooting guide flags this as a common reason retrieval "doesn't work": the passages were found and then cut off.
  4. The embedding model. Pick one that's good at your language and domain. The MTEB leaderboard compares them on retrieval tasks.

When RAG struggles. Questions that need the whole collection ("what are the common themes across all 400 reports?") don't fit the find-a-few-passages pattern. Neither do scanned PDFs without text, tables split across chunks, or questions phrased very differently from the source. Hybrid search (embeddings plus classic keyword search), reranking, and better PDF parsing fix most of these.

  • Similarity

    Most stacks use cosine similarity, cos θ = (a · b) / (‖a‖ ‖b‖). Many embedding models output normalised vectors, so a plain dot product gives the same ranking. Indexes such as HNSW find approximate nearest neighbours in milliseconds, even across millions of chunks.

    Prompt budget

    prompt_tokens ≈ system + question + top_k × chunk_tokens
TTFT          ≈ prompt_tokens / prefill_tps
KV_memory     grows with prompt_tokens (see the KV cache explainer)

    Keep top_k × chunk_tokens well under the context you've actually configured, not the model's advertised maximum.

    Embedding with Ollama

    ollama pull nomic-embed-text
curl http://localhost:11434/api/embed -d '{"model": "nomic-embed-text", "input": "Why is the sky blue?"}'

    The response has an embeddings array with one vector per input. llama.cpp can serve embeddings too: start llama-server with --embedding and an embedding GGUF.

    Pitfalls

    • Change the embedder, rebuild the index. Vectors from different models aren't comparable. Open WebUI asks you to reindex after switching.
    • Query and document prefixes. Some models (nomic, e5) expect task prefixes such as search_query: and search_document:; check the model card.
    • Evaluate retrieval separately. Before blaming the chat model, check whether the right chunk appears in the top-k at all.

Sources

  1. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv 2005.11401)
  2. MTEB embedding leaderboard (Hugging Face)
  3. Ollama blog: embedding models
  4. Ollama docs: context length defaults
  5. Open WebUI docs: RAG troubleshooting
  6. Sentence Transformers documentation

Other explainers: Context length and RoPE · KV cache · Memory bandwidth · Mixture of experts · GGUF vs EXL vs AWQ · Quantization · Speculative decoding · Tool calling and MCP