# What are RAG and embeddings, and how does a local model "read" your documents?

> Retrieval-augmented generation (RAG) turns your documents into embeddings (lists of numbers that capture meaning), finds the passages closest to your question, and pastes them into the model's prompt. The model never learns your files; it reads the few relevant pieces each time. Retrieval quality, not model size, usually decides the result.

Canonical: https://thelocalbench.com/learn/rag-and-embeddings · Data as of 2026-10-12 · The Local Bench (thelocalbench.com), data as of 2026-10-12, CC BY 4.0

Flags: [verified] checked against primary source · [sourced] cited · [estimate] formula · [sample] placeholder

Reviewed 2026-10-13 · REV 2026.10

## The plain version

A language model knows only what was in its training data. It has never seen your tax folder, your lab notebook or your company wiki. You could fine-tune it on them, but that's slow, expensive, and bad at precise facts. **Retrieval-augmented generation** takes the practical route instead: find the right paragraphs first, then hand them to the model along with your question.

**An everyday analogy.** An open-book exam. The student (the model) doesn't memorise the textbook; they get a few marked pages to read before answering. A good index makes all the difference. If you bookmark the wrong pages, even a brilliant student gets it wrong.

<RAGDiagram />

**What's an embedding?** An embedding model is a small neural network that turns a piece of text into a list of numbers, typically a few hundred to a few thousand of them. Texts with similar *meaning* end up with similar lists, even if they share no words: "car insurance renewal" lands near "the policy on the Honda expires in May". Searching means turning your question into the same kind of list and finding the stored passages whose lists are closest.

Embedding models are tiny compared with chat models. `nomic-embed-text`, a common default in Ollama, is a 274 MB download, so indexing runs comfortably on a CPU.

**The four settings that matter most:**

1. **Chunk size.** Documents are cut into pieces before embedding. Too small and a chunk loses context; too big and it blurs several topics together. A few hundred tokens with some overlap is the usual starting point.
2. **How many chunks you retrieve (top-k).** More chunks raise the chance the answer is included, but they also fill the prompt and slow the reply.
3. **The model's context window.** Every retrieved chunk has to fit. Ollama sets its default context length from your VRAM, and below 24 GiB that's 4K tokens. Open WebUI's own troubleshooting guide flags this as a common reason retrieval "doesn't work": the passages were found and then cut off.
4. **The embedding model.** Pick one that's good at your language and domain. The MTEB leaderboard compares them on retrieval tasks.

> **Why it matters for your rig**
>

RAG is the best way to get real work out of **modest hardware**. An 8B model on a 16 GB card, with good retrieval, will answer questions about your own documents better than a 70B model guessing from memory. The cost moves to **prefill**: 8 chunks of 500 tokens is a 4,000-token prompt to read before the first word appears. That's fast on a GPU, but it can take several seconds on a Mac or mini PC with a large model.



**When RAG struggles.** Questions that need the *whole* collection ("what are the common themes across all 400 reports?") don't fit the find-a-few-passages pattern. Neither do scanned PDFs without text, tables split across chunks, or questions phrased very differently from the source. Hybrid search (embeddings plus classic keyword search), reranking, and better PDF parsing fix most of these.

## The technical layer

### Similarity

Most stacks use cosine similarity, cos θ = (a · b) / (‖a‖ ‖b‖). Many embedding models output normalised vectors, so a plain dot product gives the same ranking. Indexes such as HNSW find approximate nearest neighbours in milliseconds, even across millions of chunks.

### Prompt budget

```
prompt_tokens ≈ system + question + top_k × chunk_tokens
TTFT          ≈ prompt_tokens / prefill_tps
KV_memory     grows with prompt_tokens (see the KV cache explainer)
```

Keep `top_k × chunk_tokens` well under the context you've actually configured, not the model's advertised maximum.

### Embedding with Ollama

```
ollama pull nomic-embed-text
curl http://localhost:11434/api/embed -d '{"model": "nomic-embed-text", "input": "Why is the sky blue?"}'
```

The response has an `embeddings` array with one vector per input. llama.cpp can serve embeddings too: start `llama-server` with `--embedding` and an embedding GGUF.

### Pitfalls

- **Change the embedder, rebuild the index.** Vectors from different models aren't comparable. Open WebUI asks you to reindex after switching.
- **Query and document prefixes.** Some models (nomic, e5) expect task prefixes such as `search_query:` and `search_document:`; check the model card.
- **Evaluate retrieval separately.** Before blaming the chat model, check whether the right chunk appears in the top-k at all.

## Sources

- [Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv 2005.11401)](https://arxiv.org/abs/2005.11401)
- [MTEB embedding leaderboard (Hugging Face)](https://huggingface.co/spaces/mteb/leaderboard)
- [Ollama blog: embedding models](https://ollama.com/blog/embedding-models)
- [Ollama docs: context length defaults](https://docs.ollama.com/context-length)
- [Open WebUI docs: RAG troubleshooting](https://docs.openwebui.com/troubleshooting/rag/)
- [Sentence Transformers documentation](https://www.sbert.net/)

Try it: https://thelocalbench.com/configurator?mode=hardware&hw=rtx-4060-ti-16g