RAG: give the model something to look up

Retrieval-augmented generation (RAG) retrieves relevant material and gives it to a language model as context for its answer. A document can change today and become available to the next request without retraining the model.

A common setup splits documents into passages and stores an embedding for each passage. A query becomes a vector too. Similarity search finds candidates; a reranker can refine them before the selected text enters the prompt. Keyword search or a combination of search methods also works.

From retrieval to generation

This compact example embeds three passages, retrieves one and passes its actual text to a small instruction model. Normalized vectors make the dot product a cosine similarity. Install sentence-transformers, transformers and torch to try it; the first run downloads the two models.

import numpy as np
from sentence_transformers import SentenceTransformer
from transformers import pipeline

documents = [
    "KV cache reuses keys and values from earlier tokens.",
    "LoRA trains low-rank updates while base weights stay frozen.",
    "Temperature changes how token probabilities are distributed.",
]
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = encoder.encode(documents, normalize_embeddings=True)
question = "What avoids recomputing earlier tokens?"
query = encoder.encode(question, normalize_embeddings=True)
source = documents[int(np.argmax(vectors @ query))]

generator = pipeline(
    "text-generation", model="HuggingFaceTB/SmolLM2-135M-Instruct")
prompt = f"Use this source to answer.\nSource: {source}\nQuestion: {question}"
result = generator([{"role": "user", "content": prompt}],
                   max_new_tokens=80, do_sample=False)
answer = result[0]["generated_text"][-1]["content"]

The language model sees the selected source text, not the search vector. Passage embeddings represent stretches of text for retrieval; they are distinct from the token embeddings inside the generator.

Retrieval sets the ceiling

An answer depends on what the system finds and how the model uses it. Missing passages, outdated documents and unrelated matches can all spoil the result. Giving a model a source does not guarantee that its answer follows it. Useful systems retain source identifiers so readers can check the evidence.

RAG changes the information available at request time. LoRA changes learned model behaviour through training. They can be used together.