Vector databases: search by a learned representation

A vector database stores vectors together with identifiers and associated data, then retrieves nearby vectors for a query. In semantic search, an embedding model maps text into that space. The database searches the representation it receives; the embedding model supplies what similarity means.

Documents and queries need compatible embeddings and a matching distance metric. Cosine similarity compares direction. For normalized vectors, an inner product gives the same ranking as cosine similarity.

The search core

Faiss is an indexing library rather than a complete database, but it makes the core operation visible. This example creates text embeddings, adds them to an exact inner-product index and retrieves matching passages. Install sentence-transformers and faiss-cpu; loading the encoder downloads its weights.

import faiss
from sentence_transformers import SentenceTransformer

passages = [
    "KV cache reuses earlier attention keys and values.",
    "LoRA trains a small update to frozen model weights.",
    "Diffusion generation repeatedly updates a noisy sample.",
]
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = encoder.encode(passages, normalize_embeddings=True).astype("float32")
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)

query = encoder.encode(["How does a model avoid recomputing old tokens?"],
                       normalize_embeddings=True).astype("float32")
scores, rows = index.search(query, k=2)
retrieved = [passages[row] for row in rows[0]]

The row identifiers map the vector matches back to the original text. A database adds facilities such as persistence, metadata filtering and updates around this operation.

Fast search trades something

An exact flat index compares the query with every stored vector. Approximate nearest-neighbour indexes reduce search work by exploring selected parts of the space, trading recall, memory and latency against each other.

A close vector is evidence of similarity under the chosen representation, not proof that a passage answers the question correctly. Filtering by document permissions or topic and reranking candidates can matter as much as the initial distance score.

In RAG, the retrieved text becomes model context. The vector database finds candidates; the language model still has to use their contents.