Production RAG & Advanced Vector Search

Retrieval-Augmented Generation grounds an LLM in your own data instead of relying on what it memorized during training. Getting RAG from a weekend demo to a production system is mostly about retrieval quality — which is where advanced vector search comes in. Both are covered here.

Table of Contents

• How RAG Works
• Document Loading and Chunking
• Embeddings
• Vector Databases
• Similarity Metrics
• Approximate Nearest Neighbor Search (HNSW)
• Hybrid Search
• Query Transformation
• Reranking
• Building the Generation Step
• Evaluating RAG Quality
• Production Concerns

How RAG Works

RAG has two phases:

1. Indexing (offline): documents → chunks → embeddings → vector database.
2. Querying (online): user question → embed → retrieve similar chunks → stuff into prompt → LLM answers.

User: "What is our refund window?"
      ↓ embed query
Vector DB: top-k similar chunks  →  "Refunds allowed within 30 days..."
      ↓
LLM answers using the retrieved chunks, with citations

Why it's useful: fresh data, fewer hallucinations, citations, and no fine-tuning bill every time your docs change.

Document Loading and Chunking

Chunking is the highest-leverage decision in RAG. Too small → chunks lose context. Too large → retrieval gets noisy and the prompt fills up.

def chunk_with_overlap(text: str, size: int = 800, overlap: int = 120):
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        # don't cut mid-sentence
        if end < len(text):
            end = text.rfind(". ", start, end) + 1 or end
        chunks.append(text[start:end].strip())
        start = end - overlap
    return chunks

The explanation: chunks of roughly 300–1000 tokens with 10–20% overlap work well for prose. Better still: split on document structure (headings, paragraphs, tables) so chunks are semantically whole.

Notes:

• Always attach metadata (source, page, section, date) to every chunk — you'll need it for citations and filtering.
• For tables and code, keep them whole rather than splitting across chunks.

Embeddings

An embedding is a list of numbers (a vector) that captures meaning: similar texts end up close together in vector space.

from openai import OpenAI

client = OpenAI()

def embed(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in resp.data]

The explanation: embed(["How do I reset my password?"]) returns a vector where questions about logins cluster together, regardless of exact wording.

Key rules:

• Use the same embedding model for indexing and querying. Mixing models silently destroys retrieval.
• Choose based on your language(s), dimension, and cost — embedding bills run over your entire corpus.
• Store the model name alongside each vector so you can re-embed when you upgrade.

Vector Databases

A vector database stores embeddings and answers "which stored vectors are closest to this query vector?"

# pgvector example
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          SERIAL PRIMARY KEY,
    content     TEXT,
    metadata    JSONB,
    embedding   vector(1536)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- Nearest neighbors for a query embedding
SELECT content, metadata, embedding <=> :query_vec AS distance
FROM chunks
ORDER BY embedding <=> :query_vec
LIMIT 8;

The explanation: <=> is cosine distance. The HNSW index makes the search approximate-but-fast instead of exact-but-slow.

Similarity Metrics

import numpy as np

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

The explanation: cosine ignores vector length and compares direction only — for most text models this is the right choice, and many providers already normalize vectors so cosine and dot product become equivalent.

Approximate Nearest Neighbor Search (HNSW)

Exact search compares the query against every vector — fine at 100k, hopeless at 500M. Approximate Nearest Neighbor (ANN) indexes trade a little recall for huge speed.

HNSW (Hierarchical Navigable Small World) builds layered graphs: coarse "highways" on top for fast traversal, dense layers below for precision. Search starts at the top and greedily descends.

# Tuning knobs (pgvector / Qdrant / hnswlib all expose these)
# m          — links per node. Higher = better recall, more memory.
# ef_construction — build-time effort. Higher = better graph, slower build.
# ef_search  — query-time effort. Higher = better recall, slower query.
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

SET hnsw.ef_search = 100;

The explanation: measure recall@k (how many of the true top-k you actually get) against latency, and tune ef_search until the trade-off fits your SLA. 95–99% recall is usually indistinguishable in answer quality.

Hybrid Search

Pure vector search misses exact terms: it will happily retrieve "return policy" for a query about "SKU-8842", because embeddings capture meaning, not identifiers.

Hybrid search combines vector similarity with keyword search (BM25) and merges results.

def hybrid_search(query: str, k: int = 8):
    vec_hits = vector_search(embed(query), k=20)       # semantic
    kw_hits  = bm25_search(query, k=20)                # exact terms
    return reciprocal_rank_fusion(vec_hits, kw_hits, k=k)

def reciprocal_rank_fusion(*ranked_lists, k=60):
    scores = {}
    for hits in ranked_lists:
        for rank, chunk in enumerate(hits):
            scores[chunk.id] = scores.get(chunk.id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

The explanation: RRF rewards chunks that appear high in both lists. Hybrid search is the single biggest quality upgrade for most production RAG systems — especially with product codes, names, and error messages.

Query Transformation

Users type bad queries. Fix them before retrieval:

• HyDE: have the LLM write a hypothetical answer, then search with that (an answer looks more like the documents than the question does).
• Query decomposition: split multi-part questions into sub-queries and retrieve for each.
• Step-back prompting: generalize ("What was our Q3 revenue?" → "How does the company report quarterly revenue?").
• Rewriting for chat: resolve pronouns using conversation history before searching.

def rewrite_with_history(query: str, history: list) -> str:
    return llm.chat([{
        "role": "user",
        "content": f"Rewrite as a standalone search query.\nHistory: {history}\nQuery: {query}"
    }])

The explanation: "and what about refunds?" becomes "What is the refund policy?" — which is actually searchable.

Reranking

First-stage retrieval (vector + BM25) is fast but crude. A reranker — a cross-encoder model — reads the query and each candidate together and scores relevance precisely.

def rerank(query: str, candidates: list, top_n: int = 5):
    scores = cross_encoder.predict([
        (query, c.content) for c in candidates
    ])
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, s in ranked[:top_n]]

The explanation: retrieve 50 cheaply, rerank down to the best 5. The LLM then sees only high-quality context — fewer distractions, better answers, and a smaller prompt.

Building the Generation Step

def answer(question: str) -> str:
    chunks = rerank(question, hybrid_search(question))
    context = "\n\n".join(
        f"[{i+1}] ({c.metadata['source']}) {c.content}" for i, c in enumerate(chunks)
    )
    return llm.chat([{
        "role": "system",
        "content": (
            "Answer using ONLY the provided context. "
            "Cite sources as [1], [2]. "
            "If the context doesn't contain the answer, say you don't know."
        ),
    }, {
        "role": "user",
        "content": f"Context:\n{context}\n\nQuestion: {question}"
    }])

The explanation: numbered chunks make citations trivial. The "only use provided context" instruction plus an explicit "say you don't know" escape hatch are what keep hallucination down.

Evaluating RAG Quality

Evaluate retrieval and generation separately:

• Retrieval metrics: recall@k, MRR (mean reciprocal rank) — is the right chunk even in the top results?
• Answer metrics: faithfulness (does the answer stick to the context?), relevance, and citation accuracy.

def faithfulness(answer: str, context: str) -> float:
    claims = llm.extract_claims(answer)
    verdicts = [llm.supported_by(claim, context) for claim in claims]
    return sum(verdicts) / max(len(verdicts), 1)

The explanation: break the answer into individual claims and check each against the retrieved context. Low faithfulness means the model is inventing things the documents never said.

Build a golden test set of question → expected source chunk → expected answer, and run it on every pipeline change.

Production Concerns

• Freshness: index updates must be incremental — re-embed only changed documents, not the whole corpus.
• Access control: filter retrieval by user permissions, or the vector DB will leak documents the user shouldn't see.
• Multi-tenancy: namespace vectors per tenant; never mix tenants in one flat index.
• Caching: cache query embeddings and frequent question→answer pairs.
• Fallbacks: if the vector DB is down, degrade to BM25 keyword search instead of failing.

-- Permission-aware retrieval: filter, THEN search
SELECT content, embedding <=> :query_vec AS distance
FROM chunks
WHERE tenant_id = :tenant_id
  AND (metadata->'acl') ?| ARRAY[:user_roles]
ORDER BY embedding <=> :query_vec
LIMIT 8;