Enterprise RAG Pipelines, Vector Stores & Internal Knowledge Assistants

The most common enterprise AI application is an assistant that answers questions from the company's own knowledge. This chapter teaches retrieval-augmented generation (RAG) — the technique — and shows you how it powers an internal knowledge assistant.

Table of Contents

• What Is RAG
• What Is a Knowledge Assistant
• The RAG Pipeline
• Chunking and Embedding
• Retrieval and Ranking
• Answering With Sources
• Applying Access Control Inside RAG
• Building a Knowledge Assistant
• Best Practices

What Is RAG

RAG (Retrieval-Augmented Generation) makes an LLM answer from a document set you control, instead of from memory. It retrieves relevant text from your knowledge base first, then asks the model to answer from that text.

Without RAG:    "What's our refund policy?" → model guesses from memory.
With RAG:       retrieve policy text → answer grounded in it.

Because the answer is grounded in retrieved documents, it's more accurate and auditable than free recall.

What Is a Knowledge Assistant

A knowledge assistant is the product: an internal bot inside Slack, Teams, or a web app that employees ask questions and get sourced answers from company documents.

Employee: "What's the travel approval limit for managers?"
Assistant: "Managers may approve up to $2,500. Source: Travel Policy §3.2"

The RAG Pipeline

Four steps, in order:

1. Ingest   — chunk documents and index them
2. Embed    — turn text into vectors
3. Retrieve — find chunks similar to the question
4. Generate — answer from retrieved chunks

Chunking and Embedding

Break long documents into pieces small enough to retrieve well, and convert each piece to a vector.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-distilroberta-v1")

def index_document(chunks):
    for chunk in chunks:
        vector = model.encode(chunk.text)     # into a numeric vector
        # Use chunk.id as the upsert key so re-indexing the same
        # chunk is idempotent — otherwise reruns create duplicates.
        vector_store.upsert(chunk.id, vector, metadata=chunk.meta)

Chunking size matters: too big loses precision, too small loses context. Plan ~300–500 tokens a chunk with overlap.

Retrieval and Ranking

At query time, embed the question and pull the nearest chunks.

q_vec = model.encode(question)
hits = vector_store.search(q_vec, top_k=5)

for h in hits:
    print(h.score, h.metadata["path"], h.text[:60])

Ranking can improve on raw vector similarity: a reranker rescores the retrieved chunks so the most relevant few rise to the top. This prevents edge context from being missed just because the whole document surfaced.

Answering With Sources

The generated answer should cite its sources — that builds trust and lets users verify.

prompt = f"""Answer using ONLY the context. Cite each fact.
Context:
hits_text = ...   # numbered chunks from retrieval
Question: {question}
Answer with bracketed source numbers like [2]."""

If nothing related is retrieved, the assistant should answer "I couldn't find that in the knowledge base" rather than guessing.

Applying Access Control Inside RAG

Retrieval must respect who is asking. Merge access rules into the query, not only the answer.

permitted = access_filter(user)         # docs this user may read
hits = vector_store.search(q_vec, top_k=10)
hits = [h for h in hits if h.doc_id in permitted]

Building a Knowledge Assistant

A working internal assistant couples RAG with the interfaces people already use.

Docs (wiki, SOPs) → index → assistant
User (in Teams/Slack) → question → filtered retrieve → answer + sources

Best Practices

• Clean and version your knowledge base; stale docs = wrong answers.
• Always cite sources in every answer.
• Filter retrieval by user permissions.
• Refresh the index when documents change.