Document Q&A System

A Document Q&A System (builds on Retrieval-Augmented Generation, or RAG) lets a user ask questions in plain English about their own documents and get grounded answers. As a Forward Deployed Engineer, this is one of the most requested tools you'll deliver: your customer has hundreds of PDFs, Word docs, or FAQs, and they want to chat with them. This tutorial builds a working version with langchain, embeddings, FAISS, and an LLM.

How RAG works
Loading documents
Splitting into chunks
Creating embeddings and a vector store
Retrieving relevant chunks
Generating the answer
Putting it together
Notes

How RAG works

Instead of asking an LLM to know your documents from memory, RAG finds the relevant text first and gives it to the model to read. The flow is: split documents into chunks → turn each chunk into a vector (embedding) → when a question arrives, find the most similar chunks → feed those chunks plus the question to the LLM.

Because the answer is built from the retrieved text, the model can reference your actual documents and stay inside them. That's what makes RAG better than raw chat for customer content.

Documents  →  Chunks  →  Embeddings  →  Vector store
                                              ↑
Question ─────────────────────────────────────┘  match
   ↓
Chunks + Question  →  LLM  →  Answer

Loading documents

First you bring the files in. langchain_community ships loaders for PDFs, text, Word, and more.

from langchain_community.document_loaders import DirectoryLoader, TextLoader

loader = DirectoryLoader("./docs/", glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()
print(len(documents), "documents loaded")

DirectoryLoader scans a folder and glob controls which file types it picks up. Each document carries the text plus metadata like the source file path.

Splitting into chunks

LLMs have limited input, and retrieval works best on small pieces. Split each document into overlapping chunks.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=80)
chunks = splitter.split_documents(documents)
print(len(chunks), "chunks")

The splitter tries to break text at natural boundaries (paragraphs, sentences) and carries a little overlap so sentences split across boundaries aren't lost. chunk_size and chunk_overlap are the knobs you tune per document type.

Creating embeddings and a vector store

Embeddings turn each chunk into a list of numbers (a vector) so similar meanings sit close together. Store them all in a vector index for fast search.

from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("faiss_index")

FAISS is an in-memory vector index — fast, works on a laptop, and persists to disk with save_local. The embedding model maps each chunk to a ~384-dimension vector. Swap in OpenAI or other embedding models just by changing the class.

Retrieving relevant chunks

Search the index for the chunks closest to a user's question.

question = "What is our return policy?"
results = vectorstore.similarity_search(question, k=3)

for r in results:
    print(r.metadata["source"], "\n", r.page_content[:80])

similarity_search returns the k chunks whose vectors are nearest to the question's vector. These retrieved chunks are the "context" you'll hand to the model.

Generating the answer

Wire retrieval and generation together with a retrieval chain.

from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains.retrieval import create_retrieval_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama   # free local LLM; swap for others

llm = ChatOllama(model="llama3")

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the context below. If the answer isn't "
               "in the context, say you don't know."),
    ("human", "Context:\n{context}\n\nQuestion:\n{input}"),
])

combine_docs = create_stuff_documents_chain(llm, prompt)
chain = create_retrieval_chain(
    vectorstore.as_retriever(search_kwargs={"k": 3}),
    combine_docs,
)

answer = chain.invoke({"input": "How many days for returns?"})
print(answer["answer"])

create_retrieval_chain splits the work into an explicit retriever and a "stuff" document chain that packs the retrieved chunks into the prompt. The prompt template's {context} and {input} slots are filled automatically. This is the current recommended API — the older RetrievalQA.from_chain_type helper is deprecated and should not be used for new work.

The system prompt is where you keep answers grounded: "from the context only, else say you don't know." Using ChatOllama keeps everything local and free; for production you'd point the same chains at a hosted model like Claude — just change the llm variable.

Note: You can also add a document_prompt to format each retrieved chunk (e.g. "Source: {source}\n{page_content}") so the model can cite where it found an answer.

Putting it together

The full system as a small, reusable function.

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains.retrieval import create_retrieval_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama

EMBEDDING_MODEL = "all-MiniLM-L6-v2"
CHUNK_SIZE, CHUNK_OVERLAP = 500, 80

def build_index(docs_dir="./docs/"):
    loader = DirectoryLoader(docs_dir, glob="**/*.txt", loader_cls=TextLoader)
    chunks = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP
    ).split_documents(loader.load())
    vs = FAISS.from_documents(chunks, HuggingFaceEmbeddings(
        model_name=EMBEDDING_MODEL))
    vs.save_local("faiss_index")
    return vs

def make_chain(vs):
    llm = ChatOllama(model="llama3")
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Answer using only the context below. "
                   "If the answer isn't in the context, say you don't know."),
        ("human", "Context:\n{context}\n\nQuestion:\n{input}"),
    ])
    combine_docs = create_stuff_documents_chain(llm, prompt)
    return create_retrieval_chain(
        vs.as_retriever(search_kwargs={"k": 3}), combine_docs)

index = build_index()          # build once, rebuild when docs change
chain = make_chain(index)
print(chain.invoke({"input": "What is our return policy?"})["answer"])
print(chain.invoke({"input": "How do I reset my password?"})["answer"])

Rebuild the index whenever documents change, then reuse it for every question.

Notes

Embedding vs LLM model are different. The embedding model maps text to vectors; the LLM writes the answer. You can mix them freely.
Tune chunk_size. Bigger chunks = more context per answer but fuzzier retrieval; smaller = sharper but the answer may miss surrounding detail.
k controls how much context you hand the model. Start at 3 and raise it if answers feel too thin.
Vector stores (FAISS, Chroma, Pinecone) all serve the same role; pick one that matches your scale.
Always rebuild the index when your documents change, or the system answers from stale content.

Ask Questions in Plain English About Your Documents (RAG)

Table of Contents