Build a RAG System with LangChain: Complete Tutorial

What is RAG and Why Does It Matter?

Retrieval Augmented Generation (RAG) combines the power of LLMs with your private data. Instead of fine-tuning (expensive, slow), RAG retrieves relevant context at query time and includes it in the prompt.

Use cases include:

• Customer support chatbots with product knowledge
• Internal documentation assistants
• Legal/medical document Q&A
• Code repository assistants

RAG Architecture Overview

A RAG system has two phases:

Indexing (offline):

1. Load documents
2. Split into chunks
3. Generate embeddings
4. Store in vector database

Retrieval (online):

1. Embed user query
2. Search vector database
3. Retrieve top-k relevant chunks
4. Send context + query to LLM
5. Return answer with sources

Setup

First, spin up a GPU on GPUBrazil for running local embeddings and LLM:

# Install dependencies
pip install langchain langchain-community langchain-huggingface
pip install chromadb sentence-transformers
pip install pypdf unstructured
pip install vllm  # For local LLM serving

Step 1: Document Loading

from langchain_community.document_loaders import (
    PyPDFLoader,
    DirectoryLoader,
    TextLoader,
    UnstructuredMarkdownLoader,
)

# Load single PDF
loader = PyPDFLoader("document.pdf")
documents = loader.load()

# Load directory of PDFs
loader = DirectoryLoader(
    "./docs/",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader,
    show_progress=True
)
documents = loader.load()

# Load multiple file types
from langchain_community.document_loaders import UnstructuredFileLoader

def load_documents(directory):
    """Load all supported documents from directory"""
    documents = []
    
    for ext, loader_cls in [
        ("*.pdf", PyPDFLoader),
        ("*.txt", TextLoader),
        ("*.md", UnstructuredMarkdownLoader),
    ]:
        loader = DirectoryLoader(directory, glob=f"**/{ext}", loader_cls=loader_cls)
        documents.extend(loader.load())
    
    return documents

docs = load_documents("./knowledge_base/")

Step 2: Text Chunking

Chunking strategy dramatically affects RAG quality:

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter,
)

# Basic recursive splitter (most versatile)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,       # Characters per chunk
    chunk_overlap=200,      # Overlap between chunks
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks from {len(documents)} documents")

# For markdown with structure
md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
)

# Semantic chunking (experimental but powerful)
from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
semantic_splitter = SemanticChunker(embeddings, breakpoint_threshold_type="percentile")

Step 3: Embeddings

Run embeddings locally on GPU for privacy and speed:

from langchain_huggingface import HuggingFaceEmbeddings
import torch

# Best open-source embedding models
EMBEDDING_MODELS = {
    "small": "BAAI/bge-small-en-v1.5",      # 33M params, fast
    "medium": "BAAI/bge-base-en-v1.5",       # 109M params
    "large": "BAAI/bge-large-en-v1.5",       # 335M params
    "best": "BAAI/bge-m3",                   # Multilingual, best quality
}

# Initialize with GPU
embeddings = HuggingFaceEmbeddings(
    model_name=EMBEDDING_MODELS["large"],
    model_kwargs={"device": "cuda"},
    encode_kwargs={
        "normalize_embeddings": True,  # For cosine similarity
        "batch_size": 32,
    }
)

# Test embedding
test_embedding = embeddings.embed_query("What is machine learning?")
print(f"Embedding dimension: {len(test_embedding)}")

Step 4: Vector Store

from langchain_community.vectorstores import Chroma
import chromadb

# Create vector store with persistence
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
    collection_name="knowledge_base"
)

# Load existing vector store
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings,
    collection_name="knowledge_base"
)

# Add new documents incrementally
vectorstore.add_documents(new_chunks)

# Search
results = vectorstore.similarity_search(
    "How do I reset my password?",
    k=5  # Return top 5 results
)

for doc in results:
    print(f"Source: {doc.metadata['source']}")
    print(f"Content: {doc.page_content[:200]}...")
    print("---")

Alternative: Qdrant (Production-Ready)

from langchain_community.vectorstores import Qdrant
from qdrant_client import QdrantClient

# Local Qdrant
client = QdrantClient(path="./qdrant_data")

vectorstore = Qdrant.from_documents(
    documents=chunks,
    embedding=embeddings,
    location=":memory:",  # or path for persistence
    collection_name="docs",
)

# With metadata filtering
results = vectorstore.similarity_search(
    "deployment guide",
    k=5,
    filter={"source": "docs/deployment.md"}
)

Step 5: Local LLM with vLLM

Run your own LLM instead of paying API costs:

# Terminal: Start vLLM server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --port 8000 \
    --tensor-parallel-size 1

from langchain_community.llms import VLLMOpenAI

# Connect to local vLLM
llm = VLLMOpenAI(
    openai_api_key="EMPTY",
    openai_api_base="http://localhost:8000/v1",
    model_name="meta-llama/Llama-3.1-8B-Instruct",
    max_tokens=1024,
    temperature=0.1,
)

# Test
response = llm.invoke("What is the capital of France?")
print(response)

Step 6: RAG Chain

from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Custom prompt for better answers
prompt_template = """Use the following pieces of context to answer the question. 
If you don't know the answer based on the context, say "I don't have information about that."
Always cite which document the information came from.

Context:
{context}

Question: {question}

Answer:"""

PROMPT = PromptTemplate(
    template=prompt_template,
    input_variables=["context", "question"]
)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # Puts all docs in context
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT}
)

# Query
result = qa_chain.invoke({"query": "How do I configure SSO?"})

print("Answer:", result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print(f"  - {doc.metadata['source']}")

Step 7: Production API

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from langchain.chains import RetrievalQA

app = FastAPI()

# Initialize RAG components (on startup)
embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-large-en-v1.5")
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
llm = VLLMOpenAI(openai_api_base="http://localhost:8000/v1", model_name="llama-3.1-8b")

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True,
)

class Query(BaseModel):
    question: str
    
class Answer(BaseModel):
    answer: str
    sources: list[str]

@app.post("/ask", response_model=Answer)
async def ask_question(query: Query):
    try:
        result = qa_chain.invoke({"query": query.question})
        
        sources = list(set(
            doc.metadata.get("source", "Unknown") 
            for doc in result["source_documents"]
        ))
        
        return Answer(
            answer=result["result"],
            sources=sources
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.post("/index")
async def index_document(file_path: str):
    """Add new document to knowledge base"""
    loader = PyPDFLoader(file_path)
    docs = loader.load()
    chunks = text_splitter.split_documents(docs)
    vectorstore.add_documents(chunks)
    return {"status": "indexed", "chunks": len(chunks)}

Advanced: Hybrid Search

Combine semantic search with keyword search for better results:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# Keyword retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5

# Semantic retriever
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Combine with equal weights
hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, semantic_retriever],
    weights=[0.5, 0.5]
)

# Use in chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=hybrid_retriever,
    return_source_documents=True,
)

Advanced: Reranking

Use a cross-encoder to rerank retrieved documents:

from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

# Initialize reranker
reranker = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
compressor = CrossEncoderReranker(model=reranker, top_n=3)

# Retrieve more, then rerank to top 3
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 10})
)

Cost Analysis

Running RAG on GPUBrazil vs. API services:

Component	API Cost	GPUBrazil Cost
Embeddings (1M tokens)	$0.10	~$0.02
LLM (100k tokens)	$3.00	~$0.40
Monthly (10k queries)	$300+	~$50

Plus: your data never leaves your infrastructure.

Production Checklist

• ✅ Use persistent vector store (Qdrant, Chroma with persistence)
• ✅ Add metadata filtering for multi-tenant systems
• ✅ Implement caching for repeated queries
• ✅ Set up hybrid search (semantic + keyword)
• ✅ Add reranking for better precision
• ✅ Monitor retrieval quality with evaluation datasets
• ✅ Implement document update/deletion pipelines
• ✅ Add authentication to your API

Conclusion

RAG is the most cost-effective way to give LLMs access to your private data. By running embeddings and LLMs on GPUBrazil, you keep your data private while saving 80%+ compared to API services.

Start with a simple implementation, measure retrieval quality, and iterate on chunking and retrieval strategies based on real user queries.