RAG (Retrieval-Augmented Generation)

RAG (Retrieval-Augmented Generation) 

1. What is RAG?

RAG (Retrieval-Augmented Generation). In simple words: instead of an AI answering only from what it remembers, we let it search for the answer first, then use what it found to write the final response.

Easy way to remember it — the open-book exam idea:

• An AI without RAG is like a student giving a closed-book exam — it can only use what's already in its head. If it never learned something, or forgot it, it might guess wrong.
• An AI with RAG is like a student giving an open-book exam — it can flip to the right page in a reference book before answering.

How RAG works, step by step:

2. Why do we even need RAG?

• AI can make things up (called "hallucinating") — RAG gives it real facts to hold onto instead of guessing
• AI's knowledge has an expiry date — it was trained on data only up to a certain point; RAG lets it fetch fresh, current info
• AI doesn't know your private stuff — like your company's internal documents; RAG can feed that in without retraining anything
• Updating is quick and cheap — just add or change documents in the database, no need to retrain the whole AI
• You can show proof — RAG can point to where it found the answer, so people can double-check it
• Less pressure on the AI to "know everything" — it just needs to read well and use what it's given

3. Comparisons

RAG vs Fine-Tuning (simple version)

Simple rule: Use RAG when the AI needs new facts. Use fine-tuning when the AI needs a new skill or style. Many real systems use both together.

RAG vs Long Context (simple version)

👉 Simple rule: Long context is like carrying the whole textbook into the exam. RAG is like bringing just the one page you need.

4. Diagrams

How RAG works, step by step:

flowchart TB
    subgraph OFFLINE[Step A: Prepare the documents]
        A[Your Documents] --> B[Cut into small pieces]
        B --> C[Turn into number-lists]
        C --> D[(Store in search database)]
    end

    subgraph ONLINE[Step B: When a question comes in]
        E[User asks a question] --> F[Turn question into numbers too]
        F --> G{Find closest matches}
        D -.-> G
        G --> H[Best matching pieces found]
        E --> I[Combine question + matching pieces]
        H --> I
        I --> J[AI reads it all and answers]
        J --> K[Final grounded answer]
    end

Which one should you use?

flowchart TD
    Start{What do you need?}
    Start -->|Facts that change often| RAG[Use RAG]
    Start -->|New tone, style, or skill| FT[Use Fine-Tuning]
    Start -->|Small, fixed set of documents| LC[Use Long Context]
    Start -->|Facts AND a new skill| Combo[Use RAG + Fine-Tuning together]

    RAG --> RAGnote[Search documents, find the best match, answer using it]
    FT --> FTnote[Retrain the AI on good examples of the new skill]
    LC --> LCnote[Just paste the whole document into the AI's input]
    Combo --> Combonote[Fine-tune for style, RAG for facts]

5. Code Example — A Tiny RAG in Python

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

# ---- Step 1: Load a model that turns text into numbers ----
embedder = SentenceTransformer('all-MiniLM-L6-v2')

documents = [
    "RAG combines retrieval with generation to ground LLM outputs in real data.",
    "Fine-tuning updates model weights using a custom labeled dataset.",
    "Vector databases store embeddings and support fast nearest-neighbor search.",
    "Chunking splits large documents into smaller retrievable passages."
]

# ---- Step 2: Turn every document into numbers, and store them ----
doc_embeddings = embedder.encode(documents).astype('float32')   # FAISS needs float32 numbers
dimension = doc_embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)      # a simple "find the closest match" search tool
index.add(doc_embeddings)

# ---- Step 3: When a question comes in, find the closest matching documents ----
def retrieve(query, k=2):
    query_vec = embedder.encode([query]).astype('float32')
    distances, indices = index.search(query_vec, k)
    return [documents[i] for i in indices[0]]

# ---- Step 4: Build the final message we'll send to the AI ----
def build_prompt(query, chunks):
    context = "\n".join(f"- {c}" for c in chunks)
    return f"""Answer using ONLY the information below. If it's not there, say so.

Information:
{context}

Question: {query}
Answer:"""

# ---- Step 5: Put it all together ----
query = "How does RAG reduce hallucination?"
retrieved_chunks = retrieve(query, k=2)
final_prompt = build_prompt(query, retrieved_chunks)

# response = your_llm_client.generate(final_prompt)   # send this to any AI model
print(final_prompt)

What's happening here:

• Step 1 — We load a small AI model whose only job is to turn sentences into numbers ("embeddings"). Similar meanings end up as similar numbers.
• Step 2 — We do this once for all our documents and save the results in a mini search tool called FAISS.
• Step 3 — When someone asks a question, we turn that into numbers too, using the exact same model, then ask FAISS: "which stored documents are closest in meaning to this?"
• Step 4 — We take the best-matching documents and glue them into a message, basically saying: "here's some info, now answer using only this."
• Step 5 — That final message is what actually gets sent to a real AI model (OpenAI, Claude, etc.) to generate the answer.

6. Common Mistakes Students Make

• Thinking RAG = zero wrong answers — it lowers the chances a lot, but the AI can still misread the info it found, or the search step can fetch the wrong pieces
• Calling "just paste everything in" RAG — that's not RAG, that's long context. Real RAG always has a search step
• Cutting documents badly — pieces too big = irrelevant text sneaks in; pieces too small = you lose context and answers get cut off mid-thought
• Never checking if the search step is actually good — always test whether the right document is actually being found (this is called "retrieval evaluation")
• Only using "meaning-based" search — this can miss exact things like ID numbers or product codes; mixing in old-school keyword search (BM25) helps catch those
• Treating "RAG vs fine-tuning" like a competition — the smart answer is knowing when to use each, or both together

7. Interview Q&A (Simple Answers)

Q1. What is RAG, in one line?

It's a way of letting an AI "look up" real information before answering, instead of relying only on what it remembers.

Q2. Why not just make the AI bigger so it "knows more"?

Because its knowledge would still be frozen at training time, expensive to update, and it still couldn't know your private data. RAG keeps knowledge and reasoning separate — so knowledge can be updated instantly and cheaply.

Q3. How do you decide between RAG and fine-tuning?

Ask: is the problem "the AI doesn't know this fact" → use RAG. Or is it "the AI knows the facts but responds wrong/badly" → use fine-tuning. Often, real systems need both.

Q4. AI models can now read huge amounts of text at once — do we still need RAG?

Yes. Feeding everything in every time is expensive, doesn't scale to millions of documents, and the AI can miss details buried in the middle of a huge text. RAG is better when the knowledge base is large or keeps growing.

Q5. How do you check if a RAG system is actually good?

Check two things: did it find the right documents (retrieval quality), and did it use them correctly and truthfully in the answer (generation quality). A common toolkit for measuring this is called RAGAS.

Q6. Why does chunk size matter?

Too big, and you drag in unrelated text along with the useful part. Too small, and you lose context or cut an answer in half. A good starting point is around 200–500 words per chunk.

Q7. What's "hybrid search"?

It's combining old-school keyword search with meaning-based (semantic) search. Keyword search catches exact matches like codes or IDs that meaning-based search sometimes misses.

Q8. Can RAG completely stop hallucination?

No. It reduces it a lot, but the AI can still go wrong if the search step grabs the wrong info, or if the AI ignores the info it was given and answers from memory anyway.