What Is RAG: Retrieval-Augmented Generation Explained

Ask a general-purpose LLM about your company’s refund policy or last week’s support tickets, and it will either say it doesn’t know or invent something plausible. Its knowledge stops at whatever was in its training data. It has never seen your documents, and it cannot go and look them up mid-conversation. Retrieval-augmented generation (RAG) is the fix: before the model answers, a separate step finds the relevant text in your own data and hands it to the model as part of the prompt.

How retrieval-augmented generation actually works

A plain LLM call is one step: prompt in, completion out, using only what the model learned during training. RAG splits that into two: retrieve, then generate.

1. Your documents are split into chunks, converted into vectors (embeddings), and stored in a vector database ahead of time.
2. At query time, the user’s question is embedded the same way and compared against the stored chunks to find the closest matches.
3. The top matches are inserted into the prompt as context, and only then does the LLM generate an answer.

Think of it as an open-book exam versus a closed-book one. A closed-book LLM answers from memory alone, which is exactly why it invents things when the question falls outside what it memorized. RAG hands it the relevant page first, so it answers from something in front of it rather than something it is reconstructing.

The model itself never changes. Its weights are exactly what they were before you added RAG; retrieval is the only new piece. That is also what makes RAG cheap to keep current: edit a document and the next query sees the edit.

Building a RAG pipeline: chunking and embedding your data

Ingestion is the half of the pipeline that runs before anyone asks a question. It turns your documents into something a similarity search can query.

Documents are too long to embed as single units. A 20-page PDF embedded as one vector averages out into something that matches every query a little and none of them well, so you split it into chunks first. A plain fixed-size splitter with overlap is enough to start:

1
2
3
4
5
6
7
8

def chunk_text(text, chunk_size=500, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

500 characters with 50 of overlap is a reasonable starting point. Production pipelines usually chunk by token count instead of character count, using a tokenizer-aware splitter so a chunk boundary doesn’t land mid-word. The overlap exists so a sentence split across two chunks still appears whole in at least one of them.

Each chunk then gets embedded and stored. Chroma is a good first vector database because it runs in-process with no server to stand up, and its default embedding function (all-MiniLM-L6-v2) needs no API key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import chromadb

client = chromadb.Client()
collection = client.create_collection(name="support-docs")

collection.add(
    ids=["doc1", "doc2", "doc3"],
    documents=[
        "The refund window is 30 days from the delivery date.",
        "Refunds go back to the original payment method within 5 business days.",
        "Store credit never expires and can be used on any future order.",
    ],
)

That is the whole ingestion phase: chunk, embed, store. It runs once per document, and again whenever a document changes.

If you’d rather not install a vector database on your machine, most of them (Chroma, Weaviate, Milvus) ship an official image and run fine in a container; see what Docker actually does if you haven’t used one before.

How retrieval works: embedding the query and ranking matches

At query time, the same embedding model turns the user’s question into a vector, and the database returns whichever stored chunks sit closest to it, using whatever distance metric it’s configured with — cosine similarity and squared L2 are the two you’ll see most:

1
2
3
4
5
6
7
8

results = collection.query(
    query_texts=["How long do I have to return something?"],
    n_results=2,
)

print(results["documents"])
# [['The refund window is 30 days from the delivery date.',
#   'Refunds go back to the original payment method within 5 business days.']]

The query never uses the word “refund” and it still matches. That is the whole reason to embed rather than grep: two pieces of text land near each other in vector space when they mean similar things, not when they share words.

The last step joins the retrieved chunks into the prompt:

1
2
3
4
5
6
7
8

context = "\n".join(results["documents"][0])

prompt = f"""Answer the question using only the context below. If the context doesn't contain the answer, say so.

Context:
{context}

Question: How long do I have to return something?"""

prompt goes to whichever chat completion API you are already calling. RAG doesn’t require a specific model or provider; it is a pattern for what you put in the prompt before you send it. If that API call needs a key, keep it out of your image the way you’d keep any other credential out: as an environment variable or a mounted secret, never a build argument. Environment Variables and Secrets in Docker covers the difference.

RAG vs fine-tuning: which one fixes your problem

Both aim to make an LLM better at your specific use case, but they change different things.

A support bot whose refund policy changes every quarter is a RAG problem: fine-tune it on last quarter’s policy and it will be confidently wrong about the new one. A model that has to emit a specific JSON schema every time, or hold a particular reasoning style, is closer to a fine-tuning problem — no amount of retrieved context changes how a model formats its output. Plenty of production systems use both: a fine-tuned model for tone and structure, fed facts from RAG.

Where retrieval-augmented generation breaks

RAG demos look easy because the demo has three documents and one obvious answer. Production systems have thousands of documents, and each of the following can quietly wreck an answer:

• Chunking that ignores structure. A fixed-size splitter that cuts a table in half, or separates a heading from the paragraph it introduces, produces chunks that read as gibberish out of context — and gibberish embeds and retrieves badly.
• A stale index. Vector search has no sense of time. If a document is updated but never re-embedded, the old version is still the one that gets retrieved, and nothing in the pipeline signals that it is wrong.
• The right chunk exists but doesn’t rank high enough. Top-k retrieval returns only the k closest matches; if the actual answer is the 15th-closest chunk and you retrieve 5, the model never sees it.
• Too much context, badly placed. Stuffing ten retrieved chunks into the prompt doesn’t mean the model weighs all ten evenly. Models are measurably worse at using information sitting in the middle of a long context than at the start or the end.
• No evaluation beyond “the demo answer looked right.” Without a test set of questions and expected answers, a retrieval regression from a chunking change or a re-indexing bug ships unnoticed.

Most of these are retrieval failures rather than model failures, and every one of them is visible if you inspect the retrieved chunks instead of only the final answer.

How to build your first RAG pipeline

Build the three-step version first: chunk a handful of real documents, embed them into Chroma or another local store, and wire the retrieved chunks into a prompt exactly as shown above. Then ask it the 10 questions you expect real users to ask. Check whether the retrieved chunks contain the answer before you judge the generated answer — a wrong retrieval makes a wrong answer inevitable no matter how the prompt is written. Only once that loop holds up on real questions is it worth tuning chunk size, swapping embedding models, or adding a reranker.

Related articles

• What Is Docker and What Is It Used For
• Environment Variables and Secrets in Docker
• What Is Docker Compose and How to Use It