RAG Explained: How AI Answers Questions from Your Documents

Imagine this: it's your first week at a new company, and you need a quick answer. You paste your employee handbook into a chat window and ask, "How many sick days roll over at year end?" The AI replies instantly, confidently, and — it turns out — completely wrong. It invented a "use-it-or-lose-it" policy your company has never had. The handbook was right there in its context window, but the model answered from habit instead of from your document.

This failure has a name-shaped hole in the conversation: the model didn't know your document. It had never seen it before. It was trained on public internet text, not on your company's handbook, your codebase's wiki, or last quarter's pricing sheet. And that gap — between what the model learned during training and what you need it to answer right now — is exactly the problem retrieval-augmented generation (RAG) was invented to solve.

In this post we'll break RAG down in plain language: what it is, why it works, how the pipeline fits together, where it already shows up in products you use, where it falls short — and a small worked example that makes the whole idea click.

The core problem: models don't know YOUR documents

A large language model is a prediction machine trained on a snapshot of public text. Ask it about Python syntax, the causes of World War I, or how photosynthesis works, and it does well — that knowledge was in its training data. But your world runs on documents the model has never seen:

• Your company's internal policies (the handbook that got the sick-day answer wrong).
• Your product's documentation as it exists today, after last week's API changes.
• Private data — support tickets, incident reports, contracts — that was never published.
• Anything written after the model's training cutoff, which is always older than you think.

When you ask about any of these, the model faces a choice: admit ignorance, or hallucinate — generate a fluent, plausible-sounding answer that has no grounding in fact. Models are relentlessly fluent, so they almost always choose the second. The sick-day answer was wrong because the model treated your handbook question like every other question: it generated the most likely-sounding continuation, not the most accurate one.

You have three broad options when a model needs to know things it wasn't trained on:

1. Fine-tune it on your documents — expensive, slow, and the model still can't quote a passage or tell you which document an answer came from.
2. Paste everything into the prompt — works for one short document, but breaks down when "your documents" means ten thousand pages.
3. RAG: fetch just the relevant passages and paste those into the prompt. That's the one this post is about.

The RAG idea in one paragraph

Retrieval-augmented generation is a simple bargain: don't ask the model to remember the answer — hand it the answer's raw material first. When a question comes in, the system searches your documents for the passages most likely to contain the answer (retrieval), then builds a prompt that includes the question plus those passages, and asks the model to answer using only what it was given (generation). The model's creativity is chained to retrieved evidence instead of wandering free. That's it. Everything else — chunking, embeddings, vector stores — is engineering in service of that one idea: put the facts in the prompt before you ask the question.

The pipeline, step by step

Here is the RAG pipeline most real systems use. The diagram below shows the full flow; we'll walk through each stage.

1. Chunking: documents into digestible pieces

Models read tokens, and every prompt has a limited context window. You can't stuff a 200-page handbook into a prompt — and even if you could, the model's attention would dilute across all those pages. So the first build-time step is chunking: splitting documents into small pieces, typically a few hundred words each.

Chunk size is a genuine trade-off, not a trivia question:

• Too small (one sentence per chunk), and a passage may lose the context that makes it true. "It applies to full-time staff" means nothing without the sentence before it.
• Too large (whole chapters), and the retrieved passage wastes prompt space on irrelevant text, diluting the answer.
• A common starting point is 200–500 words with some overlap between neighboring chunks, so an idea that straddles a boundary isn't cut in half.

Chunking is also where structure helps: splitting on section headings keeps each chunk self-contained, and many systems attach metadata (document title, page, section, date) to every chunk so answers can later cite their sources.

2. Embeddings: turning meaning into vectors

To find "the passages most likely to contain the answer," the system needs a way to compare a question with a chunk and say these are about the same thing. Keyword matching fails here: the question says "sick days roll over" while the handbook says "unused PTO balances carry forward" — same meaning, different words.

An embedding solves this by converting text into a long list of numbers — a vector — where texts with similar meaning end up as vectors pointing in similar directions. Think of it as a meaning coordinate: a chunk about "sick-day rollover" and a question about "unused PTO carry forward" land near each other in this coordinate space, even though they share almost no words.

You don't build this yourself. You call an embedding model — a small, specialized model whose whole job is producing these vectors. One important detail: embed the question and the chunks with the same model, or the coordinates won't be comparable — like measuring the question in inches and the chunks in centimeters.

3. Vector search: retrieving the top-k passages

At query time, the system embeds the user's question and searches the vector store — a database built for exactly this kind of similarity search — for the chunks whose vectors are closest to the question's vector. Closeness is usually measured by cosine similarity: how small the angle is between two vectors. Smaller angle, more similar meaning.

The search returns the top-k passages — typically 3 to 5 chunks ranked by similarity. This is the moment that makes or breaks the whole system. If the right passage is in the top-k, the model has everything it needs. If it isn't, no amount of clever prompting will save the answer. Retrieval quality is the ceiling of RAG quality — a principle worth pinning above your desk.

In practice, many systems combine vector search with classic keyword search (hybrid search): keyword search catches exact terms like product codes and names, vector search catches meaning. And a common refinement is reranking — fetching, say, 20 candidates and using a second, slower model to pick the best 5.

4. The grounded prompt: answer from THIS, not from memory

Now the system assembles the prompt. Schematically, it looks like this:

You are a helpful assistant. Answer the user's question using ONLY
the passages below. If the answer is not in the passages, say so.
Cite the passage number for each claim you make.

Passages:
[1] (Handbook, Leave policy) "Full-time employees accrue 12 sick
    days per year. Up to 5 unused days roll over to the next year."
[2] (Handbook, Leave policy) "Sick days may be used for personal
    illness or to care for an immediate family member."
[3] (FAQ) "Part-time employees accrue sick leave pro-rata..."

Question: How many sick days roll over at year end?

Answer:

The instruction is doing real work here: answer using only the passages; cite your sources; say "I don't know" if the answer isn't there. That instruction plus the retrieved evidence is what separates a RAG answer from the confident hallucination in our opening story.

5. The cited answer

The model generates its answer from the prompt above. A good RAG answer looks like this:

Up to 5 unused sick days roll over to the next year for full-time employees [1]. The policy doesn't say anything about rollover for part-time staff — the FAQ only describes pro-rata accrual [3].

Notice two things that make this trustworthy. First, the citations: you can click through to the exact passage and verify the claim. The model isn't asking for your trust; it's showing its receipts. Second, the model refused to invent a part-time rollover policy — it said what the evidence supports and stopped. That refusal is the RAG system working as designed.

Why RAG reduces hallucinations (but doesn't eliminate them)

Here's the honest claim, stated carefully: RAG reduces hallucinations — it does not eliminate them. Why does it help at all? Because hallucination is largely a symptom of absence: the model invents facts when its prompt contains nothing true to anchor to. RAG changes the model's job from "recall a fact you may never have learned" to "read this passage and restate what it says." Restating given evidence is a dramatically easier task than recalling facts from training, and models are much better at it.

Three mechanisms do the heavy lifting:

• Grounding: the answer's raw material is in the prompt, so the model doesn't need to reach into its weights for facts.
• Citations: asking for sources changes the model's behavior — it's harder to invent a claim when you've been told to point at the passage that supports it.
• Abstention: the "say you don't know" instruction gives the model an honest exit, so a retrieval miss becomes a shrug instead of a fabrication.

But the system can still fail, and the failures concentrate at the weakest links: retrieval (the wrong passages arrive, so the answer is faithfully wrong) and reasoning (the right passages arrive, but the model misreads them — combining two chunks into a conclusion neither supports). RAG fixes the knowledge problem; it does not fix the reasoning problem. Any vendor claiming RAG wipes out hallucinations entirely is selling you a certainty that doesn't exist.

Where RAG already shows up in products you know

You've almost certainly used RAG without knowing the name:

• Customer support bots that answer from a company's help center and link the article they used. The cited article is the retrieval step, made visible.
• Documentation search on developer sites where you ask a question in plain language and get an answer synthesized from several docs pages.
• "Chat with your PDF" tools — upload a contract, a paper, a manual, and ask questions. Upload → chunk → embed → index; every question is retrieval + grounded generation.
• Code assistants that answer questions about your repository by first retrieving relevant files and snippets, then explaining them.
• Enterprise search products that answer across wikis, tickets, and drives, with citations back to each source.

The pattern is always the same: search first, then generate. Once you see it, you'll spot RAG everywhere — it's become the default architecture for any AI feature that must answer from your data rather than the model's training data.

A tiny walkthrough: 3 chunks, 1 question

Let's make it concrete. This example is illustrative — the similarity scores are made up to show how the mechanism works, not measured from a real embedding model. Imagine a tiny vector store with three chunks from a fictional company's handbook:

CHUNK A (Leave policy):
"Full-time employees accrue 12 sick days per year.
 Up to 5 unused days roll over to the next year."

CHUNK B (Remote work policy):
"Employees may work remotely up to 3 days per week
 with manager approval. Core hours are 10am–3pm."

CHUNK C (Leave policy):
"Sick days may be used for personal illness or to
 care for an immediate family member."

The user asks: "How many of my sick days carry over to next year?"

The system embeds the question and compares it against the three chunk vectors. Here's what happens, conceptually:

• Chunk A wins by a wide margin: it contains "sick days" + "roll over" + "next year" — the same meaning as "sick days carry over to next year," even though the question says "carry over" where the chunk says "roll over." The embedding model bridges that wording gap. Illustrative similarity: 0.91.
• Chunk C comes second: it's about sick days, but about using them, not rolling them over. Same topic, wrong angle. Illustrative similarity: 0.68.
• Chunk B trails far behind: remote work has nothing to do with the question. Illustrative similarity: 0.21.

With top-k = 2, chunks A and C become the prompt's evidence. The grounded prompt says "answer using only these passages," and the model produces: "Up to 5 unused sick days roll over to the next year [A]." Chunk C contributes nothing to this answer, which is fine — retrieval errs on the side of including a near-miss rather than missing the winner.

Now consider how this breaks, because the failure modes are the lesson:

• Retrieval miss: suppose chunk A had been phrased as "accrued leave balances lapse per schedule" — bureaucratic enough that its vector drifts away from the question. The question then matches chunk C best, and the answer becomes "sick days may be used for illness" — a faithful answer to the wrong passage. The model did its job; retrieval failed it.
• Chunking split: suppose "Up to 5 unused days roll over" had landed in a chunk separated from "12 sick days per year." The retrieved passage says "5 unused days" — 5 days of what? The answer might hedge, or worse, guess.
• Garbage in: suppose the handbook was last updated in 2019 and the policy changed in 2024. RAG retrieves the stale passage faithfully and the model cites it confidently. RAG can only be as current and correct as the documents you feed it.

One walkthrough, three lessons: retrieval decides what the model can say, chunking decides what retrieval can find, and document quality decides whether any of it is worth saying.

RAG in pseudocode

The whole system fits in a few lines. This is conceptual pseudocode — real systems add reranking, metadata filters, and streaming, but the skeleton never changes:

# ---- BUILD TIME (run once per document set) ----
chunks = chunk(documents, size=300_words, overlap=50)
vectors = embedding_model.embed(chunks)      # same model for chunks...
index = vector_store.build(vectors, metadata=chunks.metadata)

# ---- QUERY TIME (run per question) ----
q_vec = embedding_model.embed(question)      # ...and for the question
top_k = index.search(q_vec, k=4)             # cosine similarity
prompt = f"""Answer using ONLY the passages below.
             Cite each claim. If unsure, say so.

             {format(top_k)}
             Question: {question}"""
answer = llm.generate(prompt)
return answer  # with citations to top_k

And if you'd rather see it as a sketch of real HTTP calls — embeddings plus chat completion — the shape is equally simple:

# 1. Embed the question
curl https://api.example.com/v1/embeddings \
  -d '{"input": "How many sick days roll over?"}'

# 2. Search your vector store for the closest chunks
curl https://your-vector-db/search \
  -d '{"vector": [...], "top_k": 4}'

# 3. Send question + retrieved passages to the chat model
curl https://api.example.com/v1/chat/completions \
  -d '{"messages": [
        {"role": "system",
         "content": "Answer using ONLY the passages below. Cite sources."},
        {"role": "user",
         "content": "Passages: ...\nQuestion: How many sick days roll over?"}
       ]}'

No step here is magic. The magic — such as it is — is the combination: search supplies the facts, the model supplies the language, and citations let a human verify the marriage.

Limitations: what RAG doesn't fix

RAG is the right default for "answer from my documents," but go in with eyes open:

• Retrieval can miss. Synonyms, jargon, and multi-hop questions ("which policy changed after the 2024 audit?") defeat simple similarity search. The answer is then fluent, cited, and wrong — the most dangerous failure mode, because citations make it look verified.
• Garbage in, garbage out. Stale documents, contradictory policies, duplicated content — RAG faithfully retrieves all of it. If two chunks contradict each other, the model must arbitrate, and it may pick the wrong one.
• It doesn't fix reasoning errors. RAG gives the model better ingredients; it doesn't make it a better cook. A model that misreads a table or botches arithmetic will do so with retrieved passages too.
• Chunking loses structure. Tables, footnotes, and "see section 4.2" cross-references often survive chunking badly. Long, structured documents need smarter splitting than fixed word counts.
• It costs latency and money. Every question now pays for an embedding call, a vector search, and a longer prompt (those passages aren't free tokens). For simple questions, that overhead may not be worth it.
• Privacy needs design, not hope. Your documents now flow through embedding and generation calls. Decide before building which data is approved for transmission and what must be redacted first — the architecture won't decide for you.

The takeaway

Retrieval-augmented generation is not a smarter model — it's a smarter workflow around the model. Instead of demanding that a model trained on the public internet somehow know your private documents, you fetch the relevant passages and put them in the prompt, then ask the model to answer from those. Hallucinations go down (not to zero), answers gain citations, and "I don't know" becomes an acceptable output instead of a forbidden one.

The next time a support bot answers your question and links the exact help-center paragraph it used, you'll know what happened under the hood: somebody's documents were chunked, embedded, searched, and pasted into a prompt — and the model was told, firmly, to read before it speaks.

This post is part of the AI tutorial series on javamakeuse.com — practical AI concepts explained with concrete examples, no hype required.