RAG Explained: Build Answers from Retrieved Sources

Retrieval-augmented generation, usually called RAG, combines a retrieval step with generation. The application finds relevant source material, supplies it as context, and asks a model to produce an answer grounded in that material.

The original RAG paper studied a particular model architecture. In application engineering, the term is also used for a broader family of retrieve-then-generate systems. This lesson focuses on that practical application pattern, rather than reproducing the paper’s training setup.

You will design the data flow for our fictional handbook assistant and define the boundaries that make source-based answers inspectable.

Separate ingestion from answering

Ingestion prepares sources before a user asks a question. It includes reading documents, extracting useful text, deciding how to split it, preserving source metadata, and building a searchable representation. Query-time work retrieves permitted passages and constructs the model request.

Keeping those stages separate makes updates and failures easier to reason about. A broken document parser should not silently become an answer-generation error. An index built from last month’s source should not be described as current just because the model request ran today.

Microsoft’s RAG design and evaluation guide treats the preparation, retrieval, and evaluation choices as an experimental design problem. There is no universally best combination of parser, chunk length, embedding, and ranking configuration for every corpus.

Chunk around meaning and source boundaries

Long documents may need to be split into passages. A useful chunk should preserve enough context to interpret the statement it contains. A refund rule separated from its eligibility conditions can become misleading even when both pieces were extracted correctly.

Start with document structure such as headings, paragraphs, and tables. Keep stable chunk IDs and a link back to the original document or section. Record the source version and access metadata. If a source changes, know which chunks and vectors must be replaced or removed.

Overlap between chunks can preserve neighboring context, but it also duplicates text and can inflate retrieval results. Increasing chunk size can retain conditions while using more context. Smaller chunks can improve focus while losing relationships. Choose using representative questions and inspected failures, not a single recommended token count.

Our capstone’s source documents are deliberately short, so each fixture is one retrievable unit. That simplifies the demonstration; it is not a prescription to store every real document as one chunk.

Retrieve, assemble context, then generate

For each question, the application should first establish identity and source permissions. It then retrieves a bounded set of passages and passes only the permitted context to the generator.

Each excerpt needs a stable ID, readable text, and enough metadata to understand its currency and origin. Ask the model to cite those IDs and to abstain when the evidence is inadequate. Keep the context within an explicit budget; sending every document can increase cost and distract the model from the useful evidence.

The fictional refund source says the window is 14 calendar days after purchase. If a user asks the window, the answer can cite refunds-v1. If the user asks whether a damaged item qualifies for an exception and the corpus says nothing about exceptions, the supported behavior is to explain the missing information rather than invent a policy.

Retrieved text is untrusted input. The OWASP prompt-injection guidance covers attacks carried through external content. A passage that says “ignore all rules” must not gain authority because it was retrieved. Instruction separation is useful, but also constrain tool permissions, accessible data, and output handling.

Citations need two kinds of checks

An application can check mechanically that a returned source ID appeared in the retrieved context. That prevents a class of invented or inaccessible references.

It must separately check whether the cited text supports the claim. “14 days” does not support “14 business days,” and a general policy does not establish a particular customer’s eligibility. Citation membership is easier to automate than semantic support; do not confuse them.

When sources conflict, the model should not silently choose whichever wording sounds more confident. Define an application policy: prefer the current approved version, surface the conflict, or require a human decision. Record that rule in the specification and evaluation cases.

RAG, fine-tuning, and tools solve different problems

RAG supplies information at request time. Fine-tuning adjusts model parameters using training data. A tool call lets a model request an external operation or lookup through a defined interface. These approaches can be combined, but they are not interchangeable.

For frequently changing handbook content, retrieving a maintained source is often an appropriate first experiment. It avoids treating a model’s remembered wording as the authoritative policy. That does not mean RAG automatically guarantees freshness; ingestion and deletion workflows still have to work.

You also do not need an autonomous agent for a fixed retrieval-and-answer sequence. Start with a predictable workflow when the steps are known. Anthropic’s building effective agents guidance discusses the tradeoff between simpler workflows and more flexible agent patterns.

Exercise: inspect the evidence boundary

Write three source records: a current public policy, an older conflicting policy, and a staff-only procedure. Give each a distinct ID, update date, and audience label.

Create four questions: one answered by the current policy, one requiring a missing exception, one exposing the conflict, and one asking for the staff procedure as a public user. For each, write the permitted source IDs and expected behavior before calling a model.

You have completed the exercise when you can trace every proposed answer from user identity to retrieved passages to cited claims. The public-user case must exclude staff material before context assembly. The missing-exception case must not acquire invented details just to produce a complete-sounding response.

Use manual review or an approved live model experiment to assess faithfulness. If you only simulate the answers, label the results as a design exercise. The capstone provides an offline retrieval/extraction baseline and an optional live-generation adapter; only that optional mode performs generation.

Documentation checked November 8, 2026. No live model quality or accuracy result is claimed in this lesson.