Module 6 · Lesson 13

A language model's parameters are not a live database. They do not automatically contain your private documents, and they do not automatically update every time the outside world changes.

Retrieval-Augmented Generation, usually called RAG, addresses that problem by retrieving relevant information when a question is asked and placing that information into the model's context before it generates the answer.

The open-book-exam mental model

User question
    ↓
retrieve relevant passages
    ↓
question + retrieved context
    ↓
language model
    ↓
grounded response

The model does not need to memorize every document in its weights. It can be given the relevant material at the moment it needs it.

What happens before the user asks a question?

A RAG system usually prepares its knowledge source first. Documents may be cleaned, split into chunks, embedded, indexed, and stored with metadata.

Vector similarity search is common, but a dedicated vector database is not mandatory. Some systems use traditional keyword search, full-text search, hybrid search, reranking, or combinations of these methods.

What happens at query time?

  1. The user's question is converted into a search representation.
  2. The retrieval layer finds potentially relevant passages.
  3. A reranker or other filtering step may improve the selection.
  4. The chosen passages are added to the model's context.
  5. The model generates an answer using the question and retrieved evidence.
  6. The application may attach citations back to the original sources.

RAG reduces hallucinations; it does not eliminate them

This distinction matters.

Retrieval can fail. The system can select the wrong passage. Documents can be outdated or contradictory. The model can misread the retrieved material, combine claims incorrectly, or generate a statement not supported by the sources.

RAG improves grounding because the model has evidence in context, but trustworthy systems still need retrieval evaluation, source quality controls, citation verification, and testing.

When RAG is a strong choice

RAG is particularly useful for private organizational knowledge, documentation, policies, support content, rapidly changing information, and applications where users should be able to inspect the source behind an answer.

It usually requires no retraining of the underlying language-model weights, but it absolutely requires data engineering: documents must be collected, indexed, updated, secured, and retrieved well.

Demystifying AI series

← RLHF and RLAIF   ·   Fine-Tuning vs. Prompt Engineering →