Lesson 4 of 6 · Working with Documents and Data

RAG in plain English

How does an AI answer a question about a thousand pages it cannot hold in its head at once? It doesn't. A librarian fetches the right few pages first, then the AI reads only those. That librarian is RAG.

You will see the letters RAG on feature pages, in sales pitches, and in the settings of every serious AI tool. It stands for retrieval-augmented generation, which is three words of jargon for a simple idea: search first, then answer. Once you see the shape of it, a whole category of AI behavior starts making sense, including some of the strangest wrong answers you will ever get.

The problem RAG solves

Lesson 2 left you with a bucket. The model's context window holds a lot, but not a filing cabinet. A year of invoices, a hundred contracts, every email your business has sent: none of that fits in one conversation, and even if it did, the model would be paying attention to a million things at once to answer one question.

So the tool cheats, in a good way. It stores your documents outside the model, in a searchable form. When you ask a question, it searches that store, pulls out the handful of passages that seem relevant, and puts only those into the context window next to your question. The model reads a few pages instead of a thousand, and answers from them. That is the entire trick, and every product name for it, knowledge base, project files, sources, grounding, is a label on the same three-step routine.

Tap through: from your question to the answer

You askWhat is our cancellation policy for rentals?

1. Your question is turned into a search

Before the model reads anything, the tool turns your question into a lookup. Think of a librarian hearing the question and heading for the right shelf.

Step 1 of 4

Illustrative. Real tools retrieve more chunks and score them differently, but the shape is the same: search first, then read.

The librarian

Picture a reference librarian in a library with no catalog you can browse. You walk up and ask a question. The librarian disappears into the stacks, comes back with three pages torn from three books, and hands them to a very smart reader who has never seen the library. The reader answers your question using those three pages, and nothing else.

Two separate people, two separate jobs. The librarian (retrieval) decides which pages come out. The reader (the language model) decides what they mean. When the answer is great, both did their jobs. When the answer is wrong, you need to know which one slipped, because the fixes are completely different.

Chunks: how the library is shelved

The librarian cannot fetch "the relevant part" of a book unless the book has been cut into parts. So when you upload a file to a RAG-powered feature, the tool splits it into chunks: pieces of roughly a paragraph to a page, each stored separately with a note about where it came from.

Each chunk also gets a kind of fingerprint that captures what it is about, so the search can match meaning rather than exact words. That is why asking about "cancellations" can find a chunk that says "refund if you cancel 48 hours out." It is also why the match is not perfect: a fingerprint of meaning is a blurry thing, and two chunks about refunds can look alike to the search even when one is about deposits and the other is about cancellations.

Chunk boundaries matter more than anyone expects. A table split across two chunks loses its header in the second half. A clause that says "except as noted in section 4" is useless without section 4, which may be shelved three chunks away and never fetched. The librarian brings back pieces, and pieces have edges.

The hard part, simply

The hard part is that the model only knows what the librarian brought. It has no way to notice that a better page was left on the shelf, because it never saw the shelf.

Think of a takeout order. You ask for the fish tacos, the kitchen sends chicken, and the driver hands you the bag with a smile and says "here are your tacos." The driver is not lying. The driver has no idea what you ordered. The mistake happened in the kitchen, and the bag looks exactly like a correct order.

Worked example: a Hurricane equipment rental company loads its rental agreement into an AI tool. A staff member asks "can a customer get their money back if they bail on a booking?" The search scores the deposit paragraph highest, because it contains the word refund, and never fetches the cancellation paragraph. The model reads the deposit paragraph and answers, correctly for that paragraph, "yes, the $500 deposit is refunded within 5 days." It even cites section 7. The staff member tells the customer yes. The real answer, in section 4, was "only outside 48 hours." The citation looked like proof. It was proof the model read the wrong page carefully.

Why it still misses

Retrieval fails in a few predictable ways. Each one has a habit that catches it:

Your words and the document's words differ. You say "bail," the contract says "cancel." Meaning-based search helps, but the closer your wording is to the document, the safer the match. Habit: ask twice, once in your words and once using the document's own terms.
Too many similar documents. Ten versions of the same proposal, and the search grabs chunks from three of them. Habit: keep one current version in the library and archive the rest.
The answer is spread across chunks. Half the fact is on page 4, the other half on page 30. Habit: ask for both pieces separately, then ask the combined question with both quotes pasted in.
Tables and numbers. A chunk of a table is a fragment of a grid with no header. Habit: for numeric questions, upload the sheet directly and use a code-running feature (lesson 3) rather than asking the library.
The question is about the whole, not a part. "Summarize everything" or "how many contracts mention Sand Hollow" asks the librarian to fetch every page, which is exactly what retrieval is built not to do. You will get a summary of the five chunks that scored highest. Habit: for whole-library questions, work through files one at a time.

The habit under all the habits: ask for the quote, not just the answer. A tool that shows you the passage it used lets you check whether the librarian fetched the right page. A tool that just gives an answer is asking you to trust the kitchen.

Once you know the shape, you can grade any retrieved answer in about ten seconds. First, look for the citation. No quote, no page, no source name means you cannot tell what the librarian fetched, and you should ask for it before trusting the answer. Second, read the cited passage and ask one question: does this passage actually answer what I asked, or something nearby? The deposit paragraph answered a question about refunds. It did not answer a question about cancellations. That gap is where retrieval errors hide, and it is visible to anyone who reads the quote instead of the summary.

Third, notice what is missing. If you asked about three things and the answer covers two, the third probably did not get fetched. Ask about it separately, by name. Retrieval is a search engine, and like any search engine it does better with one specific question than with three bundled together.

Where you are already using RAG

You have probably used it without the label. As of August 2026: Claude Projects on paid plans switch to retrieval automatically when a project's knowledge folder grows past what the context window can hold, which Anthropic says expands capacity by up to 10 times. Gemini Notebook (the tool that was called NotebookLM until July 2026) is retrieval end to end: every answer cites the source passage it was built from, and it will not answer from outside your sources. AI web search is the same shape with the internet as the library: search, fetch a few pages, read, answer. Lesson 5 compares the standing knowledge features side by side.

Try this yourself

Upload one long document you know well to a tool with a knowledge or project feature, and ask a question you already know the answer to, using this shape:

Answer this question using only the documents I have provided: [your question]

Before the answer, quote the exact passage or passages you are relying on, with the document name and page or section. If the documents do not contain the answer, say "not found in the provided documents" instead of guessing. If two passages seem to conflict, show me both.

Then read the quoted passage. Is it the right one? If it is, the librarian works for this kind of question. If it quoted a different section than the one you had in mind, you have just watched retrieval miss in real time, and you know to rephrase using the document's own words. Try a second question where the answer is genuinely not in the file, and make sure it says so instead of inventing one.

Next lesson5. Projects and knowledge

Last updated August 24, 2026