An internal assistant draws on your 4,000 documents. It does not read them all for every question: they would not fit in its context window, and the bill would be huge. It first looks up the few passages that matter, adds them to the request, then answers from them. That is RAG, short for retrieval-augmented generation.
Search first, answer second
A RAG system works in three steps. Ahead of time, the documents are split into passages and indexed. When a question comes in, the application searches for the passages closest to it. It then places them in the request, with an instruction to stick to them. The model no longer answers from memory. The answer is grounded in your documents, which reduces hallucinations and lets you cite the sources.
A question comes in: before calling the model, the application searches your documents.
Key concept
RAG fetches the few useful documents when the question comes in, instead of sending everything to the model. The quality of the answer depends first on the quality of the search. If the right passage is not found, the model cannot use it, and it answers anyway.
That is the RAG trap. When an answer is wrong, people blame the model, but often the search brought back the wrong passages. So measure the two separately: does the search find the right passage, and does the model draw the right answer from it?
The search misses the right passage: the model gets other passages, answers anyway and cites sources that do not answer the question.
What it means for you
For product managers
Decide which sources the assistant consults, who keeps them up to date, and what it says when it finds nothing. Show the sources it used so the user can check them. An out-of-date document base produces out-of-date answers, with the same confidence.
For tech leads
Treat the search pipeline as a product in its own right: chunking, embeddings, index, re-ranking. Test it on its own, with real questions where you already know the right source. In a startup, a standard database with a vector search extension is often enough. In a large company, the hard part lies elsewhere: dozens of sources, uneven access rights, and documents that may carry malicious instructions.
Go deeperEmbeddings
To search by meaning rather than by words, each passage is turned into an embedding: a long list of numbers that represents its meaning. Two texts with similar meanings have similar embeddings. The question "am I allowed to stay home to do my job?" therefore finds the passage "remote work: two days a week", with no word in common. These lists are computed by specialized models that do not write text (see Other kinds of models). In October 2026, Anthropic does not offer an embedding model of its own: its documentation names Voyage AI as one provider and advises comparing several.
The classic mistake
fAIling RULE I
The trap: Paste your entire codebase into the prompt
With a large document collection, pasting it all into the instructions "so the model has everything" means asking the model to find a needle in a haystack on every request, and paying for the whole haystack every time. RAG does that sorting before the model is called. You still have to check that it does it well.
Read rule I in the manifestoTry it
Gather ten questions your users really ask. For each one, find by hand the passage that answers it. Then put each question to your search tool, or to your assistant, and note whether that passage appears in the top five results. That score tells you whether the problem lies with the search or with the model.
Around this topic
Go deeperSplitting documents
Before indexing, the documents have to be split into passages, a step called chunking. Make the passages too short and they lose their context. Make them too long and they dilute the useful information. Anthropic published a telling example in September 2024: the passage "the company's revenue grew by 3% over the previous quarter" says neither which company nor which quarter. Its method adds a short sentence of context to each passage before indexing it.
Go deeperHybrid search and re-ranking
Searching by meaning sometimes misses the obvious: a contract number, a product reference, a proper name. So it is combined with classic keyword search, which is called hybrid search. A re-ranking model then rereads the best candidates and puts them back in order. In tests Anthropic published in September 2024, adding context to the passages, combined with hybrid search, cut retrieval failures by 49%, and by 67% with re-ranking.
Go deeperAccess rights on retrieved documents
An assistant must not cite a document the user is not allowed to open. The filter is applied at search time, based on the rights in the original source, never through an instruction in the prompt. In a large company, HR, legal and finance folders make this a blocker. A retrieved document can also contain text written to hijack the model: that is prompt injection, which guardrails deal with.
Go deeperLong context or retrieval?
In October 2026, Anthropic's current models accept a million tokens per request (chunks of text, often parts of words), or about 555,000 words according to its documentation. For a small, stable document base, sending everything can be enough: as early as September 2024, Anthropic advised doing so below 200,000 tokens, with prompt caching to cut the cost. For a large, changing or access-controlled base, retrieval is still needed.
What to do with it
Decide which sources the assistant searches, who keeps them current and what it says when it finds nothing: a stale base gives stale answers.
Show the sources the assistant used, so the user can check them.
Measure the search and the answer separately, starting from real questions whose right source you already know.