What Is RAG? Retrieval-Augmented Generation, Simply
A language model knows a lot about the world and nothing about your company. Retrieval-augmented generation — RAG — is the standard way to fix that without retraining anything.
- Published
The one-sentence version
RAG means: before the model answers, search your documents for the most relevant passages and put them in the prompt, so the model answers from those passages instead of from memory.
How it works, step by step
- Ingest: split your documents into chunks — a few paragraphs each.
- Index: turn each chunk into an embedding (a numeric fingerprint of its meaning) and store it in a search index.
- Retrieve: when a question arrives, find the chunks whose meaning is closest to the question, often combined with keyword search.
- Generate: give the model the question plus those chunks, with instructions to answer only from them and cite them.
When RAG is the right tool
- Answers must come from your own, changing content: policies, product docs, contracts, tickets.
- You need citations so people can check the source.
- Content changes often — re-indexing is cheap; retraining a model is not.
Where RAG goes wrong
| Failure | Cause | Fix |
|---|---|---|
| Right document, wrong answer | Chunk cut mid-table or mid-thought | Structure-aware chunking |
| Wrong document retrieved | Pure semantic search misses exact terms | Hybrid keyword + semantic search, reranking |
| Outdated answer | Old versions still indexed | Source-of-truth sync and deletion |
| Confident answer, no source | Weak instructions | Require citations; refuse when nothing relevant is found |
Frequently asked questions
Is RAG the same as training on my data?
No. RAG looks information up at question time; training changes the model itself. RAG is cheaper, faster to update and easier to audit.
Do I need a vector database?
You need some search index. Small projects can use the database you already have; larger ones benefit from a dedicated vector store.
How accurate can RAG get?
With clean data, hybrid search and a proper eval set, production assistants routinely answer the large majority of in-scope questions correctly — and refuse the rest.
