How to Evaluate a RAG System: Metrics That Matter
You cannot improve a RAG system you do not measure, and "it seems better" is not a measurement. This is how we evaluate every retrieval assistant we ship.
- Published
Build the eval set
- Real questions from users or support logs — not invented ones.
- For each: the correct answer and the source passage.
- Include hard cases: ambiguous questions, multi-document answers, outdated-policy traps.
- Include out-of-scope questions that should be refused.
Evaluate retrieval separately
Measure whether the right passage appears in the top results before looking at answers. If retrieval fails, no prompt will fix the answer. Separating the two tells you which half to work on.
Grade the answers
| Dimension | Question |
|---|---|
| Correctness | Does it match the expected answer? |
| Faithfulness | Is every claim supported by the retrieved passages? |
| Completeness | Does it cover all parts of the question? |
| Refusal | Does it decline when it should? |
LLM graders scale this well, but calibrate them: have people grade a sample and check the grader agrees before trusting it.
Run it on every change
Every change — prompt, chunking, model, embedding — runs the full eval set. Results are compared with the last release, and regressions block the deploy. This is what makes it safe to keep improving the system.
Frequently asked questions
How many eval questions do we need?
Start with 100 good ones. Grow the set with every production failure.
Are automated graders reliable?
When calibrated against human judgement, yes for most dimensions. Keep a human-reviewed sample.
Who writes the eval set?
Your domain experts, with our help structuring it. It is the most valuable artefact of the project.
