How We Build an Eval Suite for an AI Agent (Tutorial)
The eval suite is the part of an agent build that separates a product from a demo. This is the exact sequence we use to build one — for agents we wrote and for agents we inherited — in about two weeks of focused work.
- Published
Step 1–3 — source the scenarios
- Collect real inputs: transcripts, tickets, documents, tool logs. If the agent is not live yet, use the human process it replaces — the last 500 tickets your team handled are the best test set you will ever get.
- Sample 150–300 across the distribution, then deliberately over-weight the expensive failures: refunds, cancellations, anything irreversible, anything with a compliance angle.
- Add adversarial cases by hand: prompt injection inside a document or a tool result, requests for secrets, out-of-scope tasks, contradictory instructions. Twenty to thirty is enough to start.
Step 4–5 — define what "correct" means
- For each scenario, write the expected outcome as something checkable: the tool that should be called and its key arguments, the final state in the system, or a hand-off. Avoid "a good response" — that is not a test.
- Where a judgement call is unavoidable (tone, completeness), write a three-to-five-line rubric with concrete criteria and a pass bar. A model grades against it; a human audits a sample weekly.
| Field | Example |
|---|---|
| id | refund-partial-014 |
| input | Customer message + order record + policy snippet |
| expected_action | call issue_refund with amount ≤ 40% of order; then handoff |
| grader | structured (tool + args) + rubric (tone) |
| class | high-risk |
| pass_bar | 100% for class; 95% overall |
Step 6–8 — run it, wire it in, keep it honest
- Write a runner that executes each scenario against the agent with tools mocked or sandboxed, records the trace, grades it and produces a per-class pass rate and a diff against the last run.
- Put it in CI. A prompt, tool or model change that drops any high-risk class below its bar blocks the merge. This single rule prevents most production regressions.
- Split a held-out set — 20% — that nobody tunes against. Report the held-out number as the real one; the rest is for development.
Step 9 — keep it alive
- Every production failure becomes a scenario the same day. Every shadow-mode edit becomes a scenario. Prune cases that no longer reflect the business.
- Re-audit model grades monthly: sample 30, grade by hand, compare. If agreement drops, fix the rubric.
A suite built this way costs 15–25% of the agent budget and pays for itself the first time it stops a bad deploy. We build suites for agents we did not write, and we teach this method in a one-day workshop.
Frequently asked questions
How many scenarios before it is useful?
Fifty, honestly sourced, beats five hundred invented ones. Start at 150 if you can; grow from production.
How much does running it cost?
Cents to a few dollars per full run for most agents; model-graded rubrics are the main cost. Run the structured checks on every commit and the rubric set nightly if budget matters.
Can the eval set be used to compare models?
That is one of its best uses. Swap the model, run the suite, read the per-class diff. No opinions required.
