Tutorial2 min read
How We Build a RAG Assistant on Company Docs
This is the playbook we follow for internal and customer-facing assistants that answer from company documents. Each step exists because skipping it caused a problem on an earlier project.
TrueCodeAI Engineering
Agents, Voice & ML practice
- Published
RAGTutorialAI agents
The process
- Source audit: list every source, owner, format, update frequency and who may see it. Remove duplicates and outdated versions at the source.
- Eval set first: collect 100+ real questions with correct answers and the document each comes from, plus questions the assistant should refuse.
- Parsing: structure-aware extraction that keeps headings, tables and lists intact.
- Chunking: split by document structure, with headings carried into each chunk for context.
- Indexing: embeddings plus a keyword index, with metadata for source, date and permissions.
- Retrieval: hybrid search, a reranking step, and permission filters on every query.
- Generation: answer only from retrieved passages, cite each claim, refuse when nothing relevant is found.
- Evaluate and iterate until the agreed target is met on the eval set.
- Launch with feedback buttons and trace logging; review failures weekly.
What we measure
| Metric | Meaning |
|---|---|
| Retrieval hit rate | Correct source appears in the top results |
| Answer correctness | Graded against the expected answer |
| Citation accuracy | Cited passage actually supports the claim |
| Correct refusals | Out-of-scope questions declined |
| Latency and cost per answer | User experience and budget |
Timeline
A pilot on one clean source set takes two to three weeks. A production assistant with several sources, permissions and analytics takes three to five weeks.
Frequently asked questions
Which documents work best?
Well-structured text: policies, manuals, help articles. Scans and complex spreadsheets need extra parsing work.
How do you keep answers current?
Scheduled or event-driven re-indexing, with deleted documents removed from the index automatically.
Can it live in Slack or Teams?
Yes — the same assistant can serve a web widget, Slack, Teams and WhatsApp.
