Why LLMs Hallucinate and How to Reduce It
A hallucination is a confident answer that is wrong. It is not a bug that will be patched out; it is a property of how these models work. The job is to design around it.
- Published
Why it happens
A language model generates the most plausible continuation of text. Plausible and true usually overlap — but when the model lacks the fact, a plausible-sounding invention is still the most likely continuation. It has no built-in sense of "I do not know" unless you give it one.
Where risk is highest
- Specific facts: numbers, dates, names, citations, prices.
- Niche or recent topics the model saw little of.
- Questions that assume something false.
- Long answers, where errors compound.
How to reduce it
| Technique | What it does |
|---|---|
| Grounding (RAG / tools) | Answers come from retrieved data, not memory |
| Required citations | Every claim points to a source that can be checked |
| Permission to refuse | Explicitly allow and reward "I don’t know" |
| Structured outputs | Constrain answers to known fields and values |
| Verification step | A second check compares the answer to the sources |
| Evals | Measure hallucination rate on real questions every release |
Design for the residual
No technique drives the rate to zero. For high-stakes outputs — legal, medical, financial — keep a human check. For lower-stakes ones, make sources visible so users can verify with one click.
Frequently asked questions
Do bigger models hallucinate less?
Generally less on common knowledge, but all models do it. Grounding matters more than model size.
Can we measure hallucination?
Yes — with an eval set of questions and correct answers, including questions the system should refuse.
Is temperature the fix?
Lower temperature makes output more consistent, not more truthful. It helps a little; grounding helps a lot.
