Where Hallucinations Come From & the 4-Stage Defense Loop
"RAG reduces hallucinations" carries a hidden premise: the evidence entering the context window is accurate, and the model actually restricts itself to that evidence.
In production RAG systems, plausible-sounding answers that mislead users almost always trace back to three failure modes:
- Entity Drift: Semantically similar chunks about a different entity enter the candidate pool. The user asks about
deleteAccount APIerror codes, vector search retrievesdeleteUser APIdocs, and the model confidently explains the wrong endpoint. - Version Mismatch: Both legacy and current specs live in the knowledge base. The old doc gets retrieved, breaking changes disappear from the answer, and users follow an obsolete runbook.
- Fabricated Details: The hardest failure to spot. The chunk only says
deleteAccountchecks foreign-key references, and the model helpfully adds "you can usually callcleanupReferences APIfirst"—an endpoint that does not exist anywhere in your codebase.
Fabrication happens because an LLM is optimized to produce fluent, helpful text rather than strictly quote retrieved facts. Turning candidate chunks into auditable answers requires four defenses after reranking: a pre-generation threshold gate, numbered evidence injection, citation extraction, and faithfulness verification.
Pre-Generation Gating & Numbered Evidence Injection
Block under-supported queries with rerankScore first, then pass surviving chunks with explicit index numbers so every claim maps back to a concrete source.
Many RAG pipelines still shove low-scoring chunks into the prompt when nothing relevant exists, forcing the model to stitch together a plausible guess. Gating with EVIDENCE_THRESHOLD delivers two wins at once: you skip a wasted LLM call, and you give the user actionable feedback on what is missing and where to look instead of a vague "I don't know."