You tweaked chunking, swapped the embedding model, and rewrote the prompt. Everyone felt the answers “looked better,” until multi-hop production queries kept dropping critical compliance clauses. Without layered evaluation, tuning is guesswork; without tracking latency and token budgets, blindly switching to Agentic RAG makes simple lookups 3x slower.
30-Second TakeawaysDiagnose first, upgrade second
1. Evaluate retrieval and generation separately: If the right document never enters Top-K, no prompt rewrite can save the answer. Lock down Recall@K and MRR first.
2. Each RAGAS metric maps to a distinct fix: Low faithfulness points to prompt constraints or abstention; low context recall points to Top-K or chunking; low precision points to reranker thresholds.
3. Agentic RAG only pays off on multi-hop gaps: Single-document factual lookups belong on a fast fixed pipeline. Reserve iterative tool-driven retrieval for cross-document reasoning.
1. Retrieval Evaluation: “Not Found” vs. “Ranked Too Low”
RAG evaluation splits into two layers: retrieval quality checks whether evidence reached Top-K, and answer quality checks whether the model used that evidence properly. Blending both into a single score hides which layer broke.
Start with two deterministic metrics: Recall@K and MRR (Mean Reciprocal Rank). Recall@K measures the fraction of ground-truth documents that land inside the Top-K window passed to the LLM. MRR averages the reciprocal rank of the first relevant hit: rank 1 scores 1.0, rank 2 scores 0.5, and rank 4 scores 0.25.
Reading them together prevents wasted work. Low Recall@3 paired with healthy Recall@5 and MRR means the retriever found the evidence, but the reranker buried it at positions 4 or 5. Only when both Recall@5 and MRR collapse should you blame the embedding model or chunking strategy.
LAB 01 · RETRIEVAL METRICS MATRIX
Top-K Cutoff & Rank Position Workbench
Using the 5-query benchmark dataset (6 labeled ground-truth docs), drag the Top-K cutoff slider or switch retriever states to watch Recall@K and MRR diverge and trigger different fixes.
Retriever State
Live math from retrieval-metrics.ts
Top-K Window Sent to LLMK = 3
Solid green = hit inside Top-K; dashed red = relevant doc cut off outside K
Recall@3--
--
Starting gate: ≥ 80.0%
MRR--
--
Starting gate: ≥ 0.600
Missed Ground-Truth Docs
--
--
View TypeScript evaluation logic (synced with current K)
2. Generation Evaluation: Pinpointing Breakpoints with RAGAS
Passing retrieval checks does not guarantee trustworthy answers. RAGAS (docs.ragas.io) decomposes end-to-end quality into four metrics, each watching one specific link in the pipeline.
Figure 1: Two RAGAS metrics audit retrieval (Precision, Recall) and two audit generation (Faithfulness, Relevancy). Fixing the wrong layer wastes iterations.
Keep three ground rules in mind: 1) Build eval sets from real user logs—synthetic questions generated from your docs mirror doc phrasing and inflate Recall artificially. 2) 0.8 is a community starting point, not an official RAGAS law; more importantly, no single metric should drop below 0.6. 3) When Precision and Recall trade off, prioritize Context Recall: missing a required step is far riskier than including two extra paragraphs of context.
LAB 02 · RAGAS FAILURE DIAGNOSIS
RAG Incident Triage Workbench
Inspect three failing eval tickets and their four RAGAS scores. Pick the first engineering fix you would ship and verify whether you targeted the right layer.
Select Ticket
Diagnostic samples · Typical production failure modes
3. The Fixed-Pipeline Ceiling & the Real Cost of Agentic RAG
After a few tuning cycles, you will hit a class of queries where single-shot retrieval stalls: cross-document multi-hop reasoning. If Document A says “see the SOC2 Audit Retention Policy,” a single-pass retriever cannot know it needs to search for “SOC2 7-year archive” until after it reads Document A.
Agentic RAG turns the retriever into a callable tool. Instead of hardcoding one retrieval pass, the model inspects what it retrieved, decides whether evidence is complete, and issues follow-up sub-queries until it can answer or hits a hard step cap (MAX_STEPS).
That flexibility is never free. Every extra tool round adds another full-context LLM call, another latency hop, and another failure surface. Make a prediction below, then inspect the side-by-side ledger in Lab 03.
Predict First
For the multi-hop query “When deleteAccount returns 409, which deletion flow should we use, and can audit logs be physically deleted together?” (spanning 3 documents), what happens if you compare Fixed RAG bumped to Top-15 against a 2-hop Agentic RAG loop?
LAB 03 · FIXED PIPELINE VS. AGENTIC RAG
Fixed vs. Agentic Retrieval: Step-by-Step Trajectory & Cost Sandbox
Switch across three query complexities and step through both execution traces to compare evidence coverage, LLM calls, latency, and token cost.
4. Pragmatic Rollout Order & Architecture Selection
Do not build a full eval platform and multi-step Agent on day one. The highest-leverage rollout is: 20 real queries with Recall@5 baseline first, RAGAS on production bad cases second, and selective routing to Agentic RAG only for multi-hop traffic.
Avoid overfitting your tuning set. If you have 100 labeled queries, hold out 20 as a frozen validation set and only run it before shipping, so your improvements actually generalize to unseen phrasing.
Filter by Workload
Fixed RAG (Hybrid + Reranker) Default Start
Best for: FAQs, API docs, and single-doc policy lookups with strict P95 < 2s latency budgets.
Key focus: Push Recall@5 above 80% and MRR above 0.6. Around 90% of enterprise KB queries never need a dynamic loop.
Router + Conditional Escalation Production Pick
Best for: Mixed traffic where 80% of questions are straightforward and 20% span multiple linked policies.
Key focus: Gate entry with a lightweight classifier or first-pass confidence score so simple queries stay fast while hard ones escalate.
Full Agentic RAG Use Selectively
Best for: Root-cause incident triage, cross-clause compliance audits, and deep codebase investigations.
Key focus: Enforce MAX_STEPS, token caps, query deduplication, and explicit abstention fallbacks so loops cannot run away.
Minimal Offline Eval Gate Required Guard
Best for: Every change to chunk size, embedding model, prompt wording, or agentic routing rules.
Key focus: Start with 20–50 real queries for Recall@K & MRR; add RAGAS when users report hallucinations or off-topic replies, keeping every metric ≥ 0.6.
This closes the RAG module from parsing and hybrid retrieval through grounding, layered evaluation, and bounded Agentic RAG. Turning the retriever into a tool is also the natural bridge into Module 2’s core Agent Runtime, tool calling loops, and trajectory evaluation.