Prompt Regression Evaluation: Turning "Feels Better" into Comparable Numbers
You change one line in a prompt from "be concise" to "analyze in detail," or upgrade the underlying model. The API still returns 200 OK and valid JSON, yet your code-review assistant starts flagging a routine currency formatter as a security vulnerability—or quietly misses a real SQL injection. Without a fixed test suite and assertion rules, you cannot tell model variance apart from a real regression.
30-Second Summary
Regressions are rarely HTTP errors: LLM quality drops show up as schema drift, missed risks, nitpicking clean code, or doubled token bills. Catch hard failures with code assertions first.Jump to Lab 1 →
Include negative test cases: A suite full of buggy snippets tunes your prompt into an over-aggressive reviewer. Clean code cases verify that the model knows when to stay quiet.Jump to Lab 2 →
Rules before judges: Block baseline failures with schemas and deterministic checks first, then layer usage cost budgets and LLM-as-Judge only where needed.Jump to Lab 3 →
1. What "Worse Output" Actually Means: Don't Start with Another LLM
Regression testing in AI engineering treats a non-deterministic model as a function under test: fixed inputs + fixed prompt + fixed assertions = comparable experiments.
Many teams hear "evals" and immediately wire up a more expensive model as a judge. LLM judges have their place, but starting there gets the order backwards. Catch cheap, deterministic failures in code first by splitting regressions into four observable dimensions:
Dimension 1 · Hardest Contract
Format Drift
A prompt tweak makes the model wrap JSON in a Markdown fence or emit score as a string, breaking downstream parsers immediately.
Check: output_config.format + strict Zod schema
Dimension 2 · Core Logic
Content Regression
The JSON parses cleanly, but a SQL injection that used to be caught disappears behind generic advice like "consider improving code style."
Figure 1: Catch structural and semantic regressions with L1 schemas and L2 code assertions before deciding whether a prompt version can ship.
Predict First
Suppose your code-review eval suite has 4 test cases, and all 4 contain real bugs or security flaws. Your new prompt hits 100% pass rate on all 4. Is it safe to ship?
Spot on: A suite containing only broken code pushes you to make the prompt harsher and harsher. Soon the model starts inventing "security vulnerabilities" and "performance issues" in clean functions. Inspect the real model output in Lab 1 below.
Interactive Lab 01 · Real Output Trace & Negative-Case Diagnosis
Why Did review-v1 Fail on Clean Code?
Inspect the actual run report from examples/core-methods/prompt-regression/run-eval.ts on qwen-plus (temperature: 0), localized into English. Switch prompt versions and test cases to compare outputs and assertion results.
Real run data · prompt-regression-report.md
Prompt:
Test Case:
Case: missing-null-checkUsing review-v1.md
Expected Assertions (cases.json)
Suite Pass Rate: 3 / 4 (75%)PASS
Assertion ChecklistModel Score: 30
Actual qwen-plus JSON Output
Diagnosis
2. Growing Prompt Rules from Failing Cases
The constraints in review-v2.md were not guessed up front—they grew directly out of a failing test case.
In Lab 1, review-v1 saw a standard Intl.NumberFormat price helper and invented three defects: it labeled an unvalidated currency argument a "security vulnerability (information leak)," complained about missing JSDoc comments, and scored the function 72.
Once that over-review failure mode was captured, review-v2.md added three targeted guardrails:
An explicit zero-issue exit: when code has no clear defects, return an empty issues array and a score between 80 and 100.
Clear boundaries for security and performance: do not report security vulnerabilities without visible untrusted input, database, file, or network boundaries; do not flag routine object allocation or formatting helpers as performance issues.
Re-running the exact same 4 cases after those edits made clean-code pass (score 95, issues: []) with zero regressions on the first 3 defect cases. That is the core payoff of regression testing: fix a known failure mode and prove you did not break existing behavior.
3. Wiring Assertions and Release Gates into Your Workflow
Never rely on overall pass rate alone. If a P0 security case flips from pass to fail, the release gate must block the build even if the aggregate pass rate looks acceptable.
In production, keyword, type, and score checks are only the starting line. Teams typically add two more safeguards: an output length cap (catching sudden verbosity spikes that break UI layouts or inflate token spend) and repeated runs on critical cases (even at temperature: 0, backend updates or floating-point non-determinism can introduce occasional variance).
Tune Gate Thresholds and Model Variance to See When Releases Block
Layer an output character cap and multi-run stability checks onto the real run-eval.ts skeleton. Watch the release gate decision update live alongside the generated TypeScript snippet.
Gate rules & variance are simulated
4. Don't Overestimate a Minimal Script: 5 Engineering Limits
A few dozen lines of TypeScript assertions solve the first 60% of regressions—they are not a full evaluation platform.
Keyword checks are blunt: A model can mention "SQL injection" while giving a broken fix, or describe the exact same vulnerability using synonyms and trigger a false failure.
Scores are not universal: What "60/100" means shifts across prompts unless anchored to a concrete rubric.
Single runs can flake: Even with temperature: 0, provider-side model updates and hardware non-determinism create occasional drift; P0 cases need repeated runs.
No LLM-as-Judge yet: Soft qualities like "is this explanation clear?" or "is the suggested patch actionable?" cannot be verified by regex alone and eventually require a rubric-guided judge model.
No token cost tracking: Production eval runners should record response.usage.input_tokens and output_tokens so a 2% quality gain never sneaks in a 2x bill increase.
Interactive Lab 03 · Three-Stage Eval Maturity
From Minimal Script to Production Evals: What Each Layer Buys and Costs
Select an evaluation tier to compare regression coverage, runtime cost, and signal quality. The engineering rule stays constant: lock down deterministic rules first, then add model judges selectively.
Illustrative coverage & cost comparison
Write your own script first to internalize how an eval harness works. Once your suite grows past a few dozen cases—or needs CI integration and team dashboards—adopt a dedicated tool.
PromptfooPrompt Diff & CI
Closest match to this chapter's workflow. Declarative YAML organizes prompts, providers, and assertions with a built-in side-by-side web viewer for CI regression runs.
Pytest-style LLM testing framework with built-in metrics like Faithfulness and Answer Relevancy—ideal once you move into RAG retrieval and agent evals.
Hosted platform for turning failing production traces into regression dataset items with one click, connecting live observability with offline version comparisons.
Built around "programming, not prompting": automatically optimizes instructions and few-shot examples against your metric—once you already have a reliable dataset.
A runnable script is only the starting point. The real leverage comes from making four habits automatic: run and save a baseline report before editing any prompt; re-run the exact same cases afterward to diff results; whenever production or manual testing turns up a bad output, add that input to cases.json before touching the prompt; and before shipping, check P0 regressions—not just the headline pass rate.
Task Type
Do First
Best Delegated to AI
Engineer Must Verify
Acceptance Gate
AI Feature Iteration
Prepare positive & negative cases
Draft prompts, parsers, and retry logic
Output quality, token cost, fallback paths
Version diff on a fixed case suite
Bug Fix
Reproduce the bad output
Read logs, spot conflicting rules
Root cause validity and collateral regressions
Add regression case first, then verify fix
Agent Capability
Define tools & permission boundaries
Scaffold execution loop and tool wrappers
Privilege escalation, infinite loops, unrecoverable states
Inspect tool call trace and final state
Full Runnable Source Code
The complete example lives in examples/core-methods/prompt-regression/ (including cases.json, prompts/review-v1.md, prompts/review-v2.md, and run-eval.ts). With an Anthropic-compatible endpoint that supports output_config.format (such as Alibaba Cloud Bailian Qwen), run npx tsx --env-file=.env examples/core-methods/prompt-regression/run-eval.ts.
Prompt regression evaluation is ultimately an engineering habit, not a stack of frameworks. A handful of real, representative test cases—especially negative cases—is worth far more than any eval dashboard. As you move into RAG knowledge bases and autonomous agents, the exact same loop scales up: the assertions simply expand from a single JSON payload to retrieval recall, citation faithfulness, and multi-step tool trajectories.