AI Agent Guide Interactive
Module 1 · LLM Engineering Fundamentals TypeScript + Zod + output_config.format 3 Interactive Labs

Prompt Regression Evaluation: Turning "Feels Better" into Comparable Numbers

You change one line in a prompt from "be concise" to "analyze in detail," or upgrade the underlying model. The API still returns 200 OK and valid JSON, yet your code-review assistant starts flagging a routine currency formatter as a security vulnerability—or quietly misses a real SQL injection. Without a fixed test suite and assertion rules, you cannot tell model variance apart from a real regression.

30-Second Summary

1. What "Worse Output" Actually Means: Don't Start with Another LLM

Regression testing in AI engineering treats a non-deterministic model as a function under test: fixed inputs + fixed prompt + fixed assertions = comparable experiments.

Many teams hear "evals" and immediately wire up a more expensive model as a judge. LLM judges have their place, but starting there gets the order backwards. Catch cheap, deterministic failures in code first by splitting regressions into four observable dimensions:

Dimension 1 · Hardest Contract
Format Drift

A prompt tweak makes the model wrap JSON in a Markdown fence or emit score as a string, breaking downstream parsers immediately.

Check: output_config.format + strict Zod schema
Dimension 2 · Core Logic
Content Regression

The JSON parses cleanly, but a SQL injection that used to be caught disappears behind generic advice like "consider improving code style."

Check: mustMention keywords + issueTypes assertions
Dimension 3 · Restraint & UX
Style & Boundary Drift

A one-sentence summary balloons to 800 words, or a clean utility function gets flagged with three fabricated "security risks."

Check: character caps + zero-false-positive clean-code cases
Dimension 4 · Production Cost
Cost Regression

Adding 600 words of prompt instructions raises pass rate by 2%, while doubling input and output tokens on every request.

Check: log usage.input_tokens / output_tokens & latency
Fixed Test Suite cases.json (4 cases) 3 known defects + 1 clean negative case L1 · Hard Schema output_config.format Zod validates shape Fail fast on drift L2 · Code Assertions Rule-based checks Keywords / types / score Zero extra LLM cost L3 · Gate Version diff Pass rate & P0 Ship or block
Figure 1: Catch structural and semantic regressions with L1 schemas and L2 code assertions before deciding whether a prompt version can ship.
Predict First
Suppose your code-review eval suite has 4 test cases, and all 4 contain real bugs or security flaws. Your new prompt hits 100% pass rate on all 4. Is it safe to ship?
Spot on: A suite containing only broken code pushes you to make the prompt harsher and harsher. Soon the model starts inventing "security vulnerabilities" and "performance issues" in clean functions. Inspect the real model output in Lab 1 below.
Interactive Lab 01 · Real Output Trace & Negative-Case Diagnosis

Why Did review-v1 Fail on Clean Code?

Inspect the actual run report from examples/core-methods/prompt-regression/run-eval.ts on qwen-plus (temperature: 0), localized into English. Switch prompt versions and test cases to compare outputs and assertion results. Real run data · prompt-regression-report.md

Prompt:
Test Case:
Case: missing-null-check Using review-v1.md

            
Expected Assertions (cases.json)
Suite Pass Rate: 3 / 4 (75%) PASS
Assertion Checklist Model Score: 30
    Actual qwen-plus JSON Output
    
              
    Diagnosis

    2. Growing Prompt Rules from Failing Cases

    The constraints in review-v2.md were not guessed up front—they grew directly out of a failing test case.

    In Lab 1, review-v1 saw a standard Intl.NumberFormat price helper and invented three defects: it labeled an unvalidated currency argument a "security vulnerability (information leak)," complained about missing JSDoc comments, and scored the function 72.

    Once that over-review failure mode was captured, review-v2.md added three targeted guardrails:

    1. Explicit review priority: runtime bugs > security risks > performance bottlenecks > maintainability.
    2. An explicit zero-issue exit: when code has no clear defects, return an empty issues array and a score between 80 and 100.
    3. Clear boundaries for security and performance: do not report security vulnerabilities without visible untrusted input, database, file, or network boundaries; do not flag routine object allocation or formatting helpers as performance issues.

    Re-running the exact same 4 cases after those edits made clean-code pass (score 95, issues: []) with zero regressions on the first 3 defect cases. That is the core payoff of regression testing: fix a known failure mode and prove you did not break existing behavior.

    3. Wiring Assertions and Release Gates into Your Workflow

    Never rely on overall pass rate alone. If a P0 security case flips from pass to fail, the release gate must block the build even if the aggregate pass rate looks acceptable.

    In production, keyword, type, and score checks are only the starting line. Teams typically add two more safeguards: an output length cap (catching sudden verbosity spikes that break UI layouts or inflate token spend) and repeated runs on critical cases (even at temperature: 0, backend updates or floating-point non-determinism can introduce occasional variance).

    Interactive Lab 02 · Assertion Sandbox & Release Gate

    Tune Gate Thresholds and Model Variance to See When Releases Block

    Layer an output character cap and multi-run stability checks onto the real run-eval.ts skeleton. Watch the release gate decision update live alongside the generated TypeScript snippet. Gate rules & variance are simulated

    Candidate Prompt
    JSON Cap (maxChars) 1000 chars
    Stability & P0 Gate
    Case ID Priority Output Size Semantic Check Length Check Gate Status
    Release Gate: BLOCKED
    3 / 4 Passed
    View synced TypeScript eval script (run-eval.ts excerpt)
    evaluateCase + Length Gate + Multi-Run Stability

    4. Don't Overestimate a Minimal Script: 5 Engineering Limits

    A few dozen lines of TypeScript assertions solve the first 60% of regressions—they are not a full evaluation platform.

    1. Keyword checks are blunt: A model can mention "SQL injection" while giving a broken fix, or describe the exact same vulnerability using synonyms and trigger a false failure.
    2. Scores are not universal: What "60/100" means shifts across prompts unless anchored to a concrete rubric.
    3. Single runs can flake: Even with temperature: 0, provider-side model updates and hardware non-determinism create occasional drift; P0 cases need repeated runs.
    4. No LLM-as-Judge yet: Soft qualities like "is this explanation clear?" or "is the suggested patch actionable?" cannot be verified by regex alone and eventually require a rubric-guided judge model.
    5. No token cost tracking: Production eval runners should record response.usage.input_tokens and output_tokens so a 2% quality gain never sneaks in a 2x bill increase.
    Interactive Lab 03 · Three-Stage Eval Maturity

    From Minimal Script to Production Evals: What Each Layer Buys and Costs

    Select an evaluation tier to compare regression coverage, runtime cost, and signal quality. The engineering rule stays constant: lock down deterministic rules first, then add model judges selectively. Illustrative coverage & cost comparison

    Stage 1: Schema + Code Assertions Extra judge calls: 0 / case

    Hard Regressions
    75%
    Soft Quality
    25%
    Time / Cost
    Low (1x)

    5. Choosing Off-the-Shelf Eval Tools & Daily Workflow

    Write your own script first to internalize how an eval harness works. Once your suite grows past a few dozen cases—or needs CI integration and team dashboards—adopt a dedicated tool.

    Promptfoo Prompt Diff & CI

    Closest match to this chapter's workflow. Declarative YAML organizes prompts, providers, and assertions with a built-in side-by-side web viewer for CI regression runs.

    Docs: promptfoo.dev →
    DeepEval Python / RAG / Agent

    Pytest-style LLM testing framework with built-in metrics like Faithfulness and Answer Relevancy—ideal once you move into RAG retrieval and agent evals.

    Docs: deepeval.com →
    LangSmith Team Traces & Datasets

    Hosted platform for turning failing production traces into regression dataset items with one click, connecting live observability with offline version comparisons.

    Docs: LangSmith Evaluation →
    DSPy Programmatic Optimization

    Built around "programming, not prompting": automatically optimizes instructions and few-shot examples against your metric—once you already have a reliable dataset.

    Docs: dspy.ai →

    Embedding Regression Evals into Your Daily Loop

    A runnable script is only the starting point. The real leverage comes from making four habits automatic: run and save a baseline report before editing any prompt; re-run the exact same cases afterward to diff results; whenever production or manual testing turns up a bad output, add that input to cases.json before touching the prompt; and before shipping, check P0 regressions—not just the headline pass rate.

    Task Type Do First Best Delegated to AI Engineer Must Verify Acceptance Gate
    AI Feature Iteration Prepare positive & negative cases Draft prompts, parsers, and retry logic Output quality, token cost, fallback paths Version diff on a fixed case suite
    Bug Fix Reproduce the bad output Read logs, spot conflicting rules Root cause validity and collateral regressions Add regression case first, then verify fix
    Agent Capability Define tools & permission boundaries Scaffold execution loop and tool wrappers Privilege escalation, infinite loops, unrecoverable states Inspect tool call trace and final state
    Full Runnable Source Code
    The complete example lives in examples/core-methods/prompt-regression/ (including cases.json, prompts/review-v1.md, prompts/review-v2.md, and run-eval.ts). With an Anthropic-compatible endpoint that supports output_config.format (such as Alibaba Cloud Bailian Qwen), run npx tsx --env-file=.env examples/core-methods/prompt-regression/run-eval.ts.

    Prompt regression evaluation is ultimately an engineering habit, not a stack of frameworks. A handful of real, representative test cases—especially negative cases—is worth far more than any eval dashboard. As you move into RAG knowledge bases and autonomous agents, the exact same loop scales up: the assertions simply expand from a single JSON payload to retrieval recall, citation faithfulness, and multi-step tool trajectories.