AI regression testing in CI/CD

AI regression testing is the practice of running a fixed eval suite against every prompt, model, or config change, then blocking the deploy when quality drops below a defined threshold. It gives your AI functions the same pass/fail gate that software CI has had for decades -- with three structural differences that change how you set it up.

Why AI regression testing is harder than software regression testing

Software regression testing is deterministic. You call a function, you get the same output every time, and a failed assertion means something broke. AI regression testing is probabilistic: the same input can produce slightly different outputs on consecutive runs, and an 89% pass rate is not obviously worse than a 92% pass rate without a baseline to compare against.

Three structural differences follow from that:

1. Tests measure a distribution, not a value. You don't assert that output equals "billing" -- you assert that the function returns "billing" on at least 95% of runs across your golden dataset. The threshold is the gate, not the exact value.

2. The dataset is part of the system. In software testing, inputs are fixed by the spec. In AI regression testing, the golden dataset drifts over time as you discover new failure modes and promote production edge cases. The golden dataset needs continuous curation -- stale cases stop catching real regressions.

3. There are four regression triggers, not one. Software CI triggers on code changes. AI functions can regress from four distinct sources, and most CI configs only watch one.

The four triggers that cause AI regressions

Understanding each trigger is the prerequisite to wiring CI correctly.

1. Prompt changes

Every edit to a system prompt, few-shot examples, output format instructions, or chain structure is a regression risk. A simple wording change can dramatically alter an LLM's performance, fixing one issue while silently creating another. A prompt change that improves performance on the target scenario reliably degrades performance on edge cases the developer did not think about -- which is exactly what a golden dataset is built to cover.

2. Model swaps

Switching from one model to another -- including minor version bumps like gpt-4o-2024-08-06 to gpt-4o-2024-11-20 -- changes output behavior in ways that benchmarks don't predict. A model that scores better on average can score significantly worse on the specific input distribution your product sees. Model swaps should always trigger a full eval run before the config change merges.

3. Provider drift

This is the trigger most teams miss. Model providers update underlying models without incrementing the version name. A function running on a pinned model identifier can regress with no changes on your end. The only defence is a scheduled eval run -- daily or weekly -- against a recorded baseline, so drift surfaces as a delta rather than as a support ticket three weeks later.

4. Schema and tool changes

Changes to your output schema, tool definitions, or retrieval configuration are the fourth trigger. A new required field, a changed enum value, or a modified tool signature shifts what the model is expected to produce. A CI pipeline that does not trigger on model config files misses the most common source of silent regressions.

What your test suite needs to assert

A complete AI regression suite has three assertion layers:

Deterministic assertions -- fast, free, always consistent. Validate output schema (valid JSON, required fields present), enum values, format rules, latency ceiling, and cost per call. A cost assertion fails when a model's cost per call exceeds a threshold -- a regression shows up as a red test, not a surprise on next month's bill. Run these on every PR; they give feedback in seconds.

Semantic similarity -- for cases where exact string matching fails because the model can answer correctly in different phrasings. Semantic similarity scores convert both the baseline and the new output into high-dimensional vector embeddings and measure their contextual closeness -- eliminating the false-negative failures that make exact-match useless for LLM evaluation.

Model-graded scoring (LLM-as-judge) -- for free-text dimensions like tone, faithfulness, and instruction adherence. LLM-as-a-Judge uses a large language model to score quality of another model's output based on a predefined rubric. Two requirements before trusting it as a gate: set the judge's temperature to 0 (non-zero temperature produces non-deterministic scores), and calibrate agreement against human-labeled examples before treating it as a hard block.

For a classification function, layer all three: JSON schema validation (deterministic), field-value matching (deterministic), and a rubric check on the explanation field (model-graded). Fail fast on the cheap checks before paying for the expensive ones.

Wiring the four triggers into CI

The trigger config is where most teams leave gaps. A standard CI pipeline triggers on every push or PR. An AI regression pipeline needs path-based triggers that cover all four sources:

# .github/workflows/fn-eval.yml
on:
  pull_request:
    paths:
      - "prompts/**"         # trigger 1: prompt changes
      - "fn.yml"             # triggers 2 & 4: model + schema config
      - "tools/**"           # trigger 4: tool definitions
  schedule:
    - cron: "0 6 * * *"      # trigger 3: provider drift -- daily baseline check

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run regression suite
        run: fn eval classify --dataset tickets.jsonl --threshold 0.95

The paths: filter means the eval only runs when a relevant file changes -- not on every docs commit -- keeping CI fast. The daily schedule covers provider drift independently of any code change.

With fnAI, the eval command compares the candidate version against the last recorded baseline automatically:

fn eval classify --dataset tickets.jsonl --threshold 0.95

Sample output:

$ fn eval classify --dataset tickets.jsonl --threshold 0.95

 96 / 100 cases passed
x case 23: expected "billing", got "support"
x case 44: expected "billing", got "account"
x case 71: expected "high", got "medium"
x case 88: expected "billing", got "shipping"

score: 96.0%  threshold 95%  -- pass
vs v3: -1.8% (within tolerance)
consistency: 5/5 runs identical
cost: $0.003/call   latency p95: 142ms

A 1.8% regression against the previous version shows up in the PR report, not in a support queue a week later. If it crosses the threshold, the build exits non-zero and the deploy never runs.

The runtime approach

Building this pipeline with glue scripts is possible, but you end up owning four moving parts: the dataset store, the eval runner, the baseline tracker, and the CI trigger config. When the dataset or judge changes, the baseline comparison becomes meaningless unless you re-run history.

fnAI treats regression testing as a first-class part of the function lifecycle. Every version -- prompt, model config, tool definitions -- is immutable and content-addressed. fn eval compares against the last passing version automatically. A daily drift check runs at the runtime level, so you get notified when provider behavior shifts without owning a cron job in your own infrastructure.

The function stays the same call from your application code:

const result = await fn.run("classify", { ticket: rawText });

What changes under the hood -- model version, prompt version, eval baseline -- is the runtime's responsibility. Your application code does not change when a prompt regression is caught and rolled back, or when the runtime promotes a better-scoring model version after it clears the eval gate.

Start with 20-50 golden cases, a 0.95 threshold, and one CI step triggered on prompt file changes. That catches the most common regression -- the prompt edit that fixes one case and silently breaks five others -- with under five minutes of setup. Expand to 100-300 cases once you have production traffic to draw from; a production golden set of 100-300 highly diverse prompt-response pairs provides statistical significance without excessive CI overhead.

Ready to add regression gates to your AI functions? Join the fnAI waitlist for early access.

FAQ

What is the difference between AI regression testing and a pre-ship eval? A pre-ship eval is a single run before deploying a new version. AI regression testing is the broader system: golden dataset management, baseline tracking, CI trigger config covering all four regression sources, and a scheduled drift check. The eval is one component; regression testing is the full discipline.

Which CI trigger do most teams miss? Provider drift -- when a model provider silently updates the model behind a pinned API identifier with no code change on your end. The fix is a scheduled eval run against a recorded baseline. Without it, provider-side changes accumulate invisibly until they surface as user-facing failures.

How large should my golden dataset be? Start with 20-50 cases for a smoke-check on every PR. Expand to 100-300 for a full regression suite once you have production traffic to draw from. Prioritize cases from real incidents -- those are the edge cases that actually matter.

When should I use model-graded scoring vs. deterministic assertions? Use deterministic assertions for everything that has a correct answer (schema, field values, cost, latency). Use model-graded scoring (LLM-as-judge) for subjective dimensions -- tone, faithfulness, instruction adherence -- where no single correct string exists. Always run deterministic checks first: they are faster, cheaper, and more reliable.

How do I catch regressions from a model swap before it reaches production? Trigger your eval suite on any change to your model config file (fn.yml or equivalent). The eval exits non-zero if the new model scores below your threshold vs. the current baseline, and the deploy never runs. If the new model scores better, the eval records the new baseline and the previous version stays available for rollback.