What is the difference between fn eval and production monitoring?

fn eval runs a fixed golden dataset before deployment to catch regressions against known inputs. Production monitoring scores live traffic after deployment to catch distribution drift and novel failures. Both gates are necessary — eval is the cheaper, earlier one.

// blog

July 22, 20266 min read

How to evaluate an LLM prompt before shipping it

Q: How do I catch regressions from silent model updates?

Wire fn eval into CI so it runs on every PR that touches the prompt, model config, or output schema. A recorded baseline score is the only way to detect a silent drift from a provider checkpoint update.

Evaluating an LLM prompt means running it against a fixed dataset and blocking the deploy when the pass rate falls below your threshold. Here's how.

Evaluating an LLM prompt before shipping means running a version of that prompt against a fixed dataset of real inputs, scoring each output field-by-field against expected values, and blocking the deploy when the pass rate falls below a threshold you set. It is the equivalent of a unit test suite for a deterministic function — except the output is probabilistic, so you measure the fraction of outputs that pass rather than requiring every output to match exactly.

Why "it looked good in testing" is not a shipping criterion

Manual review is what most teams start with. You tweak the prompt, run a few examples, it looks good, you ship it. Three days later support flags that billing tickets are being tagged "shipping" — and nobody knows when it started.

The problem is structural. A simple wording change can dramatically alter an LLM's performance, fixing one issue while silently creating another. A model provider quietly updates a checkpoint under the same API identifier. Temperature set at 0.7 adds variance that only surfaces at scale. None of this shows up in a handful of eyeball tests.

You need a baseline — a fixed dataset and a recorded score — so you can answer the one question that matters after any change: is this version better or worse than the last one?

What goes in a golden dataset

A golden dataset is a JSONL file where each line pairs an input with an expected output:

{"input": "ticket-001.pdf", "expected": {"category": "billing", "priority": "high"}}
{"input": "ticket-002.pdf", "expected": {"category": "support", "priority": "low"}}

Start with 20–50 cases for a smoke-check that runs on every pull request, then expand to 100–300 for a full regression suite. A production golden set typically needs 100–300 highly diverse, mutually exclusive prompt-response pairs — enough for statistical significance without excessive CI overhead.

The best cases come from production traffic: real inputs, including the edge cases that tripped you up before. Every incident your AI feature experiences is a new golden case waiting to be captured. You do not need to assert every output field. If you only care about category, assert only category. Keeping assertions tight reduces false failures from fields that are allowed to vary.

How to set a threshold and run the eval

Once you have a dataset, set a numeric pass threshold. For a classification function, 0.95 — meaning 95% of asserted fields must match — is a reasonable starting point. Below that number, the deploy does not happen.

With fnAI:

fn eval classify --dataset tickets.jsonl --threshold 0.95

Sample output:

$ fn eval classify --dataset tickets.jsonl --threshold 0.95

✓ 47 / 50 cases passed
✗ case 23: expected "billing", got "support"
✗ case 31: expected "high", got "medium"
✗ case 44: expected "billing", got "account"

score: 94.0% (threshold 95%) ✗ fail
vs v2: −1.2% ▼
consistency: 5 / 5 runs identical

The command scores field-by-field and exits non-zero below 0.95 — so CI fails and the deploy never runs. fnAI also compares the new version against the last recorded run automatically. A quiet 2% regression shows up as a delta in the report, not in a support ticket a week later.

Why a single eval run is not enough

LLMs are non-deterministic. A prompt that passes 97% of cases on one run might pass 91% on another if temperature is above 0. A pass rate from a single run is a lucky draw, not a grade.

fn eval classify --dataset tickets.jsonl --runs 5

The --runs flag executes each case N times and reports a consistency figure alongside the score: how many of the five runs produced identical output. A prompt scoring 97% with 2/5 consistency on the same input is telling you something about variance that will surface in production at scale — especially under latency pressure or with bursty input distributions.

How to make it a CI gate

Running an eval once before the first ship is not enough. Regressions come from changes you make — a reworded system instruction, a tighter output schema — and from changes that happen to you: silent model checkpoint updates, provider behavior drift, input distribution shift.

The fix is to run fn eval on every change that touches the prompt, the model config, or the output schema:

# .github/workflows/fn-eval.yml
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Evaluate before deploy
        run: fn eval classify --dataset tickets.jsonl --threshold 0.95

Because fn eval exits non-zero below threshold, the CI job fails and the deploy never runs. Every PR that touches a prompt, a model version, or a retrieval configuration should trigger an eval run against the golden dataset. A PR that regresses quality past the acceptable threshold does not merge.

That is the same rule software teams apply to unit tests — it just took the AI ecosystem a few years to apply it to prompts.

For speed, run a 20–50 case smoke set on every PR, and a full 200–500 case regression set on merge to main. The smoke set gives feedback in seconds; the full suite runs on merge where pipeline latency matters less.

When to use model-graded scoring

Field matching works cleanly for structured output. For free-text — summarization, explanation, tone — there is no single correct string to match against. That is where LLM-as-judge comes in.

LLM-as-a-Judge uses a large language model with an evaluation prompt to rate generated text based on criteria you define — handling subjective dimensions like tone, instruction adherence, and factual accuracy that a regex cannot.

Two things to get right before you trust it as an automated gate:

Calibrate first. Measure judge agreement against 20–30 human-labeled examples specific to your task. A judge achieving less than 80% agreement with human evaluators on your task type is not reliable enough for automated blocking decisions.
Set the judge's temperature to 0. A non-deterministic judge produces non-deterministic scores — defeating the point of a repeatable gate.

For structured output with occasional open-ended fields, layer both approaches: deterministic field matching for the structured parts, model-graded scoring for the free-text fields. The fnAI evaluation guide notes that LLM judge scores should be treated as a signal, not ground truth — a human review cadence stays the calibration layer that keeps automated scoring honest.

What a passing eval does — and doesn't — guarantee

A passing eval tells you this version performs at or above your quality bar on the inputs you tested. It does not guarantee the model never hallucinates on inputs your golden set did not cover, or that quality holds six weeks later as the input distribution drifts.

That is why pre-ship eval and production monitoring are two different gates. The eval answers: "Is this version ready to replace the last one?" Monitoring answers: "Is it still working now?" Both are necessary. The eval is the cheaper, earlier one — and the one most teams skip until the first regression finds them.

If your AI function ships without a passing eval, you are not deploying software. You are betting.

FAQ

How many test cases do I need to evaluate an LLM prompt? Start with 20–50 cases for a smoke-check that runs on every PR. Expand to 100–300 for a full regression suite. Prioritize cases from real production traffic and from past incidents — those are the edge cases that actually matter.

What pass threshold should I set for my LLM eval? 0.95 is a reasonable starting point for a structured classification function. Calibrate based on the cost of a wrong answer in your use case — a medical routing function warrants a tighter bar than a tag suggestion.

How do I catch regressions from silent model updates? Wire fn eval into CI so it runs on every PR that touches the prompt, model config, or output schema. A recorded baseline score is the only way to detect a silent drift from a provider checkpoint update.

When should I use LLM-as-judge instead of field matching? Use field matching for structured output — JSON with enumerated categories or numeric values. Use LLM-as-judge for free-text output where tone, faithfulness, or instruction adherence needs to be assessed. Always calibrate the judge against human-labeled examples first.

Does a passing eval guarantee the prompt is safe to ship? No. A passing eval means the version performs at your quality bar on your test inputs. Production monitoring is the second gate — it catches distribution drift and novel failures that your golden set did not cover.

Ready to stop shipping prompts on gut feel? Join the fnAI waitlist — fnAI is an AI Function Runtime that bakes fn eval into the deploy path, so every version is certified better than the last before it goes live.