Evaluation

Evaluation in fn is a command, not a notebook. You define a dataset and a threshold; fn turns every change into a pass/fail signal.

Dataset format

A dataset is a JSONL file — one JSON object per line — with an input and an expected field:

{"input": "ticket-001.pdf", "expected": {"category": "billing", "priority": "high"}}
{"input": "ticket-002.pdf", "expected": {"category": "support", "priority": "low"}}

expected may be a full output object or a partial one. Only the keys you specify are scored, so you can assert on category alone while ignoring confidence.

Scoring

fn eval classify --dataset tickets.jsonl --threshold 0.95

Each case is graded field-by-field. The reported score is the fraction of asserted fields that matched. A score below --threshold exits non-zero.

Open-ended output

Field matching works when the output is structured. For free-text or agent output, exact match is the wrong tool: use model-graded checks instead. Treat those scores as a signal, not ground truth — LLM judges carry positional and self-preference bias and drift between runs. fn tracks graded scores per version so drift is visible, but a human review cadence stays the calibration layer that keeps automated scoring honest.

Multi-run consistency

Models are non-deterministic. Use --runs to execute each case multiple times and measure variance:

fn eval classify --dataset tickets.jsonl --runs 5

The report includes a consistency figure — how many of the N runs produced identical output — alongside the score.

Regression detection

Every eval is compared to the last recorded run for that function. The report shows the delta versus the previous version, so a quiet 2% regression doesn't slip through.

CI gates

Because fn eval exits non-zero below threshold, gating a deploy is one step:

- name: Evaluate function
  run: fn eval classify --dataset tickets.jsonl --threshold 0.95

If the eval fails, the job fails, and the deploy never runs.