// evaluation
Know a version is better before anyone sees it.
An LLM evaluation platform scores a prompt or agent against your own data, so you know whether a change made it better or worse before users see it. Functional AI is one, built into the same platform that runs the Agentic Function and traces every call: datasets, Score Sets, experiments across models, a trust score, findings with prompt fixes, and behaviour tests for agents.
New to the category? What is an LLM evaluation platform? covers the definition and what to look for.
What the evaluation platform is made of
Datasets
Real inputs, with expected outputs where you have them. Type them in, upload CSV, JSON or Excel, or flag a production call in two clicks. Dataset versions are frozen, so results stay comparable.
Score Sets
A named group of checks: code checks (free, deterministic), reference comparisons, AI judges, and labelled answers. Generate one from the definition before a single row of data exists, or pick checks by hand.
Experiments
One version, one dataset version, a Score Set, one or more models, several repeats. Every combination is one call, and a finished experiment never changes.
Trust score and findings
A score from 0 to 100 per model, pass rate per check, cost per run, p95 latency, stability by field across repeats, a confusion matrix for enum outputs, and findings that come with a fix for the prompt.
How do I evaluate an LLM prompt before shipping it?
- 01Build a dataset from real inputs. Export last month's tickets, upload the file, save a dataset version.
- 02Generate a Score Set from the definition: enum and required-field checks, a priority range, a judge on the free-text field, latency and cost.
- 03Run an experiment: two models, three repeats each.
- 04Read the result: trust score per model, the confusion matrix, and the findings. Apply the suggested fix as the next version.
- 05Compare the two versions on the same dataset and Score Set, then deploy the better one on the cheaper model.
The whole loop, from an empty project to a deployed classifier, fits in an afternoon. The step-by-step is in the documentation.
Five questions, one experiment each
- Is it consistent?
- Run one input five times; stability by field needs no labels.
- Is it right?
- A labelled dataset gives pass rate per check and, for enum outputs, a confusion matrix.
- Which model?
- Several models in one experiment, one card each, with cost per run and p95 latency.
- Is v4 better than v3?
- One experiment per version on the same dataset version and Score Set, then compare side by side.
- Did it get worse than what's live?
- Each run is compared with the last deployed version and a drop is flagged, never silently.
It evaluates what the agent did, not just what it said.
An agent can give the right answer after calling the wrong tool, or refund before anyone approved and still write a polite note. Its definition already says which tools exist, how many times each may be called, and which need a condition first, so the behaviour checks are generated from it. Tool responses are mocked or replayed per test row, so the tests never touch your real systems.
- Tool selection, tool arguments, and path match: exact, in order, or any order
- Step efficiency and task completion, separately from producing a valid output
- Escalation behaviour: did it ask a human when it should, and only then
- A guarded tool was never called before its condition held; a capped tool was never called twice
An agent that issues refunds can be tested a thousand times without issuing one.
Why evaluation belongs next to the runtime
What you tested is what runs
Each version compiles to one hashed artifact. The experiment scored it; deploy freezes exactly it.
Every failure becomes a test
Flag a production call into a dataset in two clicks. For agents, the steps it took come with it.
The contract writes the checks
Typed outputs, soft targets, rubric lines and tool guards become checks without anyone writing them.
Evaluation never blocks a deploy: the Contract Test is the only automatic check, and a human decides what goes live. fn eval exits non-zero on failure, so a team that wants a hard bar can run it in its own pipeline. Scores are priced in credits; see pricing.
// faq
Common questions
- What is an LLM evaluation platform?
- An LLM evaluation platform scores the outputs of a prompt or agent against your own data, so you know whether a change made it better or worse before users see it. It holds the datasets, the checks that define good, and the results of every run, so versions stay comparable over time. Functional AI is one, built into the same platform that runs the Agentic Function and traces every call.
- How do I evaluate an LLM prompt before shipping it?
- Build a dataset of real inputs, with expected outputs where you have them. Generate a Score Set from the prompt's definition, or pick the checks by hand. Run an experiment: one version, over the dataset, on one or more models, several repeats each. Read the trust score, the pass rate per check, the stability of each output field across repeats, and the findings, each with a concrete fix for the prompt. Then compare the new version with what is live and deploy when the evidence says so.
- Can an LLM evaluation platform test agents, not just prompts?
- On Functional AI, yes. An agent's definition already says which tools it may call, how many times, and under which conditions, so behaviour checks are generated from it: a guarded tool was never called before its condition held, a capped tool was never called twice, no side-effecting call was repeated. Tool responses are mocked or replayed per test row, so an agent that issues refunds can be tested a thousand times without issuing one.
- Do evaluations block deploys?
- No. Deploying runs the Contract Test, an instant structural check of the definition, and nothing else. Experiment results inform a human, who decides what goes live; a space can also require a second person to approve each deploy. fn eval exits non-zero on failure, so a team that wants a hard bar can run it in its own pipeline.
- What does evaluation cost?
- Every verdict on a test output is a score, at 0.15 credits. An AI-judge verdict adds one intelligence operation, 0.6 credits. Every plan includes monthly credits, and model calls run on your own provider key with no markup. Re-scoring a version with a new Score Set reuses the cached model calls, so it costs scores but no new tokens.
- Do I need my own model key to run evaluations?
- Experiments call models on your own key: Amazon Bedrock, Anthropic, OpenAI or Google. Generating a Score Set and running the Contract Test use Functional AI's keys, and a built-in mock model lets you try the whole loop before adding a key. AI judges can run on our keys too, with tokens billed at the provider's price.