How to Build an LLM Hallucination Detector

A hallucination detector is a scoring system that checks whether an LLM's output is factually grounded — in the retrieved context, a reference answer, or a set of verifiable claims — and flags or blocks outputs that fabricate, contradict, or extrapolate beyond that ground truth.

Why does hallucination detection start before the model call?

Most hallucination detectors focus on the output. The smarter defense starts earlier: a typed contract that rejects malformed or out-of-scope inputs before a single token is generated.

When you deploy an Agentic Function in Functional AI, every call to POST /v1/run/<slug> is validated against the typed input schema defined in function.yaml. A query outside that schema is rejected immediately — no model call, no hallucination opportunity, no tokens spent. Bad inputs are a primary driver of hallucinated output: the model receives ambiguous context and fabricates a plausible-sounding answer rather than refusing. Cutting those calls at the boundary is cheaper and more reliable than scoring outputs after the damage is done.

When should you run hallucination detection?

Pre-deployment evaluation catches regressions before they ship. Run your agent against a ground-truth dataset, score each output for factuality, groundedness, and contradiction, then inspect the delta from the last version. This is where you catch a prompt rewrite that quietly dropped factuality from 94% to 87%.

Production trace scoring applies the same scorers to live traffic after deployment. Stale retrieval indexes, upstream model changes, and real edge cases produce failure modes that only surface at scale — on queries you never thought to include in a golden dataset.

Runtime guardrails inspect outputs before delivery and block or route high-risk responses. These add latency and over-trigger on open-ended tasks. Apply them selectively on high-stakes workflows: billing queries, legal summaries, compliance-adjacent answers.

The stages depend on each other: production scorer tuning requires pre-deployment validation, and regression coverage requires flagging live failures into your eval dataset.

What methods work for LLM hallucination detection?

LLM-as-a-judge is the most flexible approach. A second model reviews the first model's output against a rubric — "Is this answer fully supported by the retrieved context? Score 0–1 with a one-sentence explanation." It handles open-ended factuality checks that rules cannot. The failure mode is rubric drift: when the upstream prompt changes and the rubric is not updated, scores shift without anyone noticing. Treat the judge prompt as a versioned dependency alongside the prompt it evaluates.

Groundedness scoring for RAG checks whether the answer is supported by retrieved chunks. RAGTruth (ACL 2024), a corpus of nearly 18,000 annotated RAG responses, confirmed that even with retrieval augmentation, LLMs regularly produce claims unsupported by or contradicting the retrieved content. Three checks to implement:

• Context relevance: did the retriever return chunks that actually answer the query?
• Faithfulness: does the answer stay within what those chunks say?
• Citation validity: does each factual claim trace to a specific retrieved chunk?

Deterministic pre-checks run before any model scorer and cost nothing. Field presence, JSON parseability, citation count above a required minimum, output length within expected bounds. Run these first — they catch a meaningful share of failures at near-zero cost before you spend tokens on scoring.

Consistency sampling generates multiple responses to the same prompt and measures disagreement. High disagreement correlates with hallucination on factual tasks. Useful for pre-deployment evaluation; too expensive for high-volume production scoring.

Which hallucination failure modes slip past standard detectors?

Silent hallucinations pass every rubric but are subtly wrong. A summary that preserves all explicit facts but drops a critical qualifier — "this only applies to enterprise plans" — reads as grounded but misleads users. Rubrics that only check for contradiction miss this. Add completeness and material-omission checks when high-stakes answers need to carry full context.

Tool-call hallucinations occur when an agent invents parameters for a tool call: a fabricated user ID, a date outside the valid range, a URL it made up. The model's output is a function call, not prose, so prose-based groundedness checks miss it entirely. Test tool-call behavior with mocked tools and defined schemas — verify the agent only passes parameters grounded in the actual input and prior context.

In Functional AI, guarded tools are invisible to the model until their activation condition is satisfied. A tool requiring a confirmed user ID cannot be called with a fabricated one — the model can't see that tool until the guard holds. Structural control, not scored control.

Judge drift happens when the judge rubric falls out of sync with the prompt it's evaluating. After a rewrite, the old rubric may reward behaviors the new prompt no longer produces, or penalize outputs that are now correct. Calibrate against a human-labeled set every time the prompt changes — not just when scores dip.

How do you build a hallucination detector end-to-end?

Start with your function definition, deploy, then run the eval. Get started for self-serve access, or talk to an engineer if you want a Functional AI engineer to walk through the initial experiment configuration with you.

fn login
fn new function.yaml
fn deploy
fn eval

Each fn eval run executes an Experiment: your dataset against your Score Set, across the model and repeat configuration you chose. The result is a trust score, findings, and — when a scorer flags a response — suggested prompt fixes. fn eval exits non-zero on failure, so you can drive it from your own CI pipeline:

# .github/workflows/eval.yml
- name: Run hallucination eval
  run: fn eval

When a production trace surfaces a suspicious output, flag it into a test case from the trace view — two clicks. That case joins the dataset your next fn eval run scores against. A hallucination that reached a user immediately tightens regression coverage for the next deploy.

Because what you tested is what runs — the artifact is hashed and immutable — rolling back is one command:

fn rollback

URL and invoke keys stay the same. Your integration keeps calling the same endpoint; Functional AI routes traffic to the previous version.

How does hallucination detection work differently for RAG agents?

For retrieval-augmented agents, instrument context relevance, faithfulness, and citation validity as separate scored spans — not a single composite output score.

A faithfulness score of 0.4 against a context-relevance score of 0.95 tells you the retriever found the right chunks but the model ignored them — a generation-side fix, probably a prompt change that reinforces grounding instructions. A context-relevance score of 0.4 points at the retrieval pipeline: stale indexes, poor chunking, or a drifting query rewriter. The composite output score can't tell you which one. Per-span scores in fn logs can.

FAQ

What is a hallucination detector?

A hallucination detector is a scoring system that checks whether an LLM's output is factually grounded — in the retrieved context, a reference answer, or verifiable claims — and flags or blocks outputs that fabricate, contradict, or extrapolate beyond that ground truth. It runs pre-deployment against a golden dataset, on live production traces, and optionally as a runtime guardrail before outputs are delivered.

What's the difference between a hallucination detector and a runtime guardrail?

A hallucination detector scores outputs for factual accuracy and groundedness, surfacing failures for review and regression. A guardrail inspects outputs before delivery and can block or rewrite them in real time. Detection works at every lifecycle stage; guardrails operate only at runtime and add latency. Most teams implement detection across evals and production monitoring first, then add guardrails selectively on high-risk workflows.

How reliable is LLM-as-a-judge for hallucination detection?

Reliable in production when the rubric is concrete, anchored to examples, validated against human-labeled outputs, and treated as a versioned dependency alongside the prompt it evaluates. The primary failure mode is rubric drift — vague or stale criteria produce inconsistent scores that don't surface real regressions, especially after a prompt rewrite.

How does hallucination detection work differently for RAG agents?

RAG hallucination detection separates three checks: context relevance (did the retriever return relevant chunks?), faithfulness (does the answer stay within those chunks?), and citation validity (does each claim trace to a specific chunk?). RAGTruth (ACL 2024) confirmed that LLMs produce unsupported claims even when relevant context is retrieved. Scoring these as separate spans lets you distinguish a retrieval failure from a generation failure — the root cause and the fix differ significantly.

Can deterministic checks replace LLM-based scoring?

No — they complement it. Deterministic checks (field presence, JSON parseability, citation count, output length) catch a significant share of failures at near-zero cost and should run first. LLM-as-a-judge and groundedness scorers handle the open-ended factuality and faithfulness checks that rules cannot express.

How do I turn a production hallucination into a regression test?

In Functional AI, flag a failed production trace into a test case in two clicks from the trace view. It joins the dataset your next fn eval run scores against. fn eval exits non-zero on failure, so the new test case can fail your CI step before the same regression ships again.