RAG Evaluation: Metrics and Failure Modes

RAG evaluation is the practice of scoring a retrieval-augmented generation pipeline across two independent dimensions—retrieval quality and generation faithfulness—so that a change to chunking strategy, embedding model, top-k, or the generation prompt is measured before it ships rather than after a user files a bug report. Because the retriever and the generator fail independently, a single accuracy score hides the actual failure mode; you need one metric per layer.

Part of our guide to LLM evaluation.

Key takeaways

• Faithfulness and answer relevance measure the generator; context precision and context recall measure the retriever. Both layers fail silently in production.

• High faithfulness combined with low context recall means the model is accurately summarizing incomplete information—as dangerous as hallucination, and far harder to detect.

• The retrieval layer is where production RAG most commonly degrades: fixed-size chunking, stale embedding models, and miscalibrated top-k all erode quietly as the corpus grows and query patterns shift.

• Seed your evaluation dataset with real production queries, not synthetic ones—intent distributions from actual traffic are what your scores will guard against.

• Wrap your RAG pipeline as an Agentic Function with a typed output contract: structurally invalid responses are rejected before any tokens are spent, and the exact artifact you evaluated is the one that runs in production.

What is RAG evaluation?

A RAG pipeline has two components: a retriever that pulls chunks from a knowledge base, and a generator (an LLM) that produces an answer from those chunks. Evaluation must assess both separately. The retriever might return irrelevant chunks; the generator might hallucinate beyond what any chunk supports; or both components might perform well on the initial query distribution while silently degrading as the product evolves and the corpus grows.

Traditional NLP metrics—BLEU and ROUGE—are inadequate for RAG. Both measure token overlap against a reference answer, not factual grounding in retrieved context. They were designed for machine translation, not for verifying that a model's claims are supported by a document corpus. A response that paraphrases source text accurately but in different words may score poorly; a response that fabricates a plausible-sounding fact may score well.

Reference-free evaluation, introduced by the RAGAS framework (Es et al., 2023), uses an LLM judge to score each metric without requiring a human-labeled ground-truth answer for every query. This is what makes systematic evaluation tractable at production scale. For a broader look at how LLM-as-judge fits into an evaluation strategy, see LLM as Judge: How It Works and When to Use It.

What are the key RAG evaluation metrics?

Faithfulness measures whether every claim in the generated answer is supported by at least one retrieved chunk. The LLM-as-judge decomposes the answer into atomic statements and checks each one against the context. A score of 0.72 means 28% of the model's claims have no grounding—those are hallucinations. High faithfulness means low hallucination risk; the model is staying inside the evidence it was given.

What it diagnoses: Generator overreach—the LLM is asserting facts that no retrieved chunk supports.

Answer relevance measures whether the generated answer actually addresses the question. A response can be perfectly faithful (every claim grounded) and still fail here if it answers a related but different question. This is the metric that catches prompt drift: the generator becomes topically related but unfocused.

What it diagnoses: Instruction failure—the model is producing adjacent content rather than directly answering the question.

Context precision measures the signal-to-noise ratio of retrieved chunks. If you retrieve five chunks and three are irrelevant, your precision is 0.4. Low precision forces the generator to filter noise, which raises the probability of hallucination or missed focus. This metric does not require a ground-truth answer.

What it diagnoses: Retriever over-retrieval—top-k is too high, or the embedding model is returning semantically adjacent but contextually irrelevant chunks.

Context recall measures whether the retrieved chunks contain all the information needed to answer the question correctly. This is the only metric that requires a ground-truth reference: the judge identifies the facts in the reference answer and checks which ones appear in the retrieved context. A recall of 0.6 means 40% of the facts needed to answer correctly were never retrieved. No amount of prompt engineering recovers information that was not in the context window.

What it diagnoses: Retriever under-retrieval—relevant passages are missing, the chunking strategy split them across boundaries, or top-k is too low.

How do you evaluate a RAG system?

Step 1: Define the typed contract. Before running any evals, specify the input and output schema for your RAG pipeline. In Functional AI, this lives in function.yaml. The typed contract rejects malformed inputs before any tokens are spent and catches output regressions where the model stops returning required fields—like sources—after a prompt change.

# function.yaml
slug: support-rag
input:
  type: object
  properties:
    question: { type: string }
    user_tier: { type: string, enum: [free, pro, enterprise] }
  required: [question, user_tier]
output:
  type: object
  properties:
    answer: { type: string }
    sources: { type: array, items: { type: string } }
  required: [answer, sources]

Step 2: Build a test dataset from production queries. Synthetic questions drift from actual user intent; the questions your users actually ask are the ones your scores need to guard. Aim for 50–200 question-context-answer triples—a 20-question set produces scores with too much variance to act on confidently. Include adversarial cases: questions that require multi-chunk synthesis, questions outside the knowledge base's scope, and ambiguous queries.

Each dataset row needs: question, contexts (the chunks your retriever actually returned, not ideal chunks), answer (the generated response), and ground_truth (a reference answer—required for context recall, optional for the other three metrics).

Step 3: Score the retrieval layer first. Context precision and context recall tell you whether retrieval is the bottleneck. If context recall is below 0.75, fix retrieval before touching the prompt. The right chunks have to be in the context window; nothing downstream compensates for their absence.

fn eval support-rag --key fnk_…

Step 4: Score the generation layer. Once retrieval metrics are stable, score faithfulness and answer relevance. These catch generator regressions that prompt edits, model upgrades, or system prompt changes introduce.

Step 5: Integrate into CI. Every change to the RAG pipeline—prompt template, embedding model config, chunking parameters, or top-k—should run the full eval suite before deploy. fn eval exits non-zero on regression, so you can call it from your own CI pipeline alongside your unit tests.

Evals inform a human deploy decision; they do not block deploys automatically. The deploy step runs only the Contract Test (instant structural validation of the output shape). The eval scores are what the engineer reviews before approving.

If your team wants help wiring up the full pipeline—retrieval, evaluation, and production tracing—talk to an engineer.

Which RAG failure modes degrade production silently?

The "accurate but incomplete" trap. Teams monitor faithfulness and ignore context recall. A faithfulness score of 0.93 looks healthy—until you check context recall at 0.55. The pipeline is accurately summarizing the half of the knowledge base it happened to retrieve and confidently presenting that as the complete answer. Users get responses that sound authoritative but omit critical detail.

Chunk boundary failures. Fixed-size character chunking splits tables, numbered lists, and multi-paragraph arguments across boundaries. The retriever returns fragments that lack the surrounding context needed to interpret them. The generator hallucinates to fill the gap. The fix is structure-aware chunking: split on paragraph breaks, section headers, or sentence boundaries rather than character count.

Top-k miscalibration. Top-k is set at development time and rarely revisited. As the corpus grows, the same top-k may now return mostly irrelevant chunks for queries that used to work well. Run context precision and context recall at multiple top-k values against your current production dataset regularly—not just once at initial build.

Embedding model drift. If the embedding model changes—by your own upgrade or a silent provider update—the distance relationships in your vector store no longer match what the new model produces. Context recall collapses while faithfulness scores stay high, because the generator faithfully synthesizes whichever wrong chunks the misaligned index returned. See hallucination detection for the monitoring patterns that catch this before users do.

Query distribution shift. The eval dataset built at launch reflects month-one user behavior. By month three, query patterns have evolved as the product is used differently. Re-seed your dataset from production queries every 30–60 days. In Functional AI, a failed production call can be flagged into a test case in two clicks—get started to see how that production feedback loop works.

After each deploy, watch for calls with status failed or partial:

fn logs support-rag --status failed --key fnk_…

Those surface retrieval gaps that your eval dataset may not yet cover. Flag them, add them to the dataset, and re-run. Over time, your eval set converges on the real failure modes your users encounter rather than the hypothetical ones you imagined at build time.

For a broader look at tooling choices across the RAG stack, see the RAG production checklist. For the full evaluation taxonomy—evals, score sets, experiments—see the LLM evaluation hub.

FAQ

What is RAG evaluation?

RAG evaluation is the practice of scoring a retrieval-augmented generation pipeline on retrieval quality (context precision and context recall) and generation quality (faithfulness and answer relevance) so that every change to chunking, embeddings, top-k, or the generation prompt is measured before it ships.

What are the most important RAG evaluation metrics?

The four core metrics are faithfulness, answer relevance, context precision, and context recall. Context recall is the most commonly skipped because it requires a ground-truth reference to compute—but it is the only metric that reveals whether the retriever is returning everything the question needs.

How do I evaluate a RAG system without ground-truth labels?

Faithfulness, answer relevance, and context precision are reference-free: an LLM judge scores them against the question and retrieved context without needing a labeled answer. Context recall is the exception—it requires a reference answer to identify the facts that should have been retrieved. Start with the three reference-free metrics and add context recall as you label a dataset.

How often should I run RAG evals?

Run the full eval suite on every change to the retrieval or generation layer, and run it monthly against recent production queries even with no code changes. Query distribution shift and corpus growth both degrade scores without any engineering change—you will not see it unless you are measuring.

What causes RAG systems to regress after launch?

The three most common causes: corpus growth that changes what the retriever returns for the same query; query distribution shift as user behavior evolves; and embedding model changes that misalign the index. Monthly eval runs against production queries catch all three before users do.

How is RAG evaluation different from standard LLM evaluation?

Standard LLM evaluation treats the model as a black box and measures output quality against a reference. RAG evaluation must also assess the retrieval step independently. A high-quality generator cannot compensate for a retriever returning the wrong chunks—the information simply is not in the context window.