How to Evaluate a RAG Pipeline: Metrics That Matter

A RAG pipeline has two separate places to fail, and most teams score it with metrics that only cover one of them. Retrieval can pull the wrong documents, or generation can ignore the right ones it was handed, and those are genuinely different bugs with different fixes. Scoring "did the final answer look okay" collapses that distinction and leaves you guessing which half of the pipeline to touch. This is a practical guide to evaluating both halves separately, and building a small evaluation set that actually catches regressions.

Why retrieval and generation need separate metrics

A RAG system's final answer depends on two independent steps working: the retriever has to surface the documents that actually contain the answer, and the generator has to use them faithfully instead of ignoring them in favor of what it already "knows." A single end-to-end quality score can't tell you which step failed. If the answer is wrong, you need to know whether the retriever handed the model the wrong material, or the right material got ignored, because the fix is completely different: better chunking and embeddings for the first, better prompting or a stronger model for the second.

Retrieval metrics: precision, recall, MRR, NDCG

Retrieval metrics measure the quality of the document set that comes back from the search step, independent of what the generator does with it afterward.

Precision measures the fraction of retrieved documents that are actually relevant, penalizing a retriever that returns a lot of noise alongside the useful chunks. Recall measures the fraction of all relevant documents that were successfully retrieved, penalizing a retriever that misses material the answer actually needed. These two pull in opposite directions, a retriever that returns everything gets perfect recall and terrible precision, so they're always worth reporting together rather than picking one.

Precision and recall don't account for ranking, though, and ranking matters because most generators only look closely at the first few chunks in context. Mean Reciprocal Rank (MRR) scores how early the first relevant document appears in the ranked results, which matters most when there's typically one clearly correct source. Normalized Discounted Cumulative Gain (NDCG) extends this to cases with multiple relevant documents of varying usefulness, weighting documents higher in the ranking more heavily, which is the more general metric when a query can have several partially-relevant sources rather than one right answer. Pinecone's guide to offline evaluation measures covers the mechanics of all four in more depth if you're implementing them from scratch.

Generation metrics: faithfulness and answer relevance

Once you know what the retriever handed the generator, the second question is what the generator did with it. Faithfulness (also called groundedness) measures whether every claim in the generated answer is actually supported by the retrieved context, rather than invented or pulled from the model's own training data. Answer relevance measures whether the generated answer actually addresses the question asked, independent of whether it's grounded, since an answer can be perfectly faithful to the retrieved context and still fail to actually answer the question.

The Ragas evaluation framework is the most widely used open implementation of this metric set and treats faithfulness, answer relevance, context precision, and context recall as a complementary group specifically so retrieval and generation failures stay distinguishable rather than collapsing into one score. In practice, faithfulness is usually scored with an LLM judge: the judge is given the retrieved context and the generated answer and asked whether each claim in the answer is supported. Our guide to using a judge model for evaluation covers the mechanics of that scoring approach and its own failure modes, which apply here as much as anywhere else a judge model is doing the scoring.

Metric definitions follow the framework described in the Ragas documentation and Pinecone's retrieval evaluation guide.

"Retrieved the wrong thing" vs. "ignored what was retrieved"

This is the distinction that makes separate metrics worth the extra setup. A low faithfulness score with high retrieval precision and recall means the right documents made it into context and the model still didn't use them properly, generating from its own prior knowledge instead. A low faithfulness score with low retrieval recall means the model never had the right material to begin with, and no amount of prompting will fix an answer that's grounded in documents that don't contain the answer.

Recent research specifically tracing failure modes at this level of detail found that models overriding retrieved evidence with their own prior knowledge is a distinct, recurring failure pattern, not a rare edge case, separate from cases where the evidence simply wasn't retrieved:

"We identify recurring facet-level failure modes, including evidence absence, evidence misalignment, and prior-driven overrides." Elchafei et al., "Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG"

That distinction should change what you fix first. If your generation metrics are bad but retrieval metrics look fine, the fix is on the prompting or model side, being more explicit that the answer must come only from supplied context, or escalating to a stronger model that follows that instruction more reliably. If retrieval metrics are the problem, better prompting won't help; you need better chunking, a different embedding model, or query rewriting before generation ever sees the right material.

Building a small RAG eval set

Start from real queries, not invented ones. Pull a sample of actual questions your system has handled or is expected to handle, then hand-label the ground truth for each: which documents in your corpus actually contain the answer, and what a correct answer looks like. This gives you the reference needed for precision, recall, and faithfulness scoring, since none of those metrics work without knowing what "relevant" and "correct" mean for each query.

Deliberately include queries your retriever is likely to struggle with: questions phrased differently than the source documents' wording, questions requiring information spread across multiple documents, and questions where the corpus genuinely doesn't contain an answer, since a good RAG system should be able to say so rather than generating a plausible-sounding guess. A small set of forty to eighty well-labeled queries covering these cases catches more real regressions than a much larger set of easy, obviously-answerable questions.

Run the retrieval and generation metrics separately against this set on every change to chunking, embeddings, retrieval parameters, or the generation prompt, and compare against your last baseline rather than an absolute bar. Our general rubric for evaluating LLM output covers the same discipline (fixed dataset, fixed scoring, comparison against baseline) for the non-RAG case, and the same process applies here with retrieval metrics added on top.

The cost angle: large retrieved context isn't free

One thing that's easy to miss when tuning a RAG pipeline purely for evaluation scores: retrieving more context to boost recall has a real cost beyond token price. Databricks' Mosaic AI research team, studying long-context RAG performance directly, found real limits to simply retrieving more:

"While retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens." Leng et al., "Long Context RAG Performance of Large Language Models"

There's also a routing-specific wrinkle worth knowing about if your pipeline sends requests to different models depending on load or cost: a large retrieved context that stays stable across calls is exactly the kind of prefix that benefits from provider-side prompt caching, and routing that context to a different model breaks the cache and can erase the savings you were routing for. Chasing a marginally better recall score by retrieving more chunks is worth weighing against both of these costs, not just against the evaluation number moving in the right direction.

Next steps

If you're building this evaluation loop for the first time, start narrow: pick one metric from each half (recall on the retrieval side, faithfulness on the generation side), label a few dozen real queries, and get a baseline before you touch chunking or prompts. For the checking that needs to happen on live traffic rather than in a batch eval, llm11's verification layer runs groundedness checks against supplied context on every response, the same faithfulness question this post covers, applied inline rather than after the fact, and escalates automatically when a check fails.

Frequently asked questions

What's the difference between RAG evaluation and general LLM evaluation?

General LLM evaluation scores the final output against a rubric. RAG evaluation adds a retrieval layer underneath that needs its own metrics, since a RAG system's answer quality depends on both what was retrieved and what the model did with it, and a single output score can't tell those two failures apart.

Which metric should I prioritize if I can only track one?

Faithfulness is usually the highest-leverage single metric, since ungrounded answers are the failure users notice fastest and trust the least. It won't tell you whether a retrieval problem is behind a low score, though, so track at least one retrieval metric like recall alongside it as soon as you can.

Do I need a large labeled dataset to evaluate RAG?

No. A well-constructed set of forty to eighty real queries with hand-labeled relevant documents and correct answers, covering easy cases, hard rephrasings, and genuinely unanswerable questions, catches most regressions that matter. Size matters less than deliberately including the cases your system actually struggles with.

Can an LLM judge score faithfulness automatically?

Yes, this is the standard approach: a judge model is given the retrieved context and generated answer and asked whether each claim is supported. It scales far better than manual review, though it inherits the same biases as any LLM-as-a-judge setup, so periodic calibration against human labels is still worth doing.