RAG Evaluation: How to Test If Your RAG Actually Works

You built a RAG chatbot. It answers questions, the demo looks great, and your friends are impressed. Then someone asks a question slightly outside your test script and it confidently invents an answer. This is the moment every RAG builder hits: the demo works, but does the system work?

RAG evaluation is how you answer that question with evidence instead of vibes. This guide covers what to measure (retrieval and generation separately), how to build a test set without spending weeks, which metrics and frameworks to use, and the failure patterns evaluation reveals.

Why RAG needs its own evaluation approach

A RAG system has two halves that fail independently:

Retrieval — did it find the right chunks?
Generation — given those chunks, did the LLM produce a correct, grounded answer?

A wrong answer can come from either half, and the fixes are completely different. Bad retrieval needs better chunking or embeddings; bad generation needs better prompts or context handling. Evaluating the end-to-end answer alone cannot tell you which half broke. Always evaluate retrieval and generation separately.

Note: "The demo looked good" is not evaluation. Demos show a system at its peak; evaluation shows it at its average — and its worst.

Step 1: build a test set

You need question-answer pairs grounded in your documents. For a student project, 50–100 questions is enough to be useful. Three ways to build them:

Method	Effort	Quality
Write by hand	High	Highest — you know what matters
LLM-generated, human-verified	Medium	Good if you verify every pair
Log real user questions	Low (after launch)	Most realistic, but needs users

The practical student path: have an LLM draft questions from your documents, then manually verify each one — check the question is answerable from the docs and the reference answer is correct. Unverified synthetic test sets measure how well your system agrees with another LLM's hallucinations.

Cover these question types deliberately:

Simple lookup — answer is in one chunk ("What is the refund deadline?")
Multi-hop — needs combining two chunks ("Which plan includes the feature announced in March?")
Unanswerable — not in the documents; the correct behavior is "I don't know"
Adversarial — contradictory instructions, ambiguous phrasing, edge cases

The unanswerable category is the most revealing. Most student RAG systems fail it badly — they answer anyway.

Step 2: evaluate retrieval

For each test question, record which chunks your retriever returned. Then compute:

Recall@k: fraction of questions where at least one relevant chunk appears in the top k results. The single most informative retrieval metric.
Precision@k: fraction of the top k that is relevant. Matters when context windows are small.
MRR (mean reciprocal rank): average of 1/(rank of first relevant chunk). Rewards putting the right chunk first.

def recall_at_k(retrieved_ids, relevant_ids, k=5):
    top_k = set(retrieved_ids[:k])
    return len(top_k & set(relevant_ids)) / len(relevant_ids)

Diagnosis guide:

Symptom	Likely cause	Fix
Low recall@k	Chunks too big/small, weak embeddings	Re-chunk, try a stronger embedding model, add hybrid search
Right chunk at rank 8, not top 3	Ranking weak	Reranker (cross-encoder) on top candidates
Relevant chunk missing entirely	Content not in index or chunked badly	Check ingestion; split on semantic boundaries

A reranker deserves special mention: retrieve 20–50 candidates with fast vector search, then score them precisely with a cross-encoder model and keep the top 5. It is the highest-ROI retrieval upgrade in most RAG systems.

Step 3: evaluate generation

Given good retrieved chunks, is the answer correct and faithful? Two families of measurement:

Human evaluation (the gold standard)

Have someone (you, a teammate) grade answers on a rubric:

Correctness: does the answer match the reference answer?
Faithfulness: is every claim supported by the retrieved chunks? (Check each sentence against the context.)
Appropriateness: right level of detail, no rambling, admits ignorance when appropriate.

Grade blind if you can — hide which system version produced the answer. Even 30 graded examples beat any automated metric for decision-making.

LLM-as-judge (the scalable approximation)

Use a strong LLM to grade answers against the reference, with a strict rubric prompt. This is what frameworks like RAGAS and DeepEval automate. It correlates reasonably with human judgment for faithfulness and correctness — and it is dramatically cheaper than human grading at scale.

Caveats: the judge has biases (it favors longer answers, and its own style). Calibrate it: grade 20 examples yourself, grade them with the judge, and check agreement before trusting it on 500.

Frameworks: RAGAS, DeepEval, and plain scripts

Tool	What it gives you	When to use
Hand-rolled scripts	Full control, zero dependencies	Student projects — start here
RAGAS	Standard metrics (faithfulness, answer relevancy, context recall/precision)	When you want comparable, citable numbers
DeepEval	Similar metrics + pytest-style test cases	When you want evaluation as CI tests

Start with scripts: recall@k for retrieval, human grading for generation. Adopt RAGAS when you need to compare configurations systematically or report numbers in a report or paper. Do not adopt a framework before you have a test set — the test set is the hard part.

The RAG triad: three numbers that matter

If you track only three metrics, make them these (the "RAG triad" popularized by TruLens):

Context relevance — is the retrieved context relevant to the question? (Retrieval quality.)
Groundedness — is the answer supported by the retrieved context? (Faithfulness.)
Answer relevance — does the answer actually address the question? (Generation quality.)

A system can score well on any two and fail the third. High context relevance + low groundedness = the LLM is ignoring your retrieval and answering from its own knowledge. That specific pattern means your prompt needs stronger grounding instructions, not better retrieval.

Common failure patterns evaluation reveals

The confident hallucination. Retrieval returned nothing relevant; the model answered anyway. Fix: add a "say I don't know" instruction and test the unanswerable category explicitly.
The context ignorer. Relevant chunks retrieved, answer contradicts them. Fix: prompt engineering ("answer ONLY from the context below"), or a smaller context window so the model attends better.
The chunk salad. Retrieved chunks are individually relevant but contradictory or from different document versions. Fix: metadata filtering (date, version), deduplication.
The slow decay. Worked in testing, degrades as documents are added. Fix: regression-test on your eval set every time the corpus changes — evaluation is not a one-time activity.
The metric mirage. Automated scores look good but users complain. Your test set does not match real usage — go log real questions.

A minimal evaluation workflow for your project

Build 50 verified Q&A pairs covering the four question types.
Run retrieval, compute recall@5. Fix retrieval until recall@5 is solid (aim high — retrieval is the foundation).
Run end-to-end, human-grade 30 answers on correctness + faithfulness.
Calibrate an LLM judge against your grades; use it to compare configurations.
Re-run the suite after every significant change (new chunks, new prompt, new model).

Keep the eval set frozen and versioned. The moment you tune your system against the test set, it stops being a test set — keep a small held-out subset you never look at until final reporting.

Where to go from here

Evaluation assumes a working RAG system — if you are still building, start with the RAG project guide and the vector database comparison. For framework choice around your pipeline, see LangChain vs LlamaIndex. To keep bad outputs from reaching users regardless of eval scores, read guardrails for LLM apps. The ML evaluation metrics guide covers the underlying metrics in depth. More LLM engineering in the AI & Machine Learning branch hub.

RAG Evaluation: How to Know If Your RAG Actually Works