chunking_strategies_evaluation/docs/evaluation-metrics.md

Evaluation Metrics

How scoring works in the benchmarking system.

---

Overview

The system uses LLM-as-Judge evaluation to automatically score answer quality across 4 metrics.

---

Metrics

1. Context Relevance (1-10)

Question: How relevant are the retrieved chunks to answering the question?

Score	Meaning
9-10	Chunks directly answer the question
7-8	Chunks are highly relevant
5-6	Chunks are somewhat relevant
3-4	Chunks are partially relevant
1-2	Chunks are not relevant

What it measures: Did the retrieval system find the right information?

---

2. Answer Similarity (1-10)

Question: How similar is the generated answer to the expected answer?

Score	Meaning
9-10	Nearly identical to expected
7-8	Covers most key points
5-6	Covers some key points
3-4	Partially correct
1-2	Completely different

What it measures: Did the system produce the right answer?

---

3. Faithfulness (1-10)

Question: Is the generated answer grounded in the retrieved context?

Score	Meaning
9-10	Entirely based on context
7-8	Mostly based on context
5-6	Partially based on context
3-4	Some external knowledge used
1-2	Mostly external knowledge

What it measures: Is the answer trustworthy?

---

4. Hallucination (boolean)

Question: Did the LLM invent information not in the context?

Value	Meaning
false	Answer is grounded in context
true	Answer contains invented information

What it measures: Is the answer fabricated?

---

Scoring Process

Step 1: Prepare Evaluation Context

Question: {user_question}
Context: {retrieved_chunks}
Expected Answer: {golden_answer}
Generated Answer: {system_answer}

Step 2: Send to LLM-as-Judge

prompt = f"""
Evaluate this RAG system output:

Question: {question}
Context: {context}
Expected: {expected}
Generated: {generated}

Return JSON:
{{
    "context_relevance": <1-10>,
    "answer_similarity": <1-10>,
    "faithfulness": <1-10>,
    "hallucination": <true/false>,
    "reasoning": "<explanation>"
}}
"""

Step 3: Parse Response

scores = json.loads(llm_response)
# Validate ranges
for metric in ["context_relevance", "answer_similarity", "faithfulness"]:
    scores[metric] = max(1, min(10, scores[metric]))

---

Overall Score Calculation

Each strategy gets an overall score weighted by importance:

overall = (
    context_relevance * 0.3 +      # 30% weight
    answer_similarity * 0.4 +       # 40% weight
    faithfulness * 0.3              # 30% weight
) * (1 - hallucination_rate)       # Hallucination penalty

Why These Weights?

Metric	Weight	Rationale
Answer Similarity	40%	Most important - did we get the right answer?
Context Relevance	30%	Found the right information
Faithfulness	30%	Answer is trustworthy
Hallucination	Penalty	Fabricated info is unacceptable

---

Aggregate Metrics

Per strategy, we calculate:

Metric	Calculation
avg_context_relevance	mean of all context_relevance scores
avg_answer_similarity	mean of all answer_similarity scores
avg_faithfulness	mean of all faithfulness scores
hallucination_rate	count(hallucination=True) / total_questions
total_questions	number of questions evaluated
failed_questions	questions that errored

---

Example Output

{
  "recursive": {
    "avg_context_relevance": 8.5,
    "avg_answer_similarity": 7.8,
    "avg_faithfulness": 9.2,
    "hallucination_rate": 0.05,
    "total_questions": 21,
    "failed_questions": 0
  }
}

---

Interpretation Guide

Good Scores

Metric	Target	Meaning
Context Relevance	≥ 8	Retrieval is accurate
Answer Similarity	≥ 8	Answers match expectations
Faithfulness	≥ 9	Answers are trustworthy
Hallucination Rate	≤ 0.1	Low fabrication rate

Warning Signs

Metric	Warning	Meaning
Context Relevance	< 6	Retrieval needs improvement
Answer Similarity	< 6	Answers are off-target
Faithfulness	< 7	Model is adding external knowledge
Hallucination Rate	> 0.2	High fabrication rate

---

Cost Considerations

Each evaluation uses:

Component	Cost
LLM call (gpt-4o-mini)	~$0.001 per evaluation
Input tokens	~500 per evaluation
Output tokens	~100 per evaluation

Total for 21 questions × 5 strategies:

• 105 evaluations × $0.001 = $0.105

---

Configuration

Evaluation parameters in src/benchmarking/evaluation.py:

_EVALUATION_SYSTEM_PROMPT = "You are an expert evaluator..."
_EVALUATION_USER_PROMPT = "Evaluate this RAG system output..."
model = settings.llm_model  # gpt-4o-mini
temperature = 0.0           # Deterministic
max_tokens = 500            # Response limit