Research/chunking/docs/adr/0005-retrieval-evaluation-framework.md

ADR 0005: Retrieval Evaluation Framework

Status

Accepted

Context

The current system has no metrics for retrieval quality. The previous classification accuracy was unknown, making improvement impossible. The new architecture needs measurement from day one to:

1. Establish baseline performance
2. Compare embedding models (OpenAI vs nomic)
3. Compare reranking impact
4. Detect regressions after changes
5. Tune hyperparameters (K, N, thresholds)

Decision

Implement a multi-source test set combining:

1. Historical Query Labeling

• Sample real queries from production logs
• Manually label with correct chunk IDs and domain
• Focus on edge cases and ambiguous queries
• Target: ~50-100 labeled queries

2. Synthetic Q&A Generation

• Use LLM to generate questions from each chunk
• Question + source chunk = labeled test pair
• Ensures coverage across all domains and document types
• Target: ~100-200 synthetic pairs

3. LLM Auto-Labeling with Human Review

• Use LLM to judge relevance of retrieved chunks
• Flag low-confidence or disputed labels for human review
• Accelerates labeling while maintaining quality
• Target: ~50-100 auto-labeled with review

Metrics to Track

Metric	Definition	Target
Precision@5	Fraction of top 5 results that are relevant	> 0.80
Precision@10	Fraction of top 10 results that are relevant	> 0.70
MRR	Mean Reciprocal Rank of first relevant result	> 0.85
Recall@20	Fraction of relevant docs found in top 20	> 0.90
Latency p50	Median retrieval time	< 500ms
Latency p95	95th percentile retrieval time	< 1000ms
Domain Accuracy	Correct domain in top 5 results	> 0.95

Evaluation Pipeline

Test Query → Embed → Retrieve (K=20) → Rerank → Top N → Compare to Labels → Log Metrics

Comparison Modes

1. Online vs Offline — Compare OpenAI + Cohere vs Nomic + BGE
2. With vs Without Reranking — Measure reranking impact
3. Before vs After Changes — Detect regressions

Consequences

Positive

• Measurable improvement — quantifiable baseline and progress
• Model comparison — data-driven decision on embedding/reranking choices
• Regression detection — catch quality degradation early
• Hyperparameter tuning — optimize K, N, thresholds with data

Negative

• Upfront investment — labeling requires human effort
• Maintenance — test set needs updates as corpus grows
• Storage — need to store labels and evaluation results

Neutral

• Synthetic questions may not match real query distribution
• LLM labeling may have bias — human review essential for edge cases

Implementation Notes

• Store test set in JSON/Parquet: {query, relevant_chunk_ids, domain, source}
• Build evaluation script that runs retrieval and computes metrics
• Log results to a simple database or file for trend analysis
• Run evaluation on every significant change
• Consider integrating with LlamaIndex's evaluation module