docs(rag): add domain model and original research input

chunking/docs/adr/0005-retrieval-evaluation-framework.md

@@ -0,0 +1,86 @@
+# ADR 0005: Retrieval Evaluation Framework
+
+## Status
+
+Accepted
+
+## Context
+
+The current system has no metrics for retrieval quality. The previous classification accuracy was unknown, making improvement impossible. The new architecture needs measurement from day one to:
+
+1. Establish baseline performance
+2. Compare embedding models (OpenAI vs nomic)
+3. Compare reranking impact
+4. Detect regressions after changes
+5. Tune hyperparameters (K, N, thresholds)
+
+## Decision
+
+Implement a **multi-source test set** combining:
+
+### 1. Historical Query Labeling
+- Sample real queries from production logs
+- Manually label with correct chunk IDs and domain
+- Focus on edge cases and ambiguous queries
+- Target: ~50-100 labeled queries
+
+### 2. Synthetic Q&A Generation
+- Use LLM to generate questions from each chunk
+- Question + source chunk = labeled test pair
+- Ensures coverage across all domains and document types
+- Target: ~100-200 synthetic pairs
+
+### 3. LLM Auto-Labeling with Human Review
+- Use LLM to judge relevance of retrieved chunks
+- Flag low-confidence or disputed labels for human review
+- Accelerates labeling while maintaining quality
+- Target: ~50-100 auto-labeled with review
+
+### Metrics to Track
+
+| Metric | Definition | Target |
+|--------|------------|--------|
+| Precision@5 | Fraction of top 5 results that are relevant | > 0.80 |
+| Precision@10 | Fraction of top 10 results that are relevant | > 0.70 |
+| MRR | Mean Reciprocal Rank of first relevant result | > 0.85 |
+| Recall@20 | Fraction of relevant docs found in top 20 | > 0.90 |
+| Latency p50 | Median retrieval time | < 500ms |
+| Latency p95 | 95th percentile retrieval time | < 1000ms |
+| Domain Accuracy | Correct domain in top 5 results | > 0.95 |
+
+### Evaluation Pipeline
+
+```
+Test Query → Embed → Retrieve (K=20) → Rerank → Top N → Compare to Labels → Log Metrics
+```
+
+### Comparison Modes
+
+1. **Online vs Offline** — Compare OpenAI + Cohere vs Nomic + BGE
+2. **With vs Without Reranking** — Measure reranking impact
+3. **Before vs After Changes** — Detect regressions
+
+## Consequences
+
+### Positive
+- **Measurable improvement** — quantifiable baseline and progress
+- **Model comparison** — data-driven decision on embedding/reranking choices
+- **Regression detection** — catch quality degradation early
+- **Hyperparameter tuning** — optimize K, N, thresholds with data
+
+### Negative
+- **Upfront investment** — labeling requires human effort
+- **Maintenance** — test set needs updates as corpus grows
+- **Storage** — need to store labels and evaluation results
+
+### Neutral
+- Synthetic questions may not match real query distribution
+- LLM labeling may have bias — human review essential for edge cases
+
+## Implementation Notes
+
+- Store test set in JSON/Parquet: `{query, relevant_chunk_ids, domain, source}`
+- Build evaluation script that runs retrieval and computes metrics
+- Log results to a simple database or file for trend analysis
+- Run evaluation on every significant change
+- Consider integrating with LlamaIndex's evaluation module