60 lines
2.3 KiB
Markdown
60 lines
2.3 KiB
Markdown
# ADR 0004: Reranking Strategy
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
The current system has no reranking step. With a single collection spanning all insurance domains, initial retrieval may return chunks from multiple domains with varying relevance. A reranker can reorder results by relevance, pushing the most domain-appropriate chunks to the top.
|
|
|
|
Reranking is critical for retrieval precision when:
|
|
- Query is ambiguous (could match multiple domains)
|
|
- Initial embedding search returns noisy results
|
|
- Cross-domain similarity causes false positives
|
|
|
|
## Decision
|
|
|
|
Add a **dual reranking layer** matching the dual embedding strategy:
|
|
|
|
| Model | Use Case | Type |
|
|
|-------|----------|------|
|
|
| Cohere Rerank | Primary when internet available | Cloud API |
|
|
| BGE Reranker (`BAAI/bge-reranker-base` or `bge-reranker-v2-m3`) | Fallback during blackouts | On-prem |
|
|
|
|
### Retrieval Pipeline (Updated)
|
|
|
|
1. **Query embedding** — OpenAI or nomic depending on connectivity
|
|
2. **Initial retrieval** — Top-K chunks (K=20-50) from Qdrant
|
|
3. **Reranking** — Score and reorder chunks by relevance
|
|
4. **Top-N selection** — Return top N (N=5-10) for generation
|
|
|
|
### Why BGE Reranker for Local
|
|
- `bge-reranker-v2-m3` supports multilingual (Persian/English)
|
|
- Cross-encoder architecture provides high accuracy
|
|
- Runs on CPU if needed (slower) or GPU for production speed
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
- **Higher precision** — reranker separates relevant from superficially similar
|
|
- **Domain disambiguation** — pushes correct domain to top even with noisy initial retrieval
|
|
- **Consistent architecture** — same dual-model pattern as embeddings and generation
|
|
- **Graceful degradation** — local reranker works during blackouts
|
|
|
|
### Negative
|
|
- **Added latency** — reranking adds 50-200ms depending on model and hardware
|
|
- **Additional complexity** — one more model to manage and monitor
|
|
- **GPU recommended** — BGE reranker is slow on CPU for large candidate sets
|
|
|
|
### Neutral
|
|
- Need to tune K (initial retrieval size) and N (final output size)
|
|
- May need different thresholds for online vs offline reranking quality
|
|
|
|
## Implementation Notes
|
|
|
|
- Use Cohere's `/rerank` API when available
|
|
- Use `FlagEmbedding` or `sentence-transformers` for BGE locally
|
|
- Consider `bge-reranker-v2-m3` for best multilingual support
|
|
- Start with K=20 initial candidates, N=5 final results; tune based on metrics
|