Research/chunking/docs/adr/0004-reranking-strategy.md

ADR 0004: Reranking Strategy

Status

Accepted

Context

The current system has no reranking step. With a single collection spanning all insurance domains, initial retrieval may return chunks from multiple domains with varying relevance. A reranker can reorder results by relevance, pushing the most domain-appropriate chunks to the top.

Reranking is critical for retrieval precision when:

• Query is ambiguous (could match multiple domains)
• Initial embedding search returns noisy results
• Cross-domain similarity causes false positives

Decision

Add a dual reranking layer matching the dual embedding strategy:

Model	Use Case	Type
Cohere Rerank	Primary when internet available	Cloud API
BGE Reranker (BAAI/bge-reranker-base or bge-reranker-v2-m3)	Fallback during blackouts	On-prem

Retrieval Pipeline (Updated)

1. Query embedding — OpenAI or nomic depending on connectivity
2. Initial retrieval — Top-K chunks (K=20-50) from Qdrant
3. Reranking — Score and reorder chunks by relevance
4. Top-N selection — Return top N (N=5-10) for generation

Why BGE Reranker for Local

• bge-reranker-v2-m3 supports multilingual (Persian/English)
• Cross-encoder architecture provides high accuracy
• Runs on CPU if needed (slower) or GPU for production speed

Consequences

Positive

• Higher precision — reranker separates relevant from superficially similar
• Domain disambiguation — pushes correct domain to top even with noisy initial retrieval
• Consistent architecture — same dual-model pattern as embeddings and generation
• Graceful degradation — local reranker works during blackouts

Negative

• Added latency — reranking adds 50-200ms depending on model and hardware
• Additional complexity — one more model to manage and monitor
• GPU recommended — BGE reranker is slow on CPU for large candidate sets

Neutral

• Need to tune K (initial retrieval size) and N (final output size)
• May need different thresholds for online vs offline reranking quality

Implementation Notes

• Use Cohere's /rerank API when available
• Use FlagEmbedding or sentence-transformers for BGE locally
• Consider bge-reranker-v2-m3 for best multilingual support
• Start with K=20 initial candidates, N=5 final results; tune based on metrics