# ADR 0004: Reranking Strategy

## Status

Accepted

## Context

The current system has no reranking step. With a single collection spanning all insurance domains, initial retrieval may return chunks from multiple domains with varying relevance. A reranker can reorder results by relevance, pushing the most domain-appropriate chunks to the top.

Reranking is critical for retrieval precision when:
- Query is ambiguous (could match multiple domains)
- Initial embedding search returns noisy results
- Cross-domain similarity causes false positives

## Decision

Add a **dual reranking layer** matching the dual embedding strategy:

| Model | Use Case | Type |
|-------|----------|------|
| Cohere Rerank | Primary when internet available | Cloud API |
| BGE Reranker (`BAAI/bge-reranker-base` or `bge-reranker-v2-m3`) | Fallback during blackouts | On-prem |

### Retrieval Pipeline (Updated)

1. **Query embedding** — OpenAI or nomic depending on connectivity
2. **Initial retrieval** — Top-K chunks (K=20-50) from Qdrant
3. **Reranking** — Score and reorder chunks by relevance
4. **Top-N selection** — Return top N (N=5-10) for generation

### Why BGE Reranker for Local
- `bge-reranker-v2-m3` supports multilingual (Persian/English)
- Cross-encoder architecture provides high accuracy
- Runs on CPU if needed (slower) or GPU for production speed

## Consequences

### Positive
- **Higher precision** — reranker separates relevant from superficially similar
- **Domain disambiguation** — pushes correct domain to top even with noisy initial retrieval
- **Consistent architecture** — same dual-model pattern as embeddings and generation
- **Graceful degradation** — local reranker works during blackouts

### Negative
- **Added latency** — reranking adds 50-200ms depending on model and hardware
- **Additional complexity** — one more model to manage and monitor
- **GPU recommended** — BGE reranker is slow on CPU for large candidate sets

### Neutral
- Need to tune K (initial retrieval size) and N (final output size)
- May need different thresholds for online vs offline reranking quality

## Implementation Notes

- Use Cohere's `/rerank` API when available
- Use `FlagEmbedding` or `sentence-transformers` for BGE locally
- Consider `bge-reranker-v2-m3` for best multilingual support
- Start with K=20 initial candidates, N=5 final results; tune based on metrics