chatbot_v3/docs/adr/0003-agent-hybrid-retrieval.md

0003. Agent hybrid retrieval (dense + dense + sparse + late-interaction rerank)

Status

Accepted

Context

The AI agent needs to retrieve the most relevant chunks for a query against the chunks collection defined in ADR-0001, which already provisions four named vectors — dense_nomic, dense_openai, sparse, and late_interaction — for exactly this purpose. The goal is higher relevance than single-dense-vector search by combining two independent semantic signals with lexical (sparse) retrieval, then refining the result with a late-interaction (ColBERT-style) rerank.

Qdrant's Query API supports doing this as a single server-side request via prefetch stages, fusion, and a final rerank stage — avoiding a client-side fan-out of multiple separate queries. It also explicitly cautions that hybrid search/reranking can mask underlying embedding-model quality issues rather than fix them, so it shouldn't be adopted without a baseline comparison.

Both dense models are now fixed: dense_nomic uses nomic-embed-text-v2-moe (0004), which requires a search_query: task prefix on the embedded query text (mirroring search_document: on the ingestion side); dense_openai uses the OpenAI large embedding model, no prefix convention needed. sparse uses bm25-fa-norm-stop (ADR-0001). The late-interaction/reranker model is now decided — jina-colbert-v2 — see 0005 for the model comparison, the GPU dependency it introduces, and its unresolved commercial-license status.

Decision

Query shape

A single Qdrant Query API request per agent query, structured as:

1. Prefetch 1 — dense ANN: search the dense_nomic vector, top ~100–200 candidates.
2. Prefetch 2 — dense ANN: search the dense_openai vector, top ~100–200 candidates.
3. Prefetch 3 — sparse ANN: search the sparse vector, top ~100–200 candidates.
4. Fusion: combine all three prefetch results via RRF (Reciprocal Rank Fusion) as the default — per Qdrant's own guidance, RRF "ignores score magnitude, a decent default to start with" and handles the incomparable score scales across two dense models and a sparse model.
5. Rerank: apply late-interaction (late_interaction multivector, max-sim) reranking over the fused top-N (not the full collection) to produce the final top-k returned to the agent, using jina-colbert-v2 (0005).

Every prefetch and the final query carries the same tenant_id/domain filter from ADR-0001's payload schema — identical isolation guarantee to ADR-0002.

Context-window expansion

For each chunk returned by the fused/reranked query, the agent may expand its context by pulling the immediately preceding and following chunks before building the prompt. This uses the previous_chunk_id/next_chunk_id pointer fields from ADR-0001 — a single batch "retrieve points by ID" call per result chunk, not an additional filtered search. Since this runs on every returned chunk, it relies on those pointers being kept correct by ADR-0002's CRUD mutation logic (reorder/insert/delete).

Bounding rerank cost

Late-interaction rerank is the most compute-expensive stage. It is applied only to the fusion's top-N output (e.g. top 50), never to the full candidate set or the full collection, to keep latency bounded.

Evaluate before enabling in production

Before turning this on for real traffic, run an evaluation comparing hybrid

• rerank against a dense-only baseline on representative queries. Qdrant's own guidance warns against adopting hybrid search prematurely, since it can paper over embedding-model quality problems instead of addressing them. If the baseline is already good, the added complexity/cost of this pipeline should be justified with evidence, not assumed.

Consequences

Positive

• Single round-trip server-side fusion + rerank — lower latency than orchestrating multiple queries and fusing client-side in FastAPI.
• Directly reuses the vector schema and tenant isolation already established in ADR-0001/0002 — no divergent data model for the agent path.
• Bounding rerank to the fused top-N keeps the expensive late-interaction step's cost predictable regardless of collection size.

Negative

• Late-interaction multivectors are storage/compute heavy; with jina-colbert-v2 chosen (0005) this stage now also requires GPU capacity (flash_attn/CUDA), not just the Docker deployment ADR-0001 assumed.
• RRF is score-magnitude-agnostic by design — if a use case later needs absolute score information (e.g. a relevance threshold), a different fusion method (DBSF or a custom FormulaQuery) would need to be evaluated separately.
• Adds a mandatory evaluation step before production rollout, rather than shipping hybrid search immediately.
• Three prefetch stages instead of two (two dense + one sparse) means every agent query embeds the question with both dense_nomic and dense_openai — an extra embedding call and an external OpenAI API dependency on the query's critical path, not just at ingestion time.

Alternatives Considered

• Client-side fan-out (separate dense and sparse queries against Qdrant, manual RRF fusion in FastAPI): rejected — more round trips and latency than Qdrant's native server-side Query API fusion, for no functional benefit.
• Cross-encoder reranking via FastEmbed instead of late-interaction: not adopted as the initial decision, since ADR-0001 already reserves a multivector field for late-interaction rerank; noted as a possible additional future stage rather than a replacement.
• DBSF or FormulaQuery as the default fusion method: rejected as the starting point — RRF is simpler and Qdrant's recommended default; DBSF/ FormulaQuery remain available as a later optimization if empirical evaluation shows RRF underperforming.