# 0003. Agent hybrid retrieval (dense + dense + sparse + late-interaction rerank)

## Status

Accepted

## Context

The AI agent needs to retrieve the most relevant chunks for a query against
the `chunks` collection defined in ADR-0001, which already provisions four
named vectors — `dense_nomic`, `dense_openai`, `sparse`, and
`late_interaction` — for exactly this purpose. The goal is higher relevance
than single-dense-vector search by combining two independent semantic
signals with lexical (sparse) retrieval, then refining the result with a
late-interaction (ColBERT-style) rerank.

Qdrant's Query API supports doing this as a single server-side request via
`prefetch` stages, fusion, and a final rerank stage — avoiding a client-side
fan-out of multiple separate queries. It also explicitly cautions that
hybrid search/reranking can *mask* underlying embedding-model quality issues
rather than fix them, so it shouldn't be adopted without a baseline
comparison.

Both dense models are now fixed: `dense_nomic` uses `nomic-embed-text-v2-moe`
([0004](0004-docx-csv-chunking-strategy.md)), which requires a
`search_query: ` task prefix on the embedded query text (mirroring
`search_document: ` on the ingestion side); `dense_openai` uses the OpenAI
large embedding model, no prefix convention needed. `sparse` uses
`bm25-fa-norm-stop` (ADR-0001). The late-interaction/reranker model is now
decided — `jina-colbert-v2` — see [0005](0005-reranking-model-and-sparse-analyzer-selection.md)
for the model comparison, the GPU dependency it introduces, and its
unresolved commercial-license status.

## Decision

### Query shape

A single Qdrant Query API request per agent query, structured as:

1. **Prefetch 1 — dense ANN**: search the `dense_nomic` vector, top ~100–200
   candidates.
2. **Prefetch 2 — dense ANN**: search the `dense_openai` vector, top ~100–200
   candidates.
3. **Prefetch 3 — sparse ANN**: search the `sparse` vector, top ~100–200
   candidates.
4. **Fusion**: combine all three prefetch results via **RRF** (Reciprocal
   Rank Fusion) as the default — per Qdrant's own guidance, RRF "ignores
   score magnitude, a decent default to start with" and handles the
   incomparable score scales across two dense models and a sparse model.
5. **Rerank**: apply late-interaction (`late_interaction` multivector,
   max-sim) reranking over the fused top-N (not the full collection) to
   produce the final top-k returned to the agent, using `jina-colbert-v2`
   ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)).

Every prefetch and the final query carries the same `tenant_id`/`domain`
filter from ADR-0001's payload schema — identical isolation guarantee to
ADR-0002.

### Context-window expansion

For each chunk returned by the fused/reranked query, the agent may expand
its context by pulling the immediately preceding and following chunks before
building the prompt. This uses the `previous_chunk_id`/`next_chunk_id`
pointer fields from ADR-0001 — a single batch "retrieve points by ID" call
per result chunk, not an additional filtered search. Since this runs on every
returned chunk, it relies on those pointers being kept correct by ADR-0002's
CRUD mutation logic (reorder/insert/delete).

### Bounding rerank cost

Late-interaction rerank is the most compute-expensive stage. It is applied
only to the fusion's top-N output (e.g. top 50), never to the full
candidate set or the full collection, to keep latency bounded.

### Evaluate before enabling in production

Before turning this on for real traffic, run an evaluation comparing hybrid
+ rerank against a dense-only baseline on representative queries. Qdrant's
own guidance warns against adopting hybrid search prematurely, since it can
paper over embedding-model quality problems instead of addressing them. If
the baseline is already good, the added complexity/cost of this pipeline
should be justified with evidence, not assumed.

## Consequences

### Positive
- Single round-trip server-side fusion + rerank — lower latency than
  orchestrating multiple queries and fusing client-side in FastAPI.
- Directly reuses the vector schema and tenant isolation already established
  in ADR-0001/0002 — no divergent data model for the agent path.
- Bounding rerank to the fused top-N keeps the expensive late-interaction
  step's cost predictable regardless of collection size.

### Negative
- Late-interaction multivectors are storage/compute heavy; with
  `jina-colbert-v2` chosen ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
  this stage now also requires GPU capacity (`flash_attn`/CUDA), not just
  the Docker deployment ADR-0001 assumed.
- RRF is score-magnitude-agnostic by design — if a use case later needs
  absolute score information (e.g. a relevance threshold), a different
  fusion method (DBSF or a custom `FormulaQuery`) would need to be
  evaluated separately.
- Adds a mandatory evaluation step before production rollout, rather than
  shipping hybrid search immediately.
- Three prefetch stages instead of two (two dense + one sparse) means every
  agent query embeds the question with both `dense_nomic` and `dense_openai`
  — an extra embedding call and an external OpenAI API dependency on the
  query's critical path, not just at ingestion time.

## Alternatives Considered

- **Client-side fan-out** (separate dense and sparse queries against Qdrant,
  manual RRF fusion in FastAPI): rejected — more round trips and latency
  than Qdrant's native server-side Query API fusion, for no functional
  benefit.
- **Cross-encoder reranking via FastEmbed instead of late-interaction**: not
  adopted as the initial decision, since ADR-0001 already reserves a
  multivector field for late-interaction rerank; noted as a possible
  *additional* future stage rather than a replacement.
- **DBSF or FormulaQuery as the default fusion method**: rejected as the
  starting point — RRF is simpler and Qdrant's recommended default; DBSF/
  FormulaQuery remain available as a later optimization if empirical
  evaluation shows RRF underperforming.