initial commit

docs/adr/0003-agent-hybrid-retrieval.md

@@ -0,0 +1,123 @@
+# 0003. Agent hybrid retrieval (dense + dense + sparse + late-interaction rerank)
+
+## Status
+
+Accepted
+
+## Context
+
+The AI agent needs to retrieve the most relevant chunks for a query against
+the `chunks` collection defined in ADR-0001, which already provisions four
+named vectors — `dense_nomic`, `dense_openai`, `sparse`, and
+`late_interaction` — for exactly this purpose. The goal is higher relevance
+than single-dense-vector search by combining two independent semantic
+signals with lexical (sparse) retrieval, then refining the result with a
+late-interaction (ColBERT-style) rerank.
+
+Qdrant's Query API supports doing this as a single server-side request via
+`prefetch` stages, fusion, and a final rerank stage — avoiding a client-side
+fan-out of multiple separate queries. It also explicitly cautions that
+hybrid search/reranking can *mask* underlying embedding-model quality issues
+rather than fix them, so it shouldn't be adopted without a baseline
+comparison.
+
+Both dense models are now fixed: `dense_nomic` uses `nomic-embed-text-v2-moe`
+([0004](0004-docx-csv-chunking-strategy.md)), which requires a
+`search_query: ` task prefix on the embedded query text (mirroring
+`search_document: ` on the ingestion side); `dense_openai` uses the OpenAI
+large embedding model, no prefix convention needed. `sparse` uses
+`bm25-fa-norm-stop` (ADR-0001). The late-interaction/reranker model is now
+decided — `jina-colbert-v2` — see [0005](0005-reranking-model-and-sparse-analyzer-selection.md)
+for the model comparison, the GPU dependency it introduces, and its
+unresolved commercial-license status.
+
+## Decision
+
+### Query shape
+
+A single Qdrant Query API request per agent query, structured as:
+
+1. **Prefetch 1 — dense ANN**: search the `dense_nomic` vector, top ~100–200
+   candidates.
+2. **Prefetch 2 — dense ANN**: search the `dense_openai` vector, top ~100–200
+   candidates.
+3. **Prefetch 3 — sparse ANN**: search the `sparse` vector, top ~100–200
+   candidates.
+4. **Fusion**: combine all three prefetch results via **RRF** (Reciprocal
+   Rank Fusion) as the default — per Qdrant's own guidance, RRF "ignores
+   score magnitude, a decent default to start with" and handles the
+   incomparable score scales across two dense models and a sparse model.
+5. **Rerank**: apply late-interaction (`late_interaction` multivector,
+   max-sim) reranking over the fused top-N (not the full collection) to
+   produce the final top-k returned to the agent, using `jina-colbert-v2`
+   ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)).
+
+Every prefetch and the final query carries the same `tenant_id`/`domain`
+filter from ADR-0001's payload schema — identical isolation guarantee to
+ADR-0002.
+
+### Context-window expansion
+
+For each chunk returned by the fused/reranked query, the agent may expand
+its context by pulling the immediately preceding and following chunks before
+building the prompt. This uses the `previous_chunk_id`/`next_chunk_id`
+pointer fields from ADR-0001 — a single batch "retrieve points by ID" call
+per result chunk, not an additional filtered search. Since this runs on every
+returned chunk, it relies on those pointers being kept correct by ADR-0002's
+CRUD mutation logic (reorder/insert/delete).
+
+### Bounding rerank cost
+
+Late-interaction rerank is the most compute-expensive stage. It is applied
+only to the fusion's top-N output (e.g. top 50), never to the full
+candidate set or the full collection, to keep latency bounded.
+
+### Evaluate before enabling in production
+
+Before turning this on for real traffic, run an evaluation comparing hybrid
++ rerank against a dense-only baseline on representative queries. Qdrant's
+own guidance warns against adopting hybrid search prematurely, since it can
+paper over embedding-model quality problems instead of addressing them. If
+the baseline is already good, the added complexity/cost of this pipeline
+should be justified with evidence, not assumed.
+
+## Consequences
+
+### Positive
+- Single round-trip server-side fusion + rerank — lower latency than
+  orchestrating multiple queries and fusing client-side in FastAPI.
+- Directly reuses the vector schema and tenant isolation already established
+  in ADR-0001/0002 — no divergent data model for the agent path.
+- Bounding rerank to the fused top-N keeps the expensive late-interaction
+  step's cost predictable regardless of collection size.
+
+### Negative
+- Late-interaction multivectors are storage/compute heavy; with
+  `jina-colbert-v2` chosen ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
+  this stage now also requires GPU capacity (`flash_attn`/CUDA), not just
+  the Docker deployment ADR-0001 assumed.
+- RRF is score-magnitude-agnostic by design — if a use case later needs
+  absolute score information (e.g. a relevance threshold), a different
+  fusion method (DBSF or a custom `FormulaQuery`) would need to be
+  evaluated separately.
+- Adds a mandatory evaluation step before production rollout, rather than
+  shipping hybrid search immediately.
+- Three prefetch stages instead of two (two dense + one sparse) means every
+  agent query embeds the question with both `dense_nomic` and `dense_openai`
+  — an extra embedding call and an external OpenAI API dependency on the
+  query's critical path, not just at ingestion time.
+
+## Alternatives Considered
+
+- **Client-side fan-out** (separate dense and sparse queries against Qdrant,
+  manual RRF fusion in FastAPI): rejected — more round trips and latency
+  than Qdrant's native server-side Query API fusion, for no functional
+  benefit.
+- **Cross-encoder reranking via FastEmbed instead of late-interaction**: not
+  adopted as the initial decision, since ADR-0001 already reserves a
+  multivector field for late-interaction rerank; noted as a possible
+  *additional* future stage rather than a replacement.
+- **DBSF or FormulaQuery as the default fusion method**: rejected as the
+  starting point — RRF is simpler and Qdrant's recommended default; DBSF/
+  FormulaQuery remain available as a later optimization if empirical
+  evaluation shows RRF underperforming.