initial commit

docs/adr/0000-adr-template.md

@@ -0,0 +1,58 @@
+# 0000. ADR process and template
+
+## Status
+
+Accepted
+
+## Context
+
+We are about to make several architecture decisions about how the Qdrant
+vector store is structured and used (ingestion, CRUD, and agent retrieval).
+`docs/adr/README.md` currently has no convention for recording these
+decisions. We need a lightweight, consistent format so future contributors
+can understand *why* the system is built the way it is, not just *what* it
+does.
+
+## Decision
+
+We will record architecturally significant decisions as Architecture Decision
+Records (ADRs) under `docs/adr/`, using this convention:
+
+- **Filename**: `NNNN-short-title.md`, zero-padded, monotonically increasing
+  (`0000`, `0001`, `0002`, ...). The number is permanent once assigned.
+- **Sections**, in this order:
+  1. Title (`# NNNN. Title`)
+  2. Status — one of `Proposed`, `Accepted`, `Superseded by NNNN`
+  3. Context — the problem, constraints, and forces at play
+  4. Decision — what we're doing, stated directly
+  5. Consequences — `Positive` and `Negative` subsections
+  6. Alternatives Considered — options we rejected and why
+- **Immutability**: once an ADR is `Accepted`, it is not edited to reflect a
+  changed decision. A changed decision gets a *new* ADR that sets the old
+  one's status to `Superseded by NNNN` and links back to it. Small
+  clarifications/typo fixes are fine to edit in place.
+- **Scope**: only decisions with real architectural weight (data model,
+  external system boundaries, protocols, cross-cutting concerns) get an ADR.
+  Routine implementation choices do not.
+
+## Consequences
+
+### Positive
+- Future contributors (and agents) can trace *why* the Qdrant schema, CRUD
+  API, and retrieval pipeline look the way they do without re-deriving it
+  from code.
+- Decisions that turn out wrong are corrected via a visible trail
+  (superseding ADRs) instead of silently rewritten history.
+
+### Negative
+- Adds minor overhead — anyone making a significant architectural change must
+  also write or update an ADR.
+
+## Alternatives Considered
+
+- **No formal record, rely on commit messages/PR descriptions**: rejected —
+  these are hard to discover later and don't survive squash-merges or
+  history rewrites.
+- **Full MADR template (with explicit "Decision Drivers" and scored option
+  comparison tables)**: rejected as too heavyweight for this project's current
+  size; we can adopt more structure later if needed via a new ADR.

docs/adr/0001-ingestion-pipeline-and-collection-schema.md

@@ -0,0 +1,237 @@
+# 0001. Ingestion pipeline and Qdrant collection schema
+
+## Status
+
+Accepted
+
+## Context
+
+FastAPI needs to accept uploaded documents (`.docx`, `.csv`), preprocess them,
+split them into chunks, embed the chunks, and store them in Qdrant. This
+collection is shared by two other pipelines that will be defined in later
+ADRs: direct CRUD/search over chunks ([0002](0002-chunk-crud-and-search-api.md))
+and hybrid retrieval for an AI agent
+([0003](0003-agent-hybrid-retrieval.md)). The schema decided here — vector
+configuration and payload fields — is load-bearing for both, so it needs to
+be right before either is built on top of it.
+
+Two Qdrant behaviors make this schema decision urgent rather than deferrable:
+
+- **Sparse vectors cannot be added to an existing unnamed-dense-vector
+  collection** — doing so requires recreating the collection. If we start
+  dense-only and add sparse/late-interaction vectors later, that's a
+  disruptive migration.
+- **Co-locating large late-interaction (multivector) representations with
+  dense vectors in the same segment degrades *all* queries**, including
+  plain dense search, unless the multivector is stored on disk.
+
+We've decided (see project discussion) that:
+- Qdrant runs self-hosted via Docker.
+- Tenancy is a **single shared collection**, partitioned by payload, not
+  collection-per-tenant — Qdrant's own multitenancy guidance is explicit that
+  per-tenant collections "do not scale past a few hundred and waste
+  resources," while payload partitioning scales to 10k+ tenants.
+- The dense embedding model is now decided — `nomic-embed-text-v2-moe`, see
+  [0004](0004-docx-csv-chunking-strategy.md). Sparse and late-interaction
+  models are **not yet chosen**; those remain swappable via FastEmbed-
+  compatible config.
+- The exact payload field list beyond the baseline below is still open for
+  discussion; this ADR proposes a starting schema, not a final one.
+
+## Decision
+
+### Collection
+
+One collection, e.g. `chunks`, shared by all tenants and domains.
+
+### Vectors (named vectors, defined at creation time)
+
+| Name | Type | Purpose | Notes |
+|---|---|---|---|
+| `dense_nomic` | dense vector | primary semantic similarity (multilingual, incl. Persian) | `nomic-embed-text-v2-moe`, 768-dim ([0004](0004-docx-csv-chunking-strategy.md)) |
+| `dense_openai` | dense vector | second semantic signal | OpenAI large embedding model (e.g. `text-embedding-3-large`), dimension per OpenAI's `dimensions` param (TBD — full 3072 vs. a truncated size) |
+| `sparse` | sparse vector | lexical/keyword-sensitive retrieval | `bm25-fa-norm-stop` — Qdrant FastEmbed's BM25 sparse encoder configured for Persian (stopword removal + normalization), not a separately trained model |
+| `late_interaction` | multivector | reserved for late-interaction rerank ([0003](0003-agent-hybrid-retrieval.md)) | `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)), `comparator: max_sim`, `hnsw_config: m=0` (rerank-only, never independently ANN-searched), stored **on disk** |
+
+All four are defined **now**, even though `late_interaction` won't be
+populated until the agent retrieval work in ADR-0003 lands, specifically to
+avoid the forced-recreation problem described above. `late_interaction` is
+configured with on-disk storage so its larger footprint doesn't degrade the
+dense/sparse query latency. Two dense vectors are provisioned deliberately —
+`dense_nomic` and `dense_openai` are two independent semantic signals, both
+prefetched and fused at query time (ADR-0003), not a primary/fallback pair.
+
+### Multitenancy / indexing config
+
+- HNSW: `m: 0` (disable the global index) + `payload_m: 16`, per Qdrant's
+  multitenant collection guidance.
+- Payload index on `tenant_id`: keyword index with `is_tenant: true` — this
+  co-locates a tenant's vectors on disk for sequential reads.
+- Payload index on `domain`: keyword index (secondary partition dimension
+  within a tenant).
+- Payload index on `file_id`: keyword index, used by CRUD lookups in
+  ADR-0002 (e.g. "delete all chunks belonging to this file").
+- Payload index on `order_id`: **float** index (Qdrant's `order_by` and
+  `Range` filter conditions only support numeric/datetime payloads, not
+  keyword/string — see the `order_id` format note below). Used for
+  `order_by` when listing/scrolling a file's chunks in sequence (ADR-0002)
+  and for previous/next lookups.
+- Payload index on `previous_chunk_id` / `next_chunk_id`: keyword index,
+  used for O(1) adjacency retrieval (see below).
+
+### Payload schema
+
+This schema is now decided for the fields below. Additional document-context
+fields (e.g. page/row position, section heading, effective/expiration dates
+for insurance policy documents) are deliberately **deferred to a future ADR**
+that will accompany the docx/csv chunking-strategy work — that decision
+involves format-specific tradeoffs not yet made.
+
+| Field | Type | Purpose |
+|---|---|---|
+| `tenant_id` | keyword, `is_tenant: true` index | tenant isolation |
+| `domain` | keyword | logical partition within a tenant (e.g. `fire`, `car` for insurance lines) |
+| `file_id` | keyword | groups chunks back to their source file (formerly referred to as `document_id` — standardized on `file_id`) |
+| `chunk_id` | keyword | stable identifier for a single chunk |
+| `content_type` | keyword | classification of the chunk's content; exact value set (e.g. `paragraph`, `table_row`, `heading`) to be finalized alongside the chunking-strategy ADR |
+| `source_filename` | keyword | original uploaded filename |
+| `source_type` | keyword (`docx` \| `csv`) | which parser produced this chunk |
+| `order_id` | float (see below) | chunk's *display* position within the file; mutable so the backend can reorder/insert chunks |
+| `chunk_index` | integer | chunk's *original ingestion* ordinal — immutable, used to derive the deterministic point ID below (kept separate from `order_id` precisely because `order_id` can change) |
+| `previous_chunk_id` | keyword, nullable | `chunk_id` of the preceding chunk in display order (`null` for the first chunk in a file) — O(1) adjacency pointer for context-window expansion in ADR-0003 |
+| `next_chunk_id` | keyword, nullable | `chunk_id` of the following chunk in display order (`null` for the last chunk in a file) — same purpose as `previous_chunk_id` |
+| `content` | text (full-text indexed) | the chunk text itself, also used for keyword search in ADR-0002 |
+| `is_active` | boolean | soft-delete / visibility flag — inactive chunks are excluded from CRUD listing and agent retrieval but retained for audit |
+| `deleted_at` | datetime, nullable | set when `is_active` transitions to `false`; distinguishes "deactivated" from "never active" |
+| `created_at` | datetime | ingestion timestamp |
+| `updated_at` | datetime | last modification timestamp |
+| `created_by` | keyword | user/service that created the chunk |
+| `updated_by` | keyword | user/service that last modified the chunk |
+| `version` | integer | optimistic-concurrency counter, used in ADR-0002 |
+| `content_hash` | keyword | hash of the chunk's raw text; lets re-ingestion detect unchanged content and skip re-embedding it |
+| `embedding_model_version` | keyword | identifies which embedding model(s) produced this chunk's vectors; needed to know which chunks require re-embedding after a future model swap |
+
+#### `order_id` format
+
+`order_id` is a **fractional float key** (e.g. `1.0`, `2.0`, `3.0`, ...),
+not a plain sequential integer and not a lexicographic string. We are not
+implementing chunk insertion/reordering yet, but choosing this format now
+means that when that feature is added, inserting a chunk between two
+existing ones (e.g. assigning it `1.5`, then `1.25` for a subsequent insert
+in the same gap) only touches that one chunk's payload — it never requires
+renumbering every subsequent chunk, the same benefit a sortable string would
+give.
+
+A string key was considered first but **does not work in Qdrant**: the
+`Range` filter condition (`gt`/`gte`/`lt`/`lte`) only supports float/integer
+payloads (datetime gets its own separate `DatetimeRange` condition), and
+`order_by` on scroll requires a payload index that supports `Range`
+filtering — so `order_by` is likewise limited to numeric/datetime fields.
+A keyword/string `order_id` could only be sorted client-side after fetching
+every chunk for a file, and couldn't support a targeted previous/next query
+at all. A float key gets native `Range`/`order_by` support instead.
+
+**Known limitation**: repeatedly inserting into the exact same gap (~50+
+times between the same two neighbors) runs into floating-point precision
+limits. Mitigate with an occasional rebalance job that respaces a file's
+`order_id` values (e.g. back to `1000, 2000, 3000, ...`); this is standard
+for any fractional-indexing scheme and not expected to be hit in normal use.
+
+#### Previous/next chunk retrieval
+
+Two ways to get a chunk's neighbors, both viable given this schema:
+
+1. **Pointer fields (primary, recommended for the agent path)**: read
+   `previous_chunk_id`/`next_chunk_id` off the retrieved chunk and fetch
+   those chunk IDs in a single batch "retrieve points by ID" call. This is
+   the intended pattern for ADR-0003's context-window expansion, since it
+   runs on every retrieved chunk and a single batch-get is cheaper than a
+   filtered query per chunk.
+2. **Range query (fallback, only viable because `order_id` is numeric)**:
+   filter `file_id = X` + `order_id < current` (Range `lt`) + `order_by desc`
+   + `limit 1` for the previous chunk; mirror with `gt`/ascending for the
+   next chunk. Useful if the pointer fields are ever missing/stale, or for
+   ad-hoc debugging.
+
+Pointer fields are only as correct as the mutation logic that maintains
+them — see ADR-0002 for how reorder/insert/delete operations keep
+`previous_chunk_id`/`next_chunk_id` in sync.
+
+### Ingestion flow
+
+1. FastAPI upload endpoint receives a `.docx` or `.csv` file.
+2. Parse: `python-docx` for `.docx`, `pandas` for `.csv`.
+3. Preprocess: clean/normalize extracted text.
+4. Chunk: split into chunks (size/overlap are tunable config, not fixed by
+   this ADR).
+5. Embed each chunk into all four vectors: `dense_nomic`, `dense_openai`,
+   `sparse`, and `late_interaction`. `dense_nomic` uses
+   `nomic-embed-text-v2-moe` with the chunk text prefixed
+   `search_document: ` ([0004](0004-docx-csv-chunking-strategy.md));
+   `dense_openai` uses the OpenAI large embedding model; `sparse` uses
+   `bm25-fa-norm-stop`; `late_interaction` uses `jina-colbert-v2`
+   ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)) — its
+   document-side multivector is computed and stored at ingestion time here,
+   while the query-side multivector is computed per-query in ADR-0003 for
+   the MAX_SIM rerank comparison.
+6. Assign a **deterministic point ID** — UUIDv5 derived from
+   `file_id` + `chunk_index` (the immutable ingestion ordinal, not the
+   mutable `order_id`) — so re-ingesting the same file upserts existing
+   chunks instead of creating duplicates, and re-ordering chunks later never
+   changes their IDs. Initialize `order_id` from `chunk_index` at ingestion
+   time (e.g. `chunk_index` 0, 1, 2 → `order_id` `1.0`, `2.0`, `3.0`), and set
+   `previous_chunk_id`/`next_chunk_id` to each chunk's immediate ingestion-order
+   neighbor (`null` at the two ends of the file).
+7. Batch upsert into Qdrant: 64–256 points per request, 2–4 parallel upload
+   streams, per Qdrant's bulk-upload guidance.
+
+## Consequences
+
+### Positive
+- Defining all four named vectors up front avoids a future forced
+  collection recreation when late-interaction rerank is added in ADR-0003.
+- Payload-partitioned multitenancy scales to large tenant counts without
+  per-tenant collection sprawl, and shares infrastructure/config across
+  ADR-0002 and ADR-0003.
+- Deterministic point IDs make ingestion idempotent — safe to re-run on the
+  same document.
+- Numeric `order_id` is natively supported by Qdrant's `Range`/`order_by`,
+  enabling both ordered listing (ADR-0002) and previous/next chunk lookups
+  (ADR-0003's context-window expansion) without client-side sorting.
+
+### Negative
+- The collection carries four named vectors and every chunk is embedded
+  into all of them at ingestion (including `late_interaction` via
+  `jina-colbert-v2`), rather than only the vectors actually queried at
+  launch — more ingestion-time compute than a leaner initial cut.
+- Document-context payload fields (page/row position, section heading,
+  effective/expiration dates, etc.) are still deferred to the
+  chunking-strategy ADR; adding them later means an additive payload
+  migration, though it won't touch the fields already decided here.
+- Two dense vectors means every chunk is embedded twice (nomic + OpenAI) at
+  ingestion time and both are queried at retrieval time — roughly double
+  the dense embedding cost/latency of a single-dense-vector design, plus an
+  external network dependency on OpenAI's API in the ingestion path.
+- `dense_openai`'s exact output dimension is still an open dependency that
+  should be pinned before ingestion is implemented — changing it later is a
+  re-embedding migration, not a config tweak.
+- `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
+  adds a hard GPU dependency to ingestion (not just query time, since the
+  document-side multivector is computed here) and its commercial license is
+  still unconfirmed.
+- `previous_chunk_id`/`next_chunk_id` are denormalized pointers — every
+  reorder/insert/delete must update the affected neighbors' payloads too
+  (see ADR-0002), or the pointers go stale. Fractional float `order_id` also
+  needs an (infrequent) rebalance job as a long-term maintenance task.
+
+## Alternatives Considered
+
+- **Collection per domain**: rejected. Domains would need materially
+  different vector configs or strict data-boundary requirements to justify
+  this; neither applies here, and it fragments tenant-scaling benefits.
+- **Collection per tenant**: rejected outright per Qdrant's own guidance —
+  doesn't scale past a few hundred tenants.
+- **Start dense-only, add sparse/late-interaction later**: rejected — Qdrant
+  requires recreating the collection to add sparse vectors to an unnamed
+  dense-only collection, which is a disruptive migration we can avoid by
+  deciding the full vector shape now.

docs/adr/0002-chunk-crud-and-search-api.md

@@ -0,0 +1,129 @@
+# 0002. Chunk CRUD and payload/keyword search API
+
+## Status
+
+Accepted
+
+## Context
+
+Beyond bulk ingestion ([0001](0001-ingestion-pipeline-and-collection-schema.md)),
+we need FastAPI endpoints for direct, fine-grained management of individual
+chunks: create a single chunk, update a chunk's content/vectors/payload,
+delete chunks (by ID or in bulk by filter), and list/search chunks by keyword
+and payload metadata (tenant, domain, file, etc.) — independent of the
+semantic hybrid retrieval used by the AI agent (ADR-0003).
+
+This pipeline operates on the same collection and payload schema defined in
+ADR-0001, and Qdrant's Points API already provides the primitives needed —
+there's no need for a custom data layer on top.
+
+## Decision
+
+Expose a thin FastAPI layer directly over Qdrant's Points API, operating on
+the `chunks` collection and payload schema from ADR-0001:
+
+| Operation | FastAPI endpoint (indicative) | Qdrant primitive |
+|---|---|---|
+| Create a chunk | `POST /chunks` | upsert (single point) |
+| Update a chunk's vectors/content | `PUT /chunks/{chunk_id}` | upsert (`update_only` mode) |
+| Partially update payload | `PATCH /chunks/{chunk_id}/payload` | `set_payload` / `overwrite_payload` |
+| Delete one chunk | `DELETE /chunks/{chunk_id}` | delete by ID |
+| Delete many chunks | `DELETE /chunks?file_id=...` | delete by filter |
+| List/paginate chunks, in order | `GET /chunks?file_id=...` | `scroll` with filter + pagination, `order_by: order_id` |
+| Reorder/insert a chunk | `PATCH /chunks/{chunk_id}/order` | `set_payload` on `order_id` only |
+| Count chunks | `GET /chunks/count` | `count` |
+| Keyword search | `GET /chunks/search?q=...` | full-text payload index on `content` (match, not semantic) |
+| Bulk multi-op edits | `POST /chunks/batch` | Qdrant `points/batch` |
+
+### Ordering and reordering
+
+Chunks belonging to a `file_id` are listed via `scroll` ordered by the
+mutable `order_id` payload field (ADR-0001) — this is what lets the frontend
+display and let a backend user modify chunk order. Because `order_id` is a
+fractional float key rather than a sequential integer position, moving or
+inserting a chunk only requires assigning it a new value between its two new
+neighbors — no renumbering of siblings, and no effect on the chunk's stable
+point ID (which is derived from the immutable `chunk_index`, not `order_id`).
+
+`PATCH /chunks/{chunk_id}/order` does more than set one field, though: since
+ADR-0001 also maintains `previous_chunk_id`/`next_chunk_id` pointers for O(1)
+adjacency lookups, a single reorder touches **up to four chunks** in one
+`points/batch` call:
+1. The moved chunk: new `order_id`, new `previous_chunk_id`/`next_chunk_id`.
+2. Its old neighbors: re-link their `previous_chunk_id`/`next_chunk_id` to
+   skip over the moved chunk.
+3. Its new neighbors: re-link them to point at the moved chunk.
+
+The same fix-up applies to insert (new chunk) and delete (soft-deleted chunk
+— see below) operations: any operation that changes a chunk's position or
+removes it from the sequence must atomically update the neighbors' pointers
+in the same `points/batch` request, or `previous_chunk_id`/`next_chunk_id`
+go stale.
+
+### Keyword search is not semantic search
+
+`GET /chunks/search` matches against the full-text payload index on
+`content` (and structured filters on `tenant_id`, `domain`, `file_id`,
+etc.) — it is filter/match-based keyword search, not embedding-based
+retrieval. This is deliberately distinct from the hybrid dense+sparse
+retrieval used by the AI agent in ADR-0003; the two "search" concepts serve
+different callers (a human/admin managing chunks vs. an agent retrieving
+context) and should not be conflated in the API or in future discussion.
+
+### Delete is soft by default
+
+`DELETE /chunks/{chunk_id}` and `DELETE /chunks?file_id=...` set
+`is_active: false` and `deleted_at` (via `set_payload`) rather than removing
+the point from Qdrant — consistent with ADR-0001's soft-delete fields. This
+keeps deleted chunks available for audit and lets `GET /chunks` and
+`GET /chunks/search` filter them out by default (`is_active: true` implied
+unless the caller explicitly asks to include inactive chunks). A hard delete
+(actual point removal, e.g. `points/delete`) is available separately for
+compliance-driven purges, not as the default CRUD behavior. Either way, the
+deleted chunk's `previous_chunk_id`/`next_chunk_id` neighbors are relinked to
+point at each other in the same `points/batch` call, so context-window
+expansion (ADR-0003) never walks into a deactivated or removed chunk.
+
+### Tenant isolation
+
+Every endpoint (read and write) has its `tenant_id` filter injected
+server-side from the authenticated request context — never accepted as
+client-supplied input in the request body or query string. This is
+non-negotiable: it's the same isolation boundary ADR-0001 relies on for
+multitenancy, and it must hold for every code path that touches the
+collection, not just ingestion.
+
+### Concurrency
+
+Updates use the `version` payload field (from ADR-0001) together with
+Qdrant's `update_filter`, giving an optimistic-concurrency-style guard
+against races between a concurrent ingestion re-run (ADR-0001) and a manual
+edit through this API.
+
+## Consequences
+
+### Positive
+- Reuses the exact collection/schema from ADR-0001 — no parallel data model
+  to keep in sync.
+- Thin mapping to Qdrant primitives keeps the API predictable and easy to
+  extend as new filter/sort needs arise.
+- Clear separation from ADR-0003's semantic search avoids API consumers
+  confusing "find chunks matching this filter" with "retrieve chunks relevant
+  to this query."
+
+### Negative
+- Keyword search here is limited to what Qdrant's full-text payload index
+  supports — no stemming/synonym handling beyond what that index offers.
+- Optimistic concurrency via `version` requires every writer (ingestion and
+  this API) to consistently read-check-write; a writer that skips this can
+  silently clobber concurrent edits.
+
+## Alternatives Considered
+
+- **Separate keyword-search infrastructure (e.g. Elasticsearch)**: rejected —
+  Qdrant's native full-text payload index already covers filter-style keyword
+  search on `content`, and adding a second search system would duplicate data
+  and infrastructure for no clear benefit at current scale.
+- **Client-supplied `tenant_id` in request body**: rejected — trusting
+  client input for the isolation boundary is a direct multitenancy security
+  risk; it must come from server-side auth context.

docs/adr/0003-agent-hybrid-retrieval.md

@@ -0,0 +1,123 @@
+# 0003. Agent hybrid retrieval (dense + dense + sparse + late-interaction rerank)
+
+## Status
+
+Accepted
+
+## Context
+
+The AI agent needs to retrieve the most relevant chunks for a query against
+the `chunks` collection defined in ADR-0001, which already provisions four
+named vectors — `dense_nomic`, `dense_openai`, `sparse`, and
+`late_interaction` — for exactly this purpose. The goal is higher relevance
+than single-dense-vector search by combining two independent semantic
+signals with lexical (sparse) retrieval, then refining the result with a
+late-interaction (ColBERT-style) rerank.
+
+Qdrant's Query API supports doing this as a single server-side request via
+`prefetch` stages, fusion, and a final rerank stage — avoiding a client-side
+fan-out of multiple separate queries. It also explicitly cautions that
+hybrid search/reranking can *mask* underlying embedding-model quality issues
+rather than fix them, so it shouldn't be adopted without a baseline
+comparison.
+
+Both dense models are now fixed: `dense_nomic` uses `nomic-embed-text-v2-moe`
+([0004](0004-docx-csv-chunking-strategy.md)), which requires a
+`search_query: ` task prefix on the embedded query text (mirroring
+`search_document: ` on the ingestion side); `dense_openai` uses the OpenAI
+large embedding model, no prefix convention needed. `sparse` uses
+`bm25-fa-norm-stop` (ADR-0001). The late-interaction/reranker model is now
+decided — `jina-colbert-v2` — see [0005](0005-reranking-model-and-sparse-analyzer-selection.md)
+for the model comparison, the GPU dependency it introduces, and its
+unresolved commercial-license status.
+
+## Decision
+
+### Query shape
+
+A single Qdrant Query API request per agent query, structured as:
+
+1. **Prefetch 1 — dense ANN**: search the `dense_nomic` vector, top ~100–200
+   candidates.
+2. **Prefetch 2 — dense ANN**: search the `dense_openai` vector, top ~100–200
+   candidates.
+3. **Prefetch 3 — sparse ANN**: search the `sparse` vector, top ~100–200
+   candidates.
+4. **Fusion**: combine all three prefetch results via **RRF** (Reciprocal
+   Rank Fusion) as the default — per Qdrant's own guidance, RRF "ignores
+   score magnitude, a decent default to start with" and handles the
+   incomparable score scales across two dense models and a sparse model.
+5. **Rerank**: apply late-interaction (`late_interaction` multivector,
+   max-sim) reranking over the fused top-N (not the full collection) to
+   produce the final top-k returned to the agent, using `jina-colbert-v2`
+   ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)).
+
+Every prefetch and the final query carries the same `tenant_id`/`domain`
+filter from ADR-0001's payload schema — identical isolation guarantee to
+ADR-0002.
+
+### Context-window expansion
+
+For each chunk returned by the fused/reranked query, the agent may expand
+its context by pulling the immediately preceding and following chunks before
+building the prompt. This uses the `previous_chunk_id`/`next_chunk_id`
+pointer fields from ADR-0001 — a single batch "retrieve points by ID" call
+per result chunk, not an additional filtered search. Since this runs on every
+returned chunk, it relies on those pointers being kept correct by ADR-0002's
+CRUD mutation logic (reorder/insert/delete).
+
+### Bounding rerank cost
+
+Late-interaction rerank is the most compute-expensive stage. It is applied
+only to the fusion's top-N output (e.g. top 50), never to the full
+candidate set or the full collection, to keep latency bounded.
+
+### Evaluate before enabling in production
+
+Before turning this on for real traffic, run an evaluation comparing hybrid
++ rerank against a dense-only baseline on representative queries. Qdrant's
+own guidance warns against adopting hybrid search prematurely, since it can
+paper over embedding-model quality problems instead of addressing them. If
+the baseline is already good, the added complexity/cost of this pipeline
+should be justified with evidence, not assumed.
+
+## Consequences
+
+### Positive
+- Single round-trip server-side fusion + rerank — lower latency than
+  orchestrating multiple queries and fusing client-side in FastAPI.
+- Directly reuses the vector schema and tenant isolation already established
+  in ADR-0001/0002 — no divergent data model for the agent path.
+- Bounding rerank to the fused top-N keeps the expensive late-interaction
+  step's cost predictable regardless of collection size.
+
+### Negative
+- Late-interaction multivectors are storage/compute heavy; with
+  `jina-colbert-v2` chosen ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
+  this stage now also requires GPU capacity (`flash_attn`/CUDA), not just
+  the Docker deployment ADR-0001 assumed.
+- RRF is score-magnitude-agnostic by design — if a use case later needs
+  absolute score information (e.g. a relevance threshold), a different
+  fusion method (DBSF or a custom `FormulaQuery`) would need to be
+  evaluated separately.
+- Adds a mandatory evaluation step before production rollout, rather than
+  shipping hybrid search immediately.
+- Three prefetch stages instead of two (two dense + one sparse) means every
+  agent query embeds the question with both `dense_nomic` and `dense_openai`
+  — an extra embedding call and an external OpenAI API dependency on the
+  query's critical path, not just at ingestion time.
+
+## Alternatives Considered
+
+- **Client-side fan-out** (separate dense and sparse queries against Qdrant,
+  manual RRF fusion in FastAPI): rejected — more round trips and latency
+  than Qdrant's native server-side Query API fusion, for no functional
+  benefit.
+- **Cross-encoder reranking via FastEmbed instead of late-interaction**: not
+  adopted as the initial decision, since ADR-0001 already reserves a
+  multivector field for late-interaction rerank; noted as a possible
+  *additional* future stage rather than a replacement.
+- **DBSF or FormulaQuery as the default fusion method**: rejected as the
+  starting point — RRF is simpler and Qdrant's recommended default; DBSF/
+  FormulaQuery remain available as a later optimization if empirical
+  evaluation shows RRF underperforming.

docs/adr/0004-docx-csv-chunking-strategy.md

@@ -0,0 +1,207 @@
+# 0004. Document parsing and chunking strategy (docx / xlsx / csv)
+
+## Status
+
+Accepted
+
+## Context
+
+ADR-0001 deferred two things to "when we start the docx/csv chunking work":
+the exact `content_type` value set, and any document-context payload fields.
+That work starts now. This ADR also fixes the dense embedding model, which
+ADR-0001 left as TBD — it directly constrains chunk sizing.
+
+A representative sample of real source files (`new_asia_csvs/`, insurance
+documents from Bimeh Asia) was inspected directly (`python-docx`/`openpyxl`)
+before deciding anything, rather than assuming a shape. Findings:
+
+- **No `.csv` files exist in practice** — the tree is entirely `.xlsx`
+  (plus two legacy `.doc` binaries `python-docx` cannot open at all).
+- **Spreadsheets** come in two shapes: clean two-column Q&A tables
+  (`q`/`a` or `سوال`/`پاسخ`), and directory/contact tables (branches, phone
+  numbers) with a merged title row, a blank separator row, and merged cells
+  (`openpyxl` only stores a merged range's value in its top-left cell — the
+  rest read as empty). Several files carry a dead second sheet.
+- **Word documents** have no consistent shape at all: some are flowing prose
+  with real `Heading` styles (rare), most are flowing prose with **no
+  heading styles**, structure only implied by text (`"1- ... پاسخ: ..."`);
+  one file is a single 35×5 table with zero body paragraphs; one file
+  alternates literal "سوال:"/"پاسخ:" paragraphs; two files contain embedded
+  images with no alt text.
+
+Given this, no single chunking algorithm can be applied uniformly — table
+content and flowing prose need different treatment, and even flowing prose
+varies in whether headings can be trusted as section boundaries.
+
+The user ran an offline comparison of five chunking strategies against this
+data and found **semantic-aware chunking** produced the highest retrieval
+accuracy, with **fixed-size chunking** (paired with the previous/next
+pointer fields from ADR-0001) a viable, simpler runner-up.
+
+Embedding model: **`nomic-embed-text-v2-moe`** (dense only — sparse and
+late-interaction models remain TBD per ADR-0001/0003). Relevant constraints
+from its model card: 768-dim output, Matryoshka-truncatable down to 256;
+**512-token max sequence length**; requires a task-instruction prefix on
+every embedded string — `search_document: ` at ingestion time, `search_query: `
+on the agent's query side (ADR-0003).
+
+## Decision
+
+### Parsing order: structural extraction before chunking
+
+Every source file is first decomposed into **structural units** — table
+rows, Q&A pairs, prose blocks — before any chunking algorithm runs. A
+chunking algorithm never sees a whole document as undifferentiated text;
+it only runs on the prose-block units, because table rows and Q&A pairs are
+already atomic and splitting them would break their meaning.
+
+1. **Walk the docx body in document order** (`doc.element.body` children,
+   not `doc.paragraphs`/`doc.tables` separately), so tables interleaved with
+   paragraphs keep their position, and nested tables inside cells are
+   handled recursively. This matters for the "bad design" case the user
+   flagged — a table cell containing an entire sub-document — which is
+   handled as: recurse into the nested table's rows first; if a single cell
+   still contains multiple paragraphs of unstructured prose, treat that
+   cell's text as its own prose block and run the chunker on it, rather
+   than emitting the whole oversized cell as one chunk.
+2. **Detect table rows** → one structural unit per row, columns joined as
+   `"{header}: {value}"` pairs (mirrors the xlsx handling below). Header
+   detection: the first row whose cells are mostly short, unique, non-empty
+   strings; title rows (merged, single populated cell) and blank separator
+   rows are skipped, not treated as headers.
+3. **Detect Q&A pattern** → one structural unit per question/answer pair,
+   via heuristics: literal "سوال"/"پاسخ" (or `q`/`a`) paragraph pairing, or
+   a leading enumerator (`"1-"`, `"2-"`) followed by a "پاسخ:"-prefixed
+   paragraph. This is intentionally a heuristic, not an LLM classification
+   step — it's cheap, deterministic, and the sample above showed the pattern
+   is simple enough (leading marker + adjacent paragraph) not to need model
+   inference per document.
+4. **Everything else** → flowing prose blocks, segmented by heading styles
+   where present (`Heading 1`/`2`), or the whole remaining run of paragraphs
+   as one block where no heading styles exist.
+
+### Chunking the prose blocks
+
+Two supported strategies, selectable by config, **semantic-aware as the
+default**:
+
+- **Semantic-aware chunking** (default): split each prose block into
+  sentences, embed each sentence, and break where adjacent-sentence
+  similarity drops below a percentile threshold — grouping semantically
+  coherent runs of sentences into a chunk. Chosen as default per the user's
+  own offline accuracy comparison.
+- **Fixed-size chunking** (config alternative): token-count-based splitting
+  with overlap. Simpler and cheaper (no embedding pass needed just to
+  decide boundaries), and viable specifically *because* ADR-0001 already
+  gives every chunk `previous_chunk_id`/`next_chunk_id` pointers — a fixed
+  chunk that cuts a thought in half can still be expanded with its
+  neighbors at retrieval time (ADR-0003).
+
+Both strategies share one hard constraint: **no chunk's embedded text may
+exceed `nomic-embed-text-v2-moe`'s 512-token sequence length** — text beyond
+that is silently truncated by the model, not an error, so chunk size must
+be bounded well under 512 tokens regardless of which strategy is active.
+
+Table-row and Q&A structural units bypass this chunker entirely — they are
+already-sized, already-atomic chunks (a row or a Q&A pair is rarely close
+to 512 tokens; if one is, it's truncated the same way, but this is not the
+common case per the sample).
+
+### Spreadsheet (xlsx / csv) handling
+
+Row = chunk, matching the table-row handling above:
+
+- Real header row detected (skipping merged title rows and blank separator
+  rows); each cell rendered as `"{column_header}: {cell_value}"`.
+- Merged cells forward-filled — a merged range's value is copied to every
+  row in that range before chunking, so each row-chunk is self-contained
+  and doesn't silently lose data that `openpyxl` only attaches to the
+  range's top-left cell.
+- Sheets with no non-empty data rows (the dead second-sheet pattern seen in
+  several sample files) are skipped, not ingested as empty chunks.
+- `.csv` is handled identically via `pandas`, even though none exist in the
+  current sample — the row → chunk mapping is the same regardless of
+  container format.
+
+### Embedding model and prefixes
+
+`nomic-embed-text-v2-moe` is the dense embedding model for ADR-0001's
+`dense` vector (768-dim, Matryoshka-truncatable to 256 if storage/latency
+later requires it — not adopted now, full 768 is the default). Every string
+sent to the model **must** carry its task prefix: `search_document: ` when
+embedding a chunk at ingestion time, `search_query: ` when embedding the
+agent's query (ADR-0003) — omitting or mismatching the prefix degrades
+retrieval quality per the model's own documentation. This is a pipeline
+invariant, not a per-call decision.
+
+### Images: LLM extraction now, self-hosted OCR later
+
+Embedded docx images are sent to the ChatGPT (vision) API at ingestion time
+to extract a text description/transcription, which becomes its own chunk
+(`content_type: image_caption`) positioned in document order via the same
+`previous_chunk_id`/`next_chunk_id` mechanism as any other chunk — not
+appended silently into a neighboring text chunk. The extraction call is
+swappable behind a small interface so it can be replaced with a self-hosted
+OCR/captioning model later without changing the chunk/payload shape.
+
+### Legacy `.doc` files
+
+The two legacy binary `.doc` files in the sample cannot be opened by
+`python-docx`. They are converted to `.docx` via headless LibreOffice
+(`soffice --headless --convert-to docx`) as a pipeline pre-processing step
+before the normal docx structural-extraction path runs — not a separate
+parser.
+
+### `content_type` value set (finalized)
+
+| Value | Meaning |
+|---|---|
+| `paragraph` | a chunk produced by the prose chunker (semantic or fixed-size) |
+| `table_row` | one row from a docx table or spreadsheet |
+| `qa_pair` | one detected question/answer pair |
+| `image_caption` | text extracted from an embedded image |
+
+## Consequences
+
+### Positive
+- Structural extraction before chunking means table rows and Q&A pairs —
+  the cleanest, highest-signal content in the sample — are never mangled by
+  a generic text chunker.
+- Semantic-aware default matches the user's own measured accuracy result
+  rather than a generic best practice; fixed-size remains available where
+  its lower cost matters, without losing adjacency thanks to ADR-0001's
+  pointer fields.
+- Fixes the dense embedding model (previously TBD in ADR-0001), unblocking
+  actual implementation of ingestion and agent retrieval.
+- Legacy `.doc` and image content are handled instead of silently dropped
+  or erroring the whole file.
+
+### Negative
+- Heuristic table/Q&A/prose detection will misclassify some future document
+  that doesn't match the patterns in this sample; it will need tuning as
+  new document batches are onboarded, not just this one.
+- Semantic chunking adds an embedding pass at chunk-boundary-decision time,
+  separate from the final chunk-embedding pass — extra ingestion latency
+  and cost versus fixed-size alone.
+- ChatGPT API image extraction is an external network dependency in the
+  ingestion path (cost, latency, availability) until the self-hosted OCR
+  replacement lands.
+- `nomic-embed-text-v2-moe`'s 512-token limit is a hard ceiling on chunk
+  size for both strategies; any future switch to a model with a shorter
+  limit would require re-tuning chunk-size config, though not the pipeline
+  shape.
+
+## Alternatives Considered
+
+- **LLM-based document classification** (send each document to an LLM to
+  decide table/Q&A/prose shape and boundaries): rejected as the default —
+  the sample showed simple heuristics suffice, and per-document LLM calls
+  add cost/latency/non-determinism the heuristics avoid. Not ruled out as a
+  future fallback for prose blocks heuristics can't confidently segment.
+  Not needed for this sample; noted as reference only for future data.
+- **Uniform fixed-size chunking everywhere** (no structural pre-extraction):
+  rejected — would split table rows and Q&A pairs mid-content, destroying
+  the cleanest signal in the source data.
+- **Self-hosted OCR/captioning from day one**: rejected for the initial
+  cut — ChatGPT API gets image extraction working now without standing up
+  and tuning a model first; revisit once volume/cost justifies it.

docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md

@@ -0,0 +1,166 @@
+# 0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus
+
+## Status
+
+Proposed — the fusion/rerank *shape* and reranker model are decided; the
+final BM25 analyzer and the commercial license status of the reranker are
+still open per the follow-up items below.
+
+## Context
+
+ADR-0003 fixed the retrieval *shape* (dense_nomic + dense_openai + sparse
+prefetch → RRF fusion → late-interaction rerank) but deliberately left two
+things open: which model powers `late_interaction`, and the sparse-side
+analyzer detail behind `bm25-fa-norm-stop` (ADR-0001). This ADR resolves
+those, driven by the corpus being **Farsi (Persian)** — a morphologically
+rich, low-resource language for most public embedding/rerank benchmarks and
+for Qdrant's own hosted tooling.
+
+Two things specific to Farsi drove this investigation rather than picking a
+generic default:
+
+- Qdrant's hosted `Qdrant/bm25` FastEmbed model's documented supported-
+  language list does not include Farsi (`fa`) stemming — using it as-is
+  would silently apply no stemming rule, or the wrong one, for this corpus.
+- Farsi's morphology (verb conjugation, ezafe constructions, high-frequency
+  function words) makes pure BM25/keyword signals noisier than in English,
+  which raised the bar on how load-bearing the rerank stage needs to be —
+  treated here as close to mandatory for quality, not optional.
+
+## Decision
+
+### 1. Fusion: RRF by default, weighted RRF as a fallback
+
+Confirms ADR-0003's fusion stage combines all three prefetch results
+(`dense_nomic`, `dense_openai`, `sparse`) via **RRF** by default. If one
+signal (typically sparse, on Farsi text) is observed to dominate the fused
+ranking unexpectedly, **weighted RRF** is the fallback — not a switch to a
+different fusion algorithm. This refines, not replaces, ADR-0003's fusion
+decision.
+
+### 2. Late-interaction reranker: jina-colbert-v2
+
+Two multilingual late-interaction (ColBERT-style) options were evaluated
+against the requirement of confirmed Farsi support:
+
+| Model | Farsi support | License | Local hosting |
+|---|---|---|---|
+| **jina-colbert-v2** | Confirmed — `fa` explicitly listed among 89 supported languages | Conflicting: HF repo metadata says `cc-by-4.0`; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only | Possible via PyLate/RAGatouille; requires `flash_attn` (CUDA GPU effectively required); reported loading issues via generic `transformers.AutoModel` |
+| **LFM2-ColBERT-350M** | Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es) | — | — |
+| BGE-M3 (alternative, not adopted) | Strong Farsi performance in the FaMTEB benchmark | Apache-2.0 — unambiguous, commercial-friendly | CPU-runnable, no `flash_attn` dependency |
+
+**jina-colbert-v2 is adopted** as the `late_interaction` reranker: it has
+explicitly confirmed Farsi support among its 89 languages, which is the
+primary requirement for this corpus. This means:
+
+- A **CUDA GPU is now a required dependency** for the rerank stage
+  (`flash_attn`), not optional infrastructure — this changes the self-hosted
+  Docker deployment assumption from ADR-0001/0003, which had not committed
+  to GPU hosting.
+- The **commercial license is unresolved** — HF repo metadata states
+  `cc-by-4.0` while Jina's own announcement states CC BY-NC-4.0
+  (non-commercial, with commercial use only via Jina's paid API/AWS/Azure
+  offering). This must be confirmed directly with Jina **before** this
+  project ships commercially on a self-hosted jina-colbert-v2 model; if
+  Jina confirms the non-commercial reading, self-hosting it commercially is
+  not an option and the fallback is Jina's paid hosted API or BGE-M3 as a
+  substitute reranker (same `late_interaction` vector shape, no schema
+  change needed either way).
+- Loading it outside PyLate/RAGatouille (e.g. generic
+  `transformers.AutoModel`) has reported issues — plan to load it through
+  one of those two libraries, not a raw `transformers` call.
+
+`late_interaction` continues to use `hnsw_config: m=0` (per ADR-0001,
+now made explicit: HNSW indexing is disabled for this vector because it is
+used only for reranking already-fetched candidates via MAX_SIM, never for
+independent ANN search — the recommended pattern for rerank-only
+multivectors) plus on-disk storage.
+
+### 3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model
+
+Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi
+normalization/stopword/stemming computed outside Qdrant Cloud Inference) is
+used instead of Qdrant's hosted `Qdrant/bm25` FastEmbed model, specifically
+because that hosted model's documented language list omits Farsi. The
+sparse vector is uploaded with `modifier="idf"`, the standard BM25 sparse
+vector configuration.
+
+Four analyzer variants were benchmarked, all sharing identical BM25 scoring
+parameters (`k=1.2`, `b=0.75`) so any accuracy difference is attributable
+entirely to the analyzer stage, not the ranking formula:
+
+| Analyzer | Description |
+|---|---|
+| `bm25-raw` | no normalization |
+| `bm25-fa-norm` | Farsi normalization only |
+| `bm25-fa-norm-stop` | normalization + stopword removal — **current best performer** |
+| `bm25-fa-norm-stem` | normalization + stemming |
+
+`bm25-fa-norm-stop` is confirmed as the sparse analyzer named in ADR-0001,
+consistent with Farsi's high density of function words (ezafe particles,
+prepositions, common verbs) adding TF/IDF noise if left in.
+
+### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula
+
+These are standard, well-validated defaults (Trotman, Puurula & Burgess,
+2014) and are not the source of the observed analyzer-variant accuracy
+differences — so formula tuning is deprioritized in favor of the analyzer
+comparison and `b` sweep in the follow-ups below.
+
+## Consequences
+
+### Positive
+- Resolves ADR-0003's two deferred decisions (reranker model, sparse
+  analyzer detail) with a corpus-specific rationale instead of a generic
+  default.
+- jina-colbert-v2 is the only evaluated option with explicitly confirmed
+  Farsi support, directly matching this project's primary requirement.
+- Isolating BM25 formula parameters from analyzer choice gives a clean,
+  defensible experimental basis for the `bm25-fa-norm-stop` selection.
+
+### Negative
+- Introduces a hard GPU dependency (`flash_attn`/CUDA) for the rerank stage
+  that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU
+  capacity, not just Docker on commodity hardware.
+- Commercial license status is unresolved; shipping this commercially on a
+  self-hosted jina-colbert-v2 model without confirming licensing with Jina
+  is a legal risk, not just a technical one.
+- Loading path is constrained to PyLate/RAGatouille due to reported
+  `transformers.AutoModel` issues — an extra library dependency and less
+  flexibility than a standard `transformers` load would give.
+- `bm25-fa-norm-stop` is provisional until compared directly against
+  `bm25-fa-norm-stem` — Persian stemming can over-collapse distinct words
+  (irregular verb conjugation, Arabic-loanword plurals), so the current
+  "best performer" result could shift.
+
+## Alternatives Considered
+
+- **BGE-M3 as the reranker**: not adopted — Apache-2.0 license and
+  CPU-runnability are attractive and it remains the fallback if
+  jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2
+  was prioritized for its explicit Farsi support.
+- **LFM2-ColBERT-350M**: rejected outright — no Farsi support among its 8
+  supported languages.
+- **Qdrant's hosted `Qdrant/bm25` FastEmbed model**: rejected — its
+  documented language support does not include Farsi stemming; using it
+  would risk silently wrong or absent stemming for this corpus.
+- **Treating "RRF vs. rerank" as either/or**: rejected — RRF alone can't
+  resolve disagreement between two distinct dense embedding spaces plus a
+  noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank
+  pipeline from ADR-0003 is kept, not replaced with either component alone.
+
+## Follow-up / Open Items
+
+1. Confirm jina-colbert-v2's commercial license status directly with Jina
+   before commercial deployment; fall back to Jina's paid hosted API or
+   BGE-M3 if the non-commercial reading is confirmed.
+2. Provision GPU capacity for self-hosted jina-colbert-v2 (`flash_attn`
+   requires CUDA) as part of the deployment plan, not an afterthought.
+3. Run an ablation: single dense model + sparse + rerank vs. the current
+   dual-dense-model + sparse + rerank setup, on real Farsi queries, to
+   justify (or drop) the second dense vector (`dense_openai`).
+4. Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation to
+   determine whether gains come from stopword removal, stemming, or both.
+5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document
+   length varies significantly across the corpus (short chat messages vs.
+   long articles) and `0.75` is a generic default, not corpus-tuned.