feat(ingestion): add bounded, benchmark-aligned embedding execution

Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
  the inline upload path, with process-wide capacity/timeout/chunk-limit
  guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
  evaluation lab, which benchmarked them against the real Farsi corpus
  (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
  at native 3072-dim), closing open items in ADR-0001/ADR-0005.

Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
  helpers, and dense/sparse adapters (analyzers.py, bm25.py,
  openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
  (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
  every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
  creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).

Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-19 17:13:32 +03:30
parent aa6d595424
commit 5c0a5938f8
33 changed files with 2455 additions and 536 deletions

View File

@@ -49,7 +49,7 @@ One collection, e.g. `chunks`, shared by all tenants and domains.
| Name | Type | Purpose | Notes |
|---|---|---|---|
| `dense_nomic` | dense vector | primary semantic similarity (multilingual, incl. Persian) | `nomic-embed-text-v2-moe`, 768-dim ([0004](0004-docx-csv-chunking-strategy.md)) |
| `dense_openai` | dense vector | second semantic signal | OpenAI large embedding model (e.g. `text-embedding-3-large`), dimension per OpenAI's `dimensions` param (TBD — full 3072 vs. a truncated size) |
| `dense_openai` | dense vector | second semantic signal | `text-embedding-3-large` at its **native 3072 dimensions** — the `dimensions` param is deliberately left unset (see below) |
| `sparse` | sparse vector | lexical/keyword-sensitive retrieval | `bm25-fa-norm-stop` — Qdrant FastEmbed's BM25 sparse encoder configured for Persian (stopword removal + normalization), not a separately trained model |
| `late_interaction` | multivector | reserved for late-interaction rerank ([0003](0003-agent-hybrid-retrieval.md)) | `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)), `comparator: max_sim`, `hnsw_config: m=0` (rerank-only, never independently ANN-searched), stored **on disk** |
@@ -61,6 +61,35 @@ dense/sparse query latency. Two dense vectors are provisioned deliberately —
`dense_nomic` and `dense_openai` are two independent semantic signals, both
prefetched and fused at query time (ADR-0003), not a primary/fallback pair.
#### Dense model endpoints and dimensions (resolved by the `emet` benchmark)
Both dense models are reached over the **same OpenAI-compatible
`/embeddings` API**, so one adapter
(`src/infrastructure/embedding/openai_compatible.py`) serves both named
vectors with different configuration:
| Named vector | Model | Endpoint | Dimensions |
|---|---|---|---|
| `dense_nomic` | `nomic-embed-text-v2-moe` | self-hosted Ollama OpenAI-compat shim | **768** (verified against the live endpoint) |
| `dense_openai` | `text-embedding-3-large` | OpenAI hosted API | **3072** (native; `dimensions` unset) |
`dense_openai`'s dimension was previously listed as an open dependency. It is
now pinned to the native 3072, because that is the configuration the `emet`
lab benchmarked — it never passed a `dimensions` argument. Setting it later
would truncate via Matryoshka and is a **re-embedding migration, not a config
tweak**, exactly as the negative consequence below warns.
Two operational notes about the self-hosted embedder, both learned by
measurement rather than assumption:
- **Cold load exceeds 150s**, far beyond `INGESTION_TIMEOUT_SECONDS`, so an
idle-then-upload would return `504`. Mitigated on both ends: Ollama's
`keep_alive` keeps the model resident, and the FastAPI lifespan warms each
dense embedder at startup (fail-soft — a down embedder must not block boot).
- **Once warm it is fast**: ~0.30s for one input and ~0.34s for a batch of 16.
Batching is therefore nearly free, which is what keeps ADR-0017's inline
ingestion viable.
### Multitenancy / indexing config
- HNSW: `m: 0` (disable the global index) + `payload_m: 16`, per Qdrant's
@@ -212,9 +241,14 @@ them — see ADR-0002 for how reorder/insert/delete operations keep
ingestion time and both are queried at retrieval time — roughly double
the dense embedding cost/latency of a single-dense-vector design, plus an
external network dependency on OpenAI's API in the ingestion path.
- `dense_openai`'s exact output dimension is still an open dependency that
should be pinned before ingestion is implemented — changing it later is a
re-embedding migration, not a config tweak.
- ~~`dense_openai`'s exact output dimension is still an open dependency~~ —
**resolved**: pinned to the native 3072 (see "Dense model endpoints and
dimensions" above). The warning still stands for any future change:
re-dimensioning is a re-embedding migration, not a config tweak.
- The `sparse` vector must be created with `modifier="idf"`. The client
computes only BM25's term-frequency saturation; without that modifier
Qdrant applies no IDF at all and lexical retrieval silently degrades
(ADR-0005).
- `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
adds a hard GPU dependency to ingestion (not just query time, since the
document-side multivector is computed here) and its commercial license is