feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -49,7 +49,7 @@ One collection, e.g. `chunks`, shared by all tenants and domains.
|
||||
| Name | Type | Purpose | Notes |
|
||||
|---|---|---|---|
|
||||
| `dense_nomic` | dense vector | primary semantic similarity (multilingual, incl. Persian) | `nomic-embed-text-v2-moe`, 768-dim ([0004](0004-docx-csv-chunking-strategy.md)) |
|
||||
| `dense_openai` | dense vector | second semantic signal | OpenAI large embedding model (e.g. `text-embedding-3-large`), dimension per OpenAI's `dimensions` param (TBD — full 3072 vs. a truncated size) |
|
||||
| `dense_openai` | dense vector | second semantic signal | `text-embedding-3-large` at its **native 3072 dimensions** — the `dimensions` param is deliberately left unset (see below) |
|
||||
| `sparse` | sparse vector | lexical/keyword-sensitive retrieval | `bm25-fa-norm-stop` — Qdrant FastEmbed's BM25 sparse encoder configured for Persian (stopword removal + normalization), not a separately trained model |
|
||||
| `late_interaction` | multivector | reserved for late-interaction rerank ([0003](0003-agent-hybrid-retrieval.md)) | `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)), `comparator: max_sim`, `hnsw_config: m=0` (rerank-only, never independently ANN-searched), stored **on disk** |
|
||||
|
||||
@@ -61,6 +61,35 @@ dense/sparse query latency. Two dense vectors are provisioned deliberately —
|
||||
`dense_nomic` and `dense_openai` are two independent semantic signals, both
|
||||
prefetched and fused at query time (ADR-0003), not a primary/fallback pair.
|
||||
|
||||
#### Dense model endpoints and dimensions (resolved by the `emet` benchmark)
|
||||
|
||||
Both dense models are reached over the **same OpenAI-compatible
|
||||
`/embeddings` API**, so one adapter
|
||||
(`src/infrastructure/embedding/openai_compatible.py`) serves both named
|
||||
vectors with different configuration:
|
||||
|
||||
| Named vector | Model | Endpoint | Dimensions |
|
||||
|---|---|---|---|
|
||||
| `dense_nomic` | `nomic-embed-text-v2-moe` | self-hosted Ollama OpenAI-compat shim | **768** (verified against the live endpoint) |
|
||||
| `dense_openai` | `text-embedding-3-large` | OpenAI hosted API | **3072** (native; `dimensions` unset) |
|
||||
|
||||
`dense_openai`'s dimension was previously listed as an open dependency. It is
|
||||
now pinned to the native 3072, because that is the configuration the `emet`
|
||||
lab benchmarked — it never passed a `dimensions` argument. Setting it later
|
||||
would truncate via Matryoshka and is a **re-embedding migration, not a config
|
||||
tweak**, exactly as the negative consequence below warns.
|
||||
|
||||
Two operational notes about the self-hosted embedder, both learned by
|
||||
measurement rather than assumption:
|
||||
|
||||
- **Cold load exceeds 150s**, far beyond `INGESTION_TIMEOUT_SECONDS`, so an
|
||||
idle-then-upload would return `504`. Mitigated on both ends: Ollama's
|
||||
`keep_alive` keeps the model resident, and the FastAPI lifespan warms each
|
||||
dense embedder at startup (fail-soft — a down embedder must not block boot).
|
||||
- **Once warm it is fast**: ~0.30s for one input and ~0.34s for a batch of 16.
|
||||
Batching is therefore nearly free, which is what keeps ADR-0017's inline
|
||||
ingestion viable.
|
||||
|
||||
### Multitenancy / indexing config
|
||||
|
||||
- HNSW: `m: 0` (disable the global index) + `payload_m: 16`, per Qdrant's
|
||||
@@ -212,9 +241,14 @@ them — see ADR-0002 for how reorder/insert/delete operations keep
|
||||
ingestion time and both are queried at retrieval time — roughly double
|
||||
the dense embedding cost/latency of a single-dense-vector design, plus an
|
||||
external network dependency on OpenAI's API in the ingestion path.
|
||||
- `dense_openai`'s exact output dimension is still an open dependency that
|
||||
should be pinned before ingestion is implemented — changing it later is a
|
||||
re-embedding migration, not a config tweak.
|
||||
- ~~`dense_openai`'s exact output dimension is still an open dependency~~ —
|
||||
**resolved**: pinned to the native 3072 (see "Dense model endpoints and
|
||||
dimensions" above). The warning still stands for any future change:
|
||||
re-dimensioning is a re-embedding migration, not a config tweak.
|
||||
- The `sparse` vector must be created with `modifier="idf"`. The client
|
||||
computes only BM25's term-frequency saturation; without that modifier
|
||||
Qdrant applies no IDF at all and lexical retrieval silently degrades
|
||||
(ADR-0005).
|
||||
- `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
|
||||
adds a hard GPU dependency to ingestion (not just query time, since the
|
||||
document-side multivector is computed here) and its commercial license is
|
||||
|
||||
Reference in New Issue
Block a user