feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -49,7 +49,7 @@ One collection, e.g. `chunks`, shared by all tenants and domains.
|
||||
| Name | Type | Purpose | Notes |
|
||||
|---|---|---|---|
|
||||
| `dense_nomic` | dense vector | primary semantic similarity (multilingual, incl. Persian) | `nomic-embed-text-v2-moe`, 768-dim ([0004](0004-docx-csv-chunking-strategy.md)) |
|
||||
| `dense_openai` | dense vector | second semantic signal | OpenAI large embedding model (e.g. `text-embedding-3-large`), dimension per OpenAI's `dimensions` param (TBD — full 3072 vs. a truncated size) |
|
||||
| `dense_openai` | dense vector | second semantic signal | `text-embedding-3-large` at its **native 3072 dimensions** — the `dimensions` param is deliberately left unset (see below) |
|
||||
| `sparse` | sparse vector | lexical/keyword-sensitive retrieval | `bm25-fa-norm-stop` — Qdrant FastEmbed's BM25 sparse encoder configured for Persian (stopword removal + normalization), not a separately trained model |
|
||||
| `late_interaction` | multivector | reserved for late-interaction rerank ([0003](0003-agent-hybrid-retrieval.md)) | `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md)), `comparator: max_sim`, `hnsw_config: m=0` (rerank-only, never independently ANN-searched), stored **on disk** |
|
||||
|
||||
@@ -61,6 +61,35 @@ dense/sparse query latency. Two dense vectors are provisioned deliberately —
|
||||
`dense_nomic` and `dense_openai` are two independent semantic signals, both
|
||||
prefetched and fused at query time (ADR-0003), not a primary/fallback pair.
|
||||
|
||||
#### Dense model endpoints and dimensions (resolved by the `emet` benchmark)
|
||||
|
||||
Both dense models are reached over the **same OpenAI-compatible
|
||||
`/embeddings` API**, so one adapter
|
||||
(`src/infrastructure/embedding/openai_compatible.py`) serves both named
|
||||
vectors with different configuration:
|
||||
|
||||
| Named vector | Model | Endpoint | Dimensions |
|
||||
|---|---|---|---|
|
||||
| `dense_nomic` | `nomic-embed-text-v2-moe` | self-hosted Ollama OpenAI-compat shim | **768** (verified against the live endpoint) |
|
||||
| `dense_openai` | `text-embedding-3-large` | OpenAI hosted API | **3072** (native; `dimensions` unset) |
|
||||
|
||||
`dense_openai`'s dimension was previously listed as an open dependency. It is
|
||||
now pinned to the native 3072, because that is the configuration the `emet`
|
||||
lab benchmarked — it never passed a `dimensions` argument. Setting it later
|
||||
would truncate via Matryoshka and is a **re-embedding migration, not a config
|
||||
tweak**, exactly as the negative consequence below warns.
|
||||
|
||||
Two operational notes about the self-hosted embedder, both learned by
|
||||
measurement rather than assumption:
|
||||
|
||||
- **Cold load exceeds 150s**, far beyond `INGESTION_TIMEOUT_SECONDS`, so an
|
||||
idle-then-upload would return `504`. Mitigated on both ends: Ollama's
|
||||
`keep_alive` keeps the model resident, and the FastAPI lifespan warms each
|
||||
dense embedder at startup (fail-soft — a down embedder must not block boot).
|
||||
- **Once warm it is fast**: ~0.30s for one input and ~0.34s for a batch of 16.
|
||||
Batching is therefore nearly free, which is what keeps ADR-0017's inline
|
||||
ingestion viable.
|
||||
|
||||
### Multitenancy / indexing config
|
||||
|
||||
- HNSW: `m: 0` (disable the global index) + `payload_m: 16`, per Qdrant's
|
||||
@@ -212,9 +241,14 @@ them — see ADR-0002 for how reorder/insert/delete operations keep
|
||||
ingestion time and both are queried at retrieval time — roughly double
|
||||
the dense embedding cost/latency of a single-dense-vector design, plus an
|
||||
external network dependency on OpenAI's API in the ingestion path.
|
||||
- `dense_openai`'s exact output dimension is still an open dependency that
|
||||
should be pinned before ingestion is implemented — changing it later is a
|
||||
re-embedding migration, not a config tweak.
|
||||
- ~~`dense_openai`'s exact output dimension is still an open dependency~~ —
|
||||
**resolved**: pinned to the native 3072 (see "Dense model endpoints and
|
||||
dimensions" above). The warning still stands for any future change:
|
||||
re-dimensioning is a re-embedding migration, not a config tweak.
|
||||
- The `sparse` vector must be created with `modifier="idf"`. The client
|
||||
computes only BM25's term-frequency saturation; without that modifier
|
||||
Qdrant applies no IDF at all and lexical retrieval silently degrades
|
||||
(ADR-0005).
|
||||
- `jina-colbert-v2` ([0005](0005-reranking-model-and-sparse-analyzer-selection.md))
|
||||
adds a hard GPU dependency to ingestion (not just query time, since the
|
||||
document-side multivector is computed here) and its commercial license is
|
||||
|
||||
@@ -62,6 +62,23 @@ from its model card: 768-dim output, Matryoshka-truncatable down to 256;
|
||||
every embedded string — `search_document: ` at ingestion time, `search_query: `
|
||||
on the agent's query side (ADR-0003).
|
||||
|
||||
> **Amendment — the task prefix is currently not applied.** The `emet`
|
||||
> benchmark that selected this model ran *without* any prefix: its Ollama
|
||||
> deployment's template is a bare `{{ .Prompt }}` passthrough that injects
|
||||
> nothing, which was verified directly against the running endpoint. The
|
||||
> prefix is not cosmetic — embedding the same Persian text with and without
|
||||
> `search_document: ` yields a cosine of only **0.5741** — so applying it at
|
||||
> ingest while the query side omits `search_query: ` would make retrieval
|
||||
> *worse* than using neither.
|
||||
>
|
||||
> Implementation therefore defaults `EMBEDDING_NOMIC_DOCUMENT_PREFIX` to
|
||||
> empty, matching the measured configuration, and exposes it as config so the
|
||||
> prefixed variant is a one-line experiment rather than a code change. The
|
||||
> model card remains the reason to expect prefixing to help; what is missing
|
||||
> is evidence on *this* corpus. Turning it on is a paired change — ingest and
|
||||
> query must move together — and should be settled by an emet run that
|
||||
> measures the pair, not by an unmeasured edit here.
|
||||
|
||||
## Decision
|
||||
|
||||
### Parsing order: structural extraction before chunking
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
|
||||
## Status
|
||||
|
||||
Proposed — the fusion/rerank *shape* and reranker model are decided; the
|
||||
final BM25 analyzer and the commercial license status of the reranker are
|
||||
still open per the follow-up items below.
|
||||
Proposed — the fusion/rerank *shape*, the reranker model, and (as of the
|
||||
`emet` benchmark, see "Benchmark outcome" below) the **BM25 analyzer** are
|
||||
decided. The commercial license status of the reranker remains open per the
|
||||
follow-up items below.
|
||||
|
||||
## Context
|
||||
|
||||
@@ -100,6 +101,50 @@ entirely to the analyzer stage, not the ranking formula:
|
||||
consistent with Farsi's high density of function words (ezafe particles,
|
||||
prepositions, common verbs) adding TF/IDF noise if left in.
|
||||
|
||||
### 3a. Benchmark outcome: `bm25-fa-norm-stop` confirmed, and where the BM25 math runs
|
||||
|
||||
The `emet` evaluation lab (`~/code/talie/emet`) ran the four-variant
|
||||
comparison above against the real Farsi corpus and confirmed
|
||||
**`bm25-fa-norm-stop`** as the winner. It is the only sparse variant promoted
|
||||
into emet's hybrid matrix (`emet/hybrid.yaml`). This closes follow-up item 4
|
||||
below.
|
||||
|
||||
The winning analyzer is a specific, reproducible artifact, ported into
|
||||
`src/infrastructure/embedding/analyzers.py` and verified token-for-token
|
||||
against emet's implementation. Its details are load-bearing:
|
||||
|
||||
- Unicode **NFC** (not NFKC), then ZWNJ → space, then Persian/Arabic-Indic
|
||||
digits → ASCII, then `ي→ی ك→ک ة→ه ؤ→و إ→ا أ→ا`.
|
||||
- Tokenizer `[^\W_]+`, which **keeps digits**. This matters for an insurance
|
||||
corpus: policy numbers, dates, and amounts are exactly the terms lexical
|
||||
retrieval should match, and the digit folding above means a query in ASCII
|
||||
digits matches a document authored in Persian ones.
|
||||
- A 51-entry stopword set (40 Persian/Arabic + 11 English, the corpus being
|
||||
mixed-script). Deliberately not a full `hazm` list.
|
||||
- No stemming — `fa_norm_stem` was the losing arm.
|
||||
|
||||
**The BM25 formula is split across two systems, deliberately.** The client
|
||||
applies term-frequency saturation, including the `k`/`b` document-length
|
||||
normalization; **IDF is supplied by Qdrant** via `modifier="idf"` on the
|
||||
sparse vector field, computed from collection-wide statistics rather than
|
||||
from a fixed client-side corpus.
|
||||
|
||||
That split is a correctness trap worth stating plainly: a `chunks` collection
|
||||
created *without* `modifier="idf"` will score these vectors as saturated term
|
||||
frequencies with no IDF weighting at all — no error, no warning, just
|
||||
materially worse lexical retrieval. The collection bootstrap must set it.
|
||||
|
||||
Document and query encoding are asymmetric in exactly one term: documents
|
||||
carry the `b` length normalization, queries do not (standard BM25 practice).
|
||||
Both sides must therefore encode through the same implementation, which is
|
||||
why the sparse port carries a `query` flag rather than leaving retrieval to
|
||||
grow a second, silently divergent encoder.
|
||||
|
||||
Term → sparse-index mapping is `blake2b(token, digest_size=8) % (2**31 - 1)`,
|
||||
a pure hash with no vocabulary table, so it needs no shared state and stays
|
||||
identical across processes and between ingest and query time. Changing the
|
||||
hash orphans every stored sparse vector: that is a re-ingestion, not a deploy.
|
||||
|
||||
### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula
|
||||
|
||||
These are standard, well-validated defaults (Trotman, Puurula & Burgess,
|
||||
@@ -159,8 +204,21 @@ comparison and `b` sweep in the follow-ups below.
|
||||
3. Run an ablation: single dense model + sparse + rerank vs. the current
|
||||
dual-dense-model + sparse + rerank setup, on real Farsi queries, to
|
||||
justify (or drop) the second dense vector (`dense_openai`).
|
||||
4. Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation to
|
||||
determine whether gains come from stopword removal, stemming, or both.
|
||||
4. ~~Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation~~ —
|
||||
**done**, see "Benchmark outcome" above. `fa_norm_stop` won; stemming was
|
||||
not adopted.
|
||||
5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document
|
||||
length varies significantly across the corpus (short chat messages vs.
|
||||
long articles) and `0.75` is a generic default, not corpus-tuned.
|
||||
6. **Recalibrate `avg_len`.** The client-side `b` term needs an average
|
||||
document length in *analyzer tokens*. The ported value (256.0) is emet's
|
||||
own placeholder, and emet measured it over short Q&A records rather than
|
||||
this service's ~400-token chunks, so it is very likely miscalibrated here.
|
||||
Exposed as `EMBEDDING_SPARSE_AVG_LEN` so it can be corrected from real
|
||||
corpus statistics without a code change.
|
||||
7. **Re-benchmark the analyzer with diacritic stripping.** `fa_norm_stop`
|
||||
does not remove harakat or tatweel, so `ســلام` and `سلام` are distinct
|
||||
terms. `src/application/ingestion/normalization.py` already strips both
|
||||
for chunk *content*; extending that to the analyzer is plausibly an
|
||||
improvement but would deviate from the measured configuration, so it
|
||||
belongs in an emet run rather than an unmeasured edit.
|
||||
|
||||
@@ -247,6 +247,16 @@ code and a terminal job row.
|
||||
|
||||
### Phase 5: Ingestion execution and Qdrant Chunk/Point CRUD
|
||||
|
||||
> **Carried forward from Phase 4 — the `chunks` collection must create the
|
||||
> `sparse` vector with `modifier="idf"`.** The BM25 adapter computes only
|
||||
> term-frequency saturation client-side; IDF comes from Qdrant's
|
||||
> collection-wide statistics. Omit the modifier and there is no error and no
|
||||
> warning — sparse scoring silently loses its IDF term and lexical retrieval
|
||||
> degrades. See ADR-0005, "Benchmark outcome".
|
||||
>
|
||||
> Collection creation must also use the pinned dimensions from ADR-0001:
|
||||
> `dense_nomic` 768, `dense_openai` 3072.
|
||||
|
||||
1. Implement the ingestion service called by the route, using the
|
||||
application-lifetime database, MinIO, Qdrant, model, and logging clients.
|
||||
2. Validate the persisted records before fetching the MinIO object.
|
||||
|
||||
Reference in New Issue
Block a user