feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -62,6 +62,23 @@ from its model card: 768-dim output, Matryoshka-truncatable down to 256;
|
||||
every embedded string — `search_document: ` at ingestion time, `search_query: `
|
||||
on the agent's query side (ADR-0003).
|
||||
|
||||
> **Amendment — the task prefix is currently not applied.** The `emet`
|
||||
> benchmark that selected this model ran *without* any prefix: its Ollama
|
||||
> deployment's template is a bare `{{ .Prompt }}` passthrough that injects
|
||||
> nothing, which was verified directly against the running endpoint. The
|
||||
> prefix is not cosmetic — embedding the same Persian text with and without
|
||||
> `search_document: ` yields a cosine of only **0.5741** — so applying it at
|
||||
> ingest while the query side omits `search_query: ` would make retrieval
|
||||
> *worse* than using neither.
|
||||
>
|
||||
> Implementation therefore defaults `EMBEDDING_NOMIC_DOCUMENT_PREFIX` to
|
||||
> empty, matching the measured configuration, and exposes it as config so the
|
||||
> prefixed variant is a one-line experiment rather than a code change. The
|
||||
> model card remains the reason to expect prefixing to help; what is missing
|
||||
> is evidence on *this* corpus. Turning it on is a paired change — ingest and
|
||||
> query must move together — and should be settled by an emet run that
|
||||
> measures the pair, not by an unmeasured edit here.
|
||||
|
||||
## Decision
|
||||
|
||||
### Parsing order: structural extraction before chunking
|
||||
|
||||
Reference in New Issue
Block a user