feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -247,6 +247,16 @@ code and a terminal job row.
|
||||
|
||||
### Phase 5: Ingestion execution and Qdrant Chunk/Point CRUD
|
||||
|
||||
> **Carried forward from Phase 4 — the `chunks` collection must create the
|
||||
> `sparse` vector with `modifier="idf"`.** The BM25 adapter computes only
|
||||
> term-frequency saturation client-side; IDF comes from Qdrant's
|
||||
> collection-wide statistics. Omit the modifier and there is no error and no
|
||||
> warning — sparse scoring silently loses its IDF term and lexical retrieval
|
||||
> degrades. See ADR-0005, "Benchmark outcome".
|
||||
>
|
||||
> Collection creation must also use the pinned dimensions from ADR-0001:
|
||||
> `dense_nomic` 768, `dense_openai` 3072.
|
||||
|
||||
1. Implement the ingestion service called by the route, using the
|
||||
application-lifetime database, MinIO, Qdrant, model, and logging clients.
|
||||
2. Validate the persisted records before fetching the MinIO object.
|
||||
|
||||
Reference in New Issue
Block a user