feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
39
.env.example
39
.env.example
@@ -45,6 +45,45 @@ INGESTION_EMBED_CONCURRENCY=4
|
||||
QDRANT_URL=http://127.0.0.1:6343
|
||||
QDRANT_API_KEY=
|
||||
|
||||
# Dense embedders (ADR-0001). Both speak an OpenAI-compatible /embeddings
|
||||
# endpoint, so one adapter serves both. Models and endpoints are the ones the
|
||||
# `emet` evaluation lab benchmarked as winners on the Farsi corpus.
|
||||
#
|
||||
# dense_nomic runs behind Ollama's OpenAI-compat shim, which accepts any
|
||||
# non-empty API key. KEEP_ALIVE holds the model resident: a cold load of
|
||||
# nomic-embed-text-v2-moe takes >150s, well past INGESTION_TIMEOUT_SECONDS,
|
||||
# so an idle-then-upload would otherwise 504.
|
||||
EMBEDDING_NOMIC_BASE_URL=http://192.168.10.10:11435/v1
|
||||
EMBEDDING_NOMIC_MODEL=nomic-embed-text-v2-moe
|
||||
EMBEDDING_NOMIC_API_KEY=sk-not-set
|
||||
EMBEDDING_NOMIC_KEEP_ALIVE=30m
|
||||
EMBEDDING_NOMIC_TIMEOUT_SECONDS=30.0
|
||||
# Empty = emet parity. The model card specifies `search_document: ` (ADR-0004),
|
||||
# but the benchmark ran without it and the prefix shifts the vector a lot
|
||||
# (cosine 0.57 on identical text) — so if you set this, the query side must
|
||||
# send `search_query: ` to match, or retrieval gets worse rather than better.
|
||||
EMBEDDING_NOMIC_DOCUMENT_PREFIX=
|
||||
|
||||
# Leave DIMENSIONS empty for text-embedding-3-large's native 3072, which is
|
||||
# what was benchmarked. Setting it truncates via Matryoshka and is a
|
||||
# re-embedding migration, not a config tweak.
|
||||
EMBEDDING_OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
EMBEDDING_OPENAI_MODEL=text-embedding-3-large
|
||||
EMBEDDING_OPENAI_API_KEY=
|
||||
EMBEDDING_OPENAI_DIMENSIONS=
|
||||
EMBEDDING_OPENAI_DOCUMENT_PREFIX=
|
||||
EMBEDDING_OPENAI_TIMEOUT_SECONDS=30.0
|
||||
|
||||
# Sparse BM25 (ADR-0001, ADR-0005): the benchmarked `bm25-fa-norm-stop`.
|
||||
# k/b saturation is applied client-side; IDF comes from Qdrant's
|
||||
# modifier="idf" on the sparse vector field. AVG_LEN is the average document
|
||||
# length in analyzer tokens — emet's placeholder, worth recalibrating from
|
||||
# real corpus statistics.
|
||||
EMBEDDING_SPARSE_ANALYZER=fa_norm_stop
|
||||
EMBEDDING_SPARSE_K=1.2
|
||||
EMBEDDING_SPARSE_B=0.75
|
||||
EMBEDDING_SPARSE_AVG_LEN=256.0
|
||||
|
||||
# Parsing and chunking (ADR-0018).
|
||||
# max_chunk_tokens is nomic-embed-text-v2-moe's sequence length; text past it
|
||||
# is silently truncated by the model, so the cap is enforced before embedding.
|
||||
|
||||
Reference in New Issue
Block a user