feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,6 +9,7 @@ dependencies = [
|
||||
"anyio>=4.11.0",
|
||||
"asyncpg>=0.31.0",
|
||||
"fastapi[standard]==0.141.1",
|
||||
"httpx>=0.28.1",
|
||||
"langgraph>=1.2.10",
|
||||
"minio>=7.2.20",
|
||||
"openpyxl>=3.1.5",
|
||||
@@ -23,7 +24,6 @@ dependencies = [
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"asgi-lifespan>=2.1.0",
|
||||
"httpx>=0.28.1",
|
||||
"pytest>=8.3.5",
|
||||
"pytest-asyncio>=0.25.3",
|
||||
"pytest-cov>=6.0.0",
|
||||
|
||||
Reference in New Issue
Block a user