feat(ingestion): add bounded, benchmark-aligned embedding execution
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
20
CLAUDE.md
20
CLAUDE.md
@@ -7,11 +7,21 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
This repo is ADR-driven and early in implementation. Working today: the FastAPI
|
||||
app factory and lifespan wiring (`src/bootstrap/`), `/healthz` and `/readyz`,
|
||||
structlog config, Postgres/MinIO/Qdrant clients (`src/infrastructure/`), five
|
||||
SQLAlchemy models with one Alembic migration, and document parsing plus
|
||||
fixed-size chunking (`src/application/ingestion/`). Not built yet: API-key auth,
|
||||
`POST /v1/files` (so `/v1` currently exposes no routes), repositories, embedding
|
||||
adapters, Qdrant collection bootstrap, and `src/agent/`. That maps to plan 001
|
||||
Phase 1 done, Phase 2 partly done, and the parsing half of Phase 5.
|
||||
SQLAlchemy models with one Alembic migration, document parsing plus fixed-size
|
||||
chunking (`src/application/ingestion/`), API-key auth, `POST`/`GET /v1/files`
|
||||
with durable two-phase job creation (`src/application/files/`), and bounded
|
||||
inline embedding: `dense_nomic`/`dense_openai` adapters over an
|
||||
OpenAI-compatible HTTP client and a `bm25-fa-norm-stop` sparse adapter
|
||||
(`src/infrastructure/embedding/`), wired into the upload path behind
|
||||
`INGESTION_MAX_CONCURRENCY` (`503`), `INGESTION_TIMEOUT_SECONDS` (`504`), and
|
||||
the chunk-count ceiling (`413`). The embedding configuration is **ported from
|
||||
the `emet` evaluation lab** (`~/code/talie/emet`), which benchmarked these
|
||||
models and analyzers on the real Farsi corpus — the analyzer and BM25 weights
|
||||
are verified token-for-token against it, so treat them as a measured artifact
|
||||
and re-benchmark rather than tune them in place (ADR-0005). Not built yet: Qdrant collection bootstrap,
|
||||
Qdrant point upserts (so uploaded chunks are parsed/embedded but not yet
|
||||
searchable), and `src/agent/`. That maps to plan 001 Phases 1-4 done, Phase 5
|
||||
not started.
|
||||
|
||||
Architecture decisions live in `docs/adr/` (18 ADRs plus the 0000 template;
|
||||
0001–0004 are `Accepted` — 0004 amended by 0018; 0014 is `Superseded by 0017`;
|
||||
|
||||
Reference in New Issue
Block a user