feat(ingestion): index embedded chunks into Qdrant on upload
Why: - POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally — chunks were parsed and embedded but never written to Qdrant, so nothing was actually searchable after upload. Changes: - upload_source_file() now calls index_chunks() after embedding, inside the same INGESTION_TIMEOUT_SECONDS window, and marks the job failed (error_code=index_failed, 502) if it raises. - Job counters (points_created, points_soft_deleted) and the response's chunks_indexed now reflect the real indexing result instead of a hardcoded zero. - Wired PointStorage through AppResources/lifespan/the files router. Impact: - A successful upload is now searchable in Qdrant by the time 201 returns.
This commit is contained in:
22
CLAUDE.md
22
CLAUDE.md
@@ -18,10 +18,14 @@ the chunk-count ceiling (`413`). The embedding configuration is **ported from
|
||||
the `emet` evaluation lab** (`~/code/talie/emet`), which benchmarked these
|
||||
models and analyzers on the real Farsi corpus — the analyzer and BM25 weights
|
||||
are verified token-for-token against it, so treat them as a measured artifact
|
||||
and re-benchmark rather than tune them in place (ADR-0005). Not built yet: Qdrant collection bootstrap,
|
||||
Qdrant point upserts (so uploaded chunks are parsed/embedded but not yet
|
||||
searchable), and `src/agent/`. That maps to plan 001 Phases 1-4 done, Phase 5
|
||||
not started.
|
||||
and re-benchmark rather than tune them in place (ADR-0005). Also working: the
|
||||
`chunks` collection bootstrap (`src/infrastructure/qdrant/collection.py`, run as
|
||||
a deployment step via `uv run python -m src.cli.qdrant_bootstrap` — never at
|
||||
startup) and tenant-scoped point upserts (`src/application/points/` behind the
|
||||
`PointStorage` port), so an upload is searchable by the time `201` returns. Not
|
||||
built yet: `/v1/points` CRUD and keyword search (plan 002), `tenant_domains`
|
||||
validation of the `domain` field, and `src/agent/`. That maps to plan 001
|
||||
Phases 1-5 done.
|
||||
|
||||
Architecture decisions live in `docs/adr/` (18 ADRs plus the 0000 template;
|
||||
0001–0004 are `Accepted` — 0004 amended by 0018; 0014 is `Superseded by 0017`;
|
||||
@@ -140,9 +144,13 @@ as its only caller-facing entry point. It dispatches on source type and owns
|
||||
the `anyio.to_thread.run_sync` + `CapacityLimiter` offload ADR-0017 requires;
|
||||
`parse_docx`/`parse_csv`/`parse_xlsx`/`chunk_document` stay in the package,
|
||||
exported mainly for their own tests, not for outside callers to reach for
|
||||
directly. Follow this pattern in `application/` as new packages are added
|
||||
there — `points/`, `retrieval/`, `threads/` — rather than exposing their
|
||||
internals as the primary surface.
|
||||
directly. `src/application/points/` follows the same shape: `index_chunks` is
|
||||
the only caller-facing entry point, owning payload construction, batching,
|
||||
the `upsert_concurrency` semaphore, and the ordering rule that the soft-delete
|
||||
sweep runs only after every upsert succeeds; `build_chunk_payload` stays
|
||||
internal. Follow this pattern in `application/` as new packages are added
|
||||
there — `retrieval/`, `threads/` — rather than exposing their internals as the
|
||||
primary surface.
|
||||
|
||||
### Resource lifetime rules (ADR-0012)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user