feat(ingestion): index embedded chunks into Qdrant on upload

Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
  — chunks were parsed and embedded but never written to Qdrant, so nothing
  was actually searchable after upload.

Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
  same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
  (error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
  chunks_indexed now reflect the real indexing result instead of a hardcoded
  zero.
- Wired PointStorage through AppResources/lifespan/the files router.

Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
This commit is contained in:
Ali Zarinkolah
2026-08-20 18:17:39 +03:30
parent d00d436e5c
commit cc915f0f1a
11 changed files with 304 additions and 24 deletions

View File

@@ -18,10 +18,14 @@ the chunk-count ceiling (`413`). The embedding configuration is **ported from
the `emet` evaluation lab** (`~/code/talie/emet`), which benchmarked these
models and analyzers on the real Farsi corpus — the analyzer and BM25 weights
are verified token-for-token against it, so treat them as a measured artifact
and re-benchmark rather than tune them in place (ADR-0005). Not built yet: Qdrant collection bootstrap,
Qdrant point upserts (so uploaded chunks are parsed/embedded but not yet
searchable), and `src/agent/`. That maps to plan 001 Phases 1-4 done, Phase 5
not started.
and re-benchmark rather than tune them in place (ADR-0005). Also working: the
`chunks` collection bootstrap (`src/infrastructure/qdrant/collection.py`, run as
a deployment step via `uv run python -m src.cli.qdrant_bootstrap` — never at
startup) and tenant-scoped point upserts (`src/application/points/` behind the
`PointStorage` port), so an upload is searchable by the time `201` returns. Not
built yet: `/v1/points` CRUD and keyword search (plan 002), `tenant_domains`
validation of the `domain` field, and `src/agent/`. That maps to plan 001
Phases 1-5 done.
Architecture decisions live in `docs/adr/` (18 ADRs plus the 0000 template;
0001–0004 are `Accepted` — 0004 amended by 0018; 0014 is `Superseded by 0017`;
@@ -140,9 +144,13 @@ as its only caller-facing entry point. It dispatches on source type and owns
the `anyio.to_thread.run_sync` + `CapacityLimiter` offload ADR-0017 requires;
`parse_docx`/`parse_csv`/`parse_xlsx`/`chunk_document` stay in the package,
exported mainly for their own tests, not for outside callers to reach for
directly. Follow this pattern in `application/` as new packages are added
there — `points/`, `retrieval/`, `threads/` — rather than exposing their
internals as the primary surface.
directly. `src/application/points/` follows the same shape: `index_chunks` is
the only caller-facing entry point, owning payload construction, batching,
the `upsert_concurrency` semaphore, and the ordering rule that the soft-delete
sweep runs only after every upsert succeeds; `build_chunk_payload` stays
internal. Follow this pattern in `application/` as new packages are added
there — `retrieval/`, `threads/` — rather than exposing their internals as the
primary surface.
### Resource lifetime rules (ADR-0012)