feat(ingestion): index embedded chunks into Qdrant on upload

Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
  — chunks were parsed and embedded but never written to Qdrant, so nothing
  was actually searchable after upload.

Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
  same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
  (error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
  chunks_indexed now reflect the real indexing result instead of a hardcoded
  zero.
- Wired PointStorage through AppResources/lifespan/the files router.

Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
This commit is contained in:
Ali Zarinkolah
2026-08-20 18:17:39 +03:30
parent d00d436e5c
commit cc915f0f1a
11 changed files with 304 additions and 24 deletions

View File

@@ -125,9 +125,12 @@ them:
- Use `(tenant_id, domain, content_sha256)` to recognize identical uploads.
- An identical active upload should return the existing source-file/job reference
rather than create a duplicate ingestion.
- A changed upload creates a new ingestion job. Existing active Qdrant points are
replaced only after the new job completes successfully, so a failed re-ingestion
does not remove a working index.
- A changed upload creates a new ingestion job. A failed re-ingestion never
removes a working index: the soft-delete sweep for a shortened file runs only
after every upsert has succeeded. Because ADR-0001's point ids are
deterministic, upserts overwrite in place, so an interrupted attempt can leave
a prefix updated — it cannot empty or partially delete the index, and a retry
converges. See ADR-0017, "Re-running an ingestion stays safe".
- Preserve the original filename in Postgres metadata. MinIO object keys remain
internal ID-based paths.
@@ -272,6 +275,11 @@ code and a terminal job row.
tenant-filtered upserts, terminal state persistence, retrying an upload, and
parser/Qdrant failure handling.
The `chunks` collection itself is provisioned by a deployment step —
`uv run python -m src.cli.qdrant_bootstrap` — not by FastAPI startup, for the
same reason ADR-0009 keeps Alembic out of startup and ADR-0012 makes LangGraph's
`.setup()` a deployment step. See ADR-0001, "Collection provisioning".
**Exit criteria:** a successful upload returns `201` with a terminal status, and
its points are retrievable only under the owning tenant's Qdrant filter. A forced
failure mid-ingestion produces a `failed` job and the right HTTP status, and