feat(ingestion): index embedded chunks into Qdrant on upload

Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
  — chunks were parsed and embedded but never written to Qdrant, so nothing
  was actually searchable after upload.

Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
  same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
  (error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
  chunks_indexed now reflect the real indexing result instead of a hardcoded
  zero.
- Wired PointStorage through AppResources/lifespan/the files router.

Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
This commit is contained in:
Ali Zarinkolah
2026-08-20 18:17:39 +03:30
parent d00d436e5c
commit cc915f0f1a
11 changed files with 304 additions and 24 deletions

View File

@@ -159,8 +159,25 @@ retry, and phase 2 has no transaction protecting it:
return the existing file/job rather than re-ingesting (plan 001).
- `tenant_id` comes from `AuthContext`, never from the request body.
- A terminal job is never transitioned back to `running`.
- Qdrant points from a failed attempt do not replace the previous successful
index; replacement happens only after a successful attempt.
- A failed attempt never *removes* content from a working index. The
soft-delete sweep that retires a shortened file's leftover points runs only
after every upsert in the attempt has succeeded.
This is deliberately weaker than "replacement happens only after a successful
attempt", which an earlier revision of this ADR claimed. That guarantee is not
achievable alongside ADR-0001's deterministic point ids: those ids are exactly
what makes a retry idempotent, and they also mean a re-ingestion overwrites
points **in place**, so a crash partway through leaves a prefix updated and the
remainder still on the old content. Buying literal atomicity would mean
generation-suffixed ids and an activation flip, which contradicts ADR-0001 and
ADR-0002's stable point ids. Staging the new points as `is_active=false` and
flipping them on success is strictly worse — the in-place overwrite would
deactivate the previously live points, silently emptying a working index if the
attempt were interrupted.
What holds instead: the index is never emptied, never partially deleted, and a
retry converges — deterministic ids rewrite every point and the sweep re-runs,
reaching the exact correct state.
### Failures are HTTP failures