feat(ingestion): index embedded chunks into Qdrant on upload
Why: - POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally — chunks were parsed and embedded but never written to Qdrant, so nothing was actually searchable after upload. Changes: - upload_source_file() now calls index_chunks() after embedding, inside the same INGESTION_TIMEOUT_SECONDS window, and marks the job failed (error_code=index_failed, 502) if it raises. - Job counters (points_created, points_soft_deleted) and the response's chunks_indexed now reflect the real indexing result instead of a hardcoded zero. - Wired PointStorage through AppResources/lifespan/the files router. Impact: - A successful upload is now searchable in Qdrant by the time 201 returns.
This commit is contained in:
@@ -159,8 +159,25 @@ retry, and phase 2 has no transaction protecting it:
|
||||
return the existing file/job rather than re-ingesting (plan 001).
|
||||
- `tenant_id` comes from `AuthContext`, never from the request body.
|
||||
- A terminal job is never transitioned back to `running`.
|
||||
- Qdrant points from a failed attempt do not replace the previous successful
|
||||
index; replacement happens only after a successful attempt.
|
||||
- A failed attempt never *removes* content from a working index. The
|
||||
soft-delete sweep that retires a shortened file's leftover points runs only
|
||||
after every upsert in the attempt has succeeded.
|
||||
|
||||
This is deliberately weaker than "replacement happens only after a successful
|
||||
attempt", which an earlier revision of this ADR claimed. That guarantee is not
|
||||
achievable alongside ADR-0001's deterministic point ids: those ids are exactly
|
||||
what makes a retry idempotent, and they also mean a re-ingestion overwrites
|
||||
points **in place**, so a crash partway through leaves a prefix updated and the
|
||||
remainder still on the old content. Buying literal atomicity would mean
|
||||
generation-suffixed ids and an activation flip, which contradicts ADR-0001 and
|
||||
ADR-0002's stable point ids. Staging the new points as `is_active=false` and
|
||||
flipping them on success is strictly worse — the in-place overwrite would
|
||||
deactivate the previously live points, silently emptying a working index if the
|
||||
attempt were interrupted.
|
||||
|
||||
What holds instead: the index is never emptied, never partially deleted, and a
|
||||
retry converges — deterministic ids rewrite every point and the sweep re-runs,
|
||||
reaching the exact correct state.
|
||||
|
||||
### Failures are HTTP failures
|
||||
|
||||
|
||||
@@ -125,9 +125,12 @@ them:
|
||||
- Use `(tenant_id, domain, content_sha256)` to recognize identical uploads.
|
||||
- An identical active upload should return the existing source-file/job reference
|
||||
rather than create a duplicate ingestion.
|
||||
- A changed upload creates a new ingestion job. Existing active Qdrant points are
|
||||
replaced only after the new job completes successfully, so a failed re-ingestion
|
||||
does not remove a working index.
|
||||
- A changed upload creates a new ingestion job. A failed re-ingestion never
|
||||
removes a working index: the soft-delete sweep for a shortened file runs only
|
||||
after every upsert has succeeded. Because ADR-0001's point ids are
|
||||
deterministic, upserts overwrite in place, so an interrupted attempt can leave
|
||||
a prefix updated — it cannot empty or partially delete the index, and a retry
|
||||
converges. See ADR-0017, "Re-running an ingestion stays safe".
|
||||
- Preserve the original filename in Postgres metadata. MinIO object keys remain
|
||||
internal ID-based paths.
|
||||
|
||||
@@ -272,6 +275,11 @@ code and a terminal job row.
|
||||
tenant-filtered upserts, terminal state persistence, retrying an upload, and
|
||||
parser/Qdrant failure handling.
|
||||
|
||||
The `chunks` collection itself is provisioned by a deployment step —
|
||||
`uv run python -m src.cli.qdrant_bootstrap` — not by FastAPI startup, for the
|
||||
same reason ADR-0009 keeps Alembic out of startup and ADR-0012 makes LangGraph's
|
||||
`.setup()` a deployment step. See ADR-0001, "Collection provisioning".
|
||||
|
||||
**Exit criteria:** a successful upload returns `201` with a terminal status, and
|
||||
its points are retrievable only under the owning tenant's Qdrant filter. A forced
|
||||
failure mid-ingestion produces a `failed` job and the right HTTP status, and
|
||||
|
||||
Reference in New Issue
Block a user