feat(qdrant): provision the chunks collection as an explicit deployment step

Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
  sparse, late_interaction) and payload indexes defined at creation time per
  ADR-0001; sparse/multivector fields cannot be added to an existing
  collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
  ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
  setup(), so it is a deployment step instead.

Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
  idempotent and schema-verifying (raises on dimension/modifier mismatch
  rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
  (python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.

Impact:
- Deployments must run the new bootstrap command before the first upload;
  see ADR-0001's new "Collection provisioning" section.
This commit is contained in:
Ali Zarinkolah
2026-08-20 18:15:21 +03:30
parent e8fb41af87
commit 5e0addcc55
8 changed files with 456 additions and 2 deletions

View File

@@ -108,6 +108,38 @@ measurement rather than assumption:
- Payload index on `previous_chunk_id` / `next_chunk_id`: keyword index,
used for O(1) adjacency retrieval (see below).
### Collection provisioning
The collection is created by an explicit **deployment step**, not by application
startup and not lazily on first write:
uv run python -m src.cli.qdrant_bootstrap
Creating a collection is DDL, and this project already keeps DDL out of the boot
and request paths: [0009](0009-postgres-sqlalchemy-alembic-schema.md) requires
Alembic for Postgres schema and forbids `create_all()` at startup, and
[0012](0012-application-resource-lifetime-and-dependency-ownership.md) makes
LangGraph's `.setup()` a deployment step for the same reason. Neither ADR named
Qdrant explicitly; this section closes that gap rather than letting the placement
be decided by whichever code happened to need it first.
Doing it in the FastAPI lifespan was rejected: it couples process boot to Qdrant
being reachable (which is `/readyz`'s job, not boot's), races across replicas,
and turns a misconfigured collection into a silent skip. Doing it lazily on first
upsert was rejected for putting DDL on a user request and hiding the
misconfiguration until traffic arrives.
`ensure_chunks_collection` is idempotent and **verifying**: against an existing
collection it compares the dense dimensions and the sparse `modifier` to the
pinned values and fails loudly on divergence. That check is the point of making
the step explicit — both properties degrade silently in production if wrong (a
missing `modifier="idf"` produces no error, just unweighted lexical retrieval).
Payload indexes are (re)created on every run, since unlike vector configuration
they can be added to a live collection. The full-text index on `content` is
therefore deferred to the keyword-search work in
[0002](0002-chunk-crud-and-search-api.md), not created here.
### Payload schema
This schema is now decided for the fields below. Additional document-context
@@ -124,7 +156,7 @@ involves format-specific tradeoffs not yet made.
| `chunk_id` | keyword | stable identifier for a single chunk |
| `content_type` | keyword | classification of the chunk's content; exact value set (e.g. `paragraph`, `table_row`, `heading`) to be finalized alongside the chunking-strategy ADR |
| `source_filename` | keyword | original uploaded filename |
| `source_type` | keyword (`docx` \| `csv`) | which parser produced this chunk |
| `source_type` | keyword (`docx` \| `xlsx` \| `csv`) | which parser produced this chunk — `xlsx` added by [0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md) |
| `order_id` | float (see below) | chunk's *display* position within the file; mutable so the backend can reorder/insert chunks |
| `chunk_index` | integer | chunk's *original ingestion* ordinal — immutable, used to derive the deterministic point ID below (kept separate from `order_id` precisely because `order_id` can change) |
| `previous_chunk_id` | keyword, nullable | `chunk_id` of the preceding chunk in display order (`null` for the first chunk in a file) — O(1) adjacency pointer for context-window expansion in ADR-0003 |
@@ -136,7 +168,7 @@ involves format-specific tradeoffs not yet made.
| `updated_at` | datetime | last modification timestamp |
| `created_by` | keyword | user/service that created the chunk |
| `updated_by` | keyword | user/service that last modified the chunk |
| `version` | integer | optimistic-concurrency counter, used in ADR-0002 |
| `version` | integer | optimistic-concurrency counter, used in ADR-0002. Ingestion currently writes `1` unconditionally: the read-check-write that makes the guard meaningful costs one read per point and belongs with the `/v1/points` write paths, so plan 002 owns it. Safe while ingestion is the only writer of a file's points; it would clobber a concurrent manual edit's counter once `/v1/points` ships. |
| `content_hash` | keyword | hash of the chunk's raw text; lets re-ingestion detect unchanged content and skip re-embedding it |
| `embedding_model_version` | keyword | identifies which embedding model(s) produced this chunk's vectors; needed to know which chunks require re-embedding after a future model swap |