feat(qdrant): provision the chunks collection as an explicit deployment step
Why: - The chunks collection needs four named vectors (dense_nomic, dense_openai, sparse, late_interaction) and payload indexes defined at creation time per ADR-0001; sparse/multivector fields cannot be added to an existing collection without recreating it, so schema drift here is expensive. - Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's setup(), so it is a deployment step instead. Changes: - src/infrastructure/qdrant/collection.py: ensure_chunks_collection(), idempotent and schema-verifying (raises on dimension/modifier mismatch rather than silently accepting a misconfigured collection). - src/cli/qdrant_bootstrap.py: the operator entry point (python -m src.cli.qdrant_bootstrap). - QdrantSettings gains collection/upsert_batch_size/upsert_concurrency. Impact: - Deployments must run the new bootstrap command before the first upload; see ADR-0001's new "Collection provisioning" section.
This commit is contained in:
@@ -108,6 +108,38 @@ measurement rather than assumption:
|
||||
- Payload index on `previous_chunk_id` / `next_chunk_id`: keyword index,
|
||||
used for O(1) adjacency retrieval (see below).
|
||||
|
||||
### Collection provisioning
|
||||
|
||||
The collection is created by an explicit **deployment step**, not by application
|
||||
startup and not lazily on first write:
|
||||
|
||||
uv run python -m src.cli.qdrant_bootstrap
|
||||
|
||||
Creating a collection is DDL, and this project already keeps DDL out of the boot
|
||||
and request paths: [0009](0009-postgres-sqlalchemy-alembic-schema.md) requires
|
||||
Alembic for Postgres schema and forbids `create_all()` at startup, and
|
||||
[0012](0012-application-resource-lifetime-and-dependency-ownership.md) makes
|
||||
LangGraph's `.setup()` a deployment step for the same reason. Neither ADR named
|
||||
Qdrant explicitly; this section closes that gap rather than letting the placement
|
||||
be decided by whichever code happened to need it first.
|
||||
|
||||
Doing it in the FastAPI lifespan was rejected: it couples process boot to Qdrant
|
||||
being reachable (which is `/readyz`'s job, not boot's), races across replicas,
|
||||
and turns a misconfigured collection into a silent skip. Doing it lazily on first
|
||||
upsert was rejected for putting DDL on a user request and hiding the
|
||||
misconfiguration until traffic arrives.
|
||||
|
||||
`ensure_chunks_collection` is idempotent and **verifying**: against an existing
|
||||
collection it compares the dense dimensions and the sparse `modifier` to the
|
||||
pinned values and fails loudly on divergence. That check is the point of making
|
||||
the step explicit — both properties degrade silently in production if wrong (a
|
||||
missing `modifier="idf"` produces no error, just unweighted lexical retrieval).
|
||||
|
||||
Payload indexes are (re)created on every run, since unlike vector configuration
|
||||
they can be added to a live collection. The full-text index on `content` is
|
||||
therefore deferred to the keyword-search work in
|
||||
[0002](0002-chunk-crud-and-search-api.md), not created here.
|
||||
|
||||
### Payload schema
|
||||
|
||||
This schema is now decided for the fields below. Additional document-context
|
||||
@@ -124,7 +156,7 @@ involves format-specific tradeoffs not yet made.
|
||||
| `chunk_id` | keyword | stable identifier for a single chunk |
|
||||
| `content_type` | keyword | classification of the chunk's content; exact value set (e.g. `paragraph`, `table_row`, `heading`) to be finalized alongside the chunking-strategy ADR |
|
||||
| `source_filename` | keyword | original uploaded filename |
|
||||
| `source_type` | keyword (`docx` \| `csv`) | which parser produced this chunk |
|
||||
| `source_type` | keyword (`docx` \| `xlsx` \| `csv`) | which parser produced this chunk — `xlsx` added by [0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md) |
|
||||
| `order_id` | float (see below) | chunk's *display* position within the file; mutable so the backend can reorder/insert chunks |
|
||||
| `chunk_index` | integer | chunk's *original ingestion* ordinal — immutable, used to derive the deterministic point ID below (kept separate from `order_id` precisely because `order_id` can change) |
|
||||
| `previous_chunk_id` | keyword, nullable | `chunk_id` of the preceding chunk in display order (`null` for the first chunk in a file) — O(1) adjacency pointer for context-window expansion in ADR-0003 |
|
||||
@@ -136,7 +168,7 @@ involves format-specific tradeoffs not yet made.
|
||||
| `updated_at` | datetime | last modification timestamp |
|
||||
| `created_by` | keyword | user/service that created the chunk |
|
||||
| `updated_by` | keyword | user/service that last modified the chunk |
|
||||
| `version` | integer | optimistic-concurrency counter, used in ADR-0002 |
|
||||
| `version` | integer | optimistic-concurrency counter, used in ADR-0002. Ingestion currently writes `1` unconditionally: the read-check-write that makes the guard meaningful costs one read per point and belongs with the `/v1/points` write paths, so plan 002 owns it. Safe while ingestion is the only writer of a file's points; it would clobber a concurrent manual edit's counter once `/v1/points` ships. |
|
||||
| `content_hash` | keyword | hash of the chunk's raw text; lets re-ingestion detect unchanged content and skip re-embedding it |
|
||||
| `embedding_model_version` | keyword | identifies which embedding model(s) produced this chunk's vectors; needed to know which chunks require re-embedding after a future model swap |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user