feat(qdrant): provision the chunks collection as an explicit deployment step

Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
  sparse, late_interaction) and payload indexes defined at creation time per
  ADR-0001; sparse/multivector fields cannot be added to an existing
  collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
  ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
  setup(), so it is a deployment step instead.

Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
  idempotent and schema-verifying (raises on dimension/modifier mismatch
  rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
  (python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.

Impact:
- Deployments must run the new bootstrap command before the first upload;
  see ADR-0001's new "Collection provisioning" section.

.env.example

@@ -44,6 +44,9 @@ INGESTION_EMBED_CONCURRENCY=4
 # Qdrant
 QDRANT_URL=http://127.0.0.1:6343
 QDRANT_API_KEY=
+QDRANT_COLLECTION=chunks
+QDRANT_UPSERT_BATCH_SIZE=128
+QDRANT_UPSERT_CONCURRENCY=4
 
 # Dense embedders (ADR-0001). Both speak an OpenAI-compatible /embeddings
 # endpoint, so one adapter serves both. Models and endpoints are the ones the

docs/adr/0001-ingestion-pipeline-and-collection-schema.md

@@ -108,6 +108,38 @@ measurement rather than assumption:
 - Payload index on `previous_chunk_id` / `next_chunk_id`: keyword index,
   used for O(1) adjacency retrieval (see below).
 
+### Collection provisioning
+
+The collection is created by an explicit **deployment step**, not by application
+startup and not lazily on first write:
+
+    uv run python -m src.cli.qdrant_bootstrap
+
+Creating a collection is DDL, and this project already keeps DDL out of the boot
+and request paths: [0009](0009-postgres-sqlalchemy-alembic-schema.md) requires
+Alembic for Postgres schema and forbids `create_all()` at startup, and
+[0012](0012-application-resource-lifetime-and-dependency-ownership.md) makes
+LangGraph's `.setup()` a deployment step for the same reason. Neither ADR named
+Qdrant explicitly; this section closes that gap rather than letting the placement
+be decided by whichever code happened to need it first.
+
+Doing it in the FastAPI lifespan was rejected: it couples process boot to Qdrant
+being reachable (which is `/readyz`'s job, not boot's), races across replicas,
+and turns a misconfigured collection into a silent skip. Doing it lazily on first
+upsert was rejected for putting DDL on a user request and hiding the
+misconfiguration until traffic arrives.
+
+`ensure_chunks_collection` is idempotent and **verifying**: against an existing
+collection it compares the dense dimensions and the sparse `modifier` to the
+pinned values and fails loudly on divergence. That check is the point of making
+the step explicit — both properties degrade silently in production if wrong (a
+missing `modifier="idf"` produces no error, just unweighted lexical retrieval).
+
+Payload indexes are (re)created on every run, since unlike vector configuration
+they can be added to a live collection. The full-text index on `content` is
+therefore deferred to the keyword-search work in
+[0002](0002-chunk-crud-and-search-api.md), not created here.
+
 ### Payload schema
 
 This schema is now decided for the fields below. Additional document-context
@@ -124,7 +156,7 @@ involves format-specific tradeoffs not yet made.
 | `chunk_id` | keyword | stable identifier for a single chunk |
 | `content_type` | keyword | classification of the chunk's content; exact value set (e.g. `paragraph`, `table_row`, `heading`) to be finalized alongside the chunking-strategy ADR |
 | `source_filename` | keyword | original uploaded filename |
-| `source_type` | keyword (`docx` \| `csv`) | which parser produced this chunk |
+| `source_type` | keyword (`docx` \| `xlsx` \| `csv`) | which parser produced this chunk — `xlsx` added by [0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md) |
 | `order_id` | float (see below) | chunk's *display* position within the file; mutable so the backend can reorder/insert chunks |
 | `chunk_index` | integer | chunk's *original ingestion* ordinal — immutable, used to derive the deterministic point ID below (kept separate from `order_id` precisely because `order_id` can change) |
 | `previous_chunk_id` | keyword, nullable | `chunk_id` of the preceding chunk in display order (`null` for the first chunk in a file) — O(1) adjacency pointer for context-window expansion in ADR-0003 |
@@ -136,7 +168,7 @@ involves format-specific tradeoffs not yet made.
 | `updated_at` | datetime | last modification timestamp |
 | `created_by` | keyword | user/service that created the chunk |
 | `updated_by` | keyword | user/service that last modified the chunk |
-| `version` | integer | optimistic-concurrency counter, used in ADR-0002 |
+| `version` | integer | optimistic-concurrency counter, used in ADR-0002. Ingestion currently writes `1` unconditionally: the read-check-write that makes the guard meaningful costs one read per point and belongs with the `/v1/points` write paths, so plan 002 owns it. Safe while ingestion is the only writer of a file's points; it would clobber a concurrent manual edit's counter once `/v1/points` ships. |
 | `content_hash` | keyword | hash of the chunk's raw text; lets re-ingestion detect unchanged content and skip re-embedding it |
 | `embedding_model_version` | keyword | identifies which embedding model(s) produced this chunk's vectors; needed to know which chunks require re-embedding after a future model swap |
 

src/cli/__init__.py

@@ -0,0 +1 @@
+"""Operator entry points that run as deployment steps, not at app startup."""

src/cli/qdrant_bootstrap.py

@@ -0,0 +1,54 @@
+"""Create the `chunks` collection — the Qdrant analogue of `alembic upgrade head`.
+
+    uv run python -m src.cli.qdrant_bootstrap
+
+A deployment step, deliberately not part of the FastAPI lifespan: collection
+creation is DDL, which ADR-0009 keeps out of application startup for Postgres
+and ADR-0012 keeps out of it for LangGraph's `.setup()`. See
+`src/infrastructure/qdrant/collection.py` for the full reasoning.
+
+Idempotent and safe to re-run. Exits non-zero if an existing collection
+diverges from the pinned schema, rather than leaving a silently degraded
+sparse index behind.
+"""
+
+import asyncio
+import sys
+
+import structlog
+
+from src.config import Settings
+from src.infrastructure.observability.logging import configure_logging
+from src.infrastructure.qdrant.client import create_client
+from src.infrastructure.qdrant.collection import (
+    CollectionSchemaMismatchError,
+    ensure_chunks_collection,
+)
+
+logger = structlog.get_logger(__name__)
+
+
+async def bootstrap(settings: Settings | None = None) -> int:
+    resolved = settings or Settings()
+    configure_logging(resolved.logging)
+    client = create_client(resolved.qdrant)
+    try:
+        created = await ensure_chunks_collection(client, collection=resolved.qdrant.collection)
+    except CollectionSchemaMismatchError as exc:
+        logger.error("qdrant.bootstrap.schema_mismatch", error=str(exc))
+        return 1
+    finally:
+        await client.close()
+
+    logger.info(
+        "qdrant.bootstrap.completed", collection=resolved.qdrant.collection, created=created
+    )
+    return 0
+
+
+def main() -> None:
+    sys.exit(asyncio.run(bootstrap()))
+
+
+if __name__ == "__main__":
+    main()

src/config.py

@@ -109,12 +109,28 @@ class ChunkingSettings(BaseSettings):
 
 
 class QdrantSettings(BaseSettings):
+    """Qdrant connection and bulk-upsert bounds (ADR-0001, ADR-0017).
+
+    `collection` names the single shared collection all tenants live in
+    (ADR-0001); it is deliberately configurable so tests can point at a
+    disposable one. The vector *dimensions* are not settings -- they are model
+    facts pinned in `src/infrastructure/qdrant/collection.py`, and changing one
+    is a re-embedding migration.
+
+    `upsert_batch_size` sits inside ADR-0001's 64-256 bulk-upload band, and
+    `upsert_concurrency` bounds in-flight batches so ingestion issues parallel
+    streams rather than an unbounded `gather` (ADR-0017).
+    """
+
     model_config = SettingsConfigDict(
         env_prefix="QDRANT_", extra="ignore", env_file=".env", env_ignore_empty=True
     )
 
     url: str = "http://127.0.0.1:6343"
     api_key: str | None = None
+    collection: str = "chunks"
+    upsert_batch_size: int = 128
+    upsert_concurrency: int = 4
 
 
 class NomicEmbeddingSettings(BaseSettings):

src/infrastructure/qdrant/collection.py

@@ -0,0 +1,182 @@
+"""The `chunks` collection schema and its provisioning (ADR-0001).
+
+**This runs as a deployment step, never at FastAPI startup.** Creating a
+collection is DDL, and the project already rules DDL out of the request/boot
+path: ADR-0009 requires Alembic for Postgres schema, and ADR-0012 makes
+LangGraph's `.setup()` a deployment step. Doing it in the lifespan would also
+couple boot to Qdrant being reachable (that is `/readyz`'s job), race across
+replicas, and hide a misconfigured collection until traffic arrives.
+
+Entry point for operators: `uv run python -m src.cli.qdrant_bootstrap`.
+
+Two properties of this schema are load-bearing and fail *silently* if wrong,
+which is why `ensure_chunks_collection` verifies rather than skips:
+
+- **`sparse` must carry `modifier=IDF`.** `src/infrastructure/embedding/bm25.py`
+  computes only BM25's term-frequency saturation; Qdrant supplies IDF from
+  collection-wide statistics. Without the modifier there is no error and no
+  warning — lexical retrieval just quietly loses its IDF term (ADR-0005).
+- **The dense dimensions are pinned**: `dense_nomic` 768,
+  `dense_openai` 3072 (native, `dimensions` deliberately unset). They are
+  constants here rather than settings because they are model facts; changing
+  one is a re-embedding migration, not a config tweak (ADR-0001).
+
+All four named vectors are defined at creation even though `late_interaction`
+stays unpopulated until ADR-0003's rerank work (ADR-0017 does not compute it at
+ingest). Sparse and multivector fields cannot be added to an existing
+collection without recreating it, so deferring them is the one thing this
+schema cannot afford.
+"""
+
+import structlog
+from qdrant_client import AsyncQdrantClient, models
+
+logger = structlog.get_logger(__name__)
+
+DENSE_NOMIC_VECTOR = "dense_nomic"
+DENSE_OPENAI_VECTOR = "dense_openai"
+SPARSE_VECTOR = "sparse"
+LATE_INTERACTION_VECTOR = "late_interaction"
+
+DENSE_NOMIC_DIMENSIONS = 768
+DENSE_OPENAI_DIMENSIONS = 3072
+# jina-colbert-v2's per-token output dimension (ADR-0005).
+LATE_INTERACTION_DIMENSIONS = 128
+
+
+class CollectionSchemaMismatchError(RuntimeError):
+    """An existing collection does not match the schema this code expects.
+
+    Raised loudly instead of returning: silently accepting a collection whose
+    dense size or sparse modifier differs is precisely the failure this
+    explicit bootstrap step exists to prevent.
+    """
+
+
+def _vectors_config() -> dict[str, models.VectorParams]:
+    return {
+        DENSE_NOMIC_VECTOR: models.VectorParams(
+            size=DENSE_NOMIC_DIMENSIONS, distance=models.Distance.COSINE
+        ),
+        DENSE_OPENAI_VECTOR: models.VectorParams(
+            size=DENSE_OPENAI_DIMENSIONS, distance=models.Distance.COSINE
+        ),
+        # Rerank-only: never independently ANN-searched, so its HNSW graph is
+        # disabled (m=0), and stored on disk so its larger footprint does not
+        # degrade dense/sparse query latency (ADR-0001).
+        LATE_INTERACTION_VECTOR: models.VectorParams(
+            size=LATE_INTERACTION_DIMENSIONS,
+            distance=models.Distance.COSINE,
+            multivector_config=models.MultiVectorConfig(
+                comparator=models.MultiVectorComparator.MAX_SIM
+            ),
+            hnsw_config=models.HnswConfigDiff(m=0),
+            on_disk=True,
+        ),
+    }
+
+
+def _sparse_vectors_config() -> dict[str, models.SparseVectorParams]:
+    return {SPARSE_VECTOR: models.SparseVectorParams(modifier=models.Modifier.IDF)}
+
+
+# (field name, schema). `order_id` is float because Qdrant's `Range` conditions
+# and `order_by` only support numeric/datetime payloads -- a keyword key could
+# only be sorted client-side after fetching every chunk (ADR-0001).
+#
+# The full-text index on `content` is deliberately absent: it belongs to plan
+# 002's keyword search, and payload indexes -- unlike vector config -- can be
+# added to a live collection later.
+_PAYLOAD_INDEXES: tuple[tuple[str, models.PayloadSchemaType | models.KeywordIndexParams], ...] = (
+    # `is_tenant` co-locates a tenant's vectors on disk for sequential reads,
+    # which is the whole point of payload-partitioned multitenancy.
+    (
+        "tenant_id",
+        models.KeywordIndexParams(type=models.KeywordIndexType.KEYWORD, is_tenant=True),
+    ),
+    ("domain", models.PayloadSchemaType.KEYWORD),
+    ("file_id", models.PayloadSchemaType.KEYWORD),
+    ("order_id", models.PayloadSchemaType.FLOAT),
+    ("previous_chunk_id", models.PayloadSchemaType.KEYWORD),
+    ("next_chunk_id", models.PayloadSchemaType.KEYWORD),
+)
+
+
+def _verify_existing(collection: str, info: models.CollectionInfo) -> None:
+    params = info.config.params
+    vectors = params.vectors
+    if not isinstance(vectors, dict):
+        raise CollectionSchemaMismatchError(
+            f"collection {collection!r} has an unnamed dense vector; ADR-0001 requires "
+            f"named vectors and this cannot be fixed without recreating the collection"
+        )
+
+    for name, expected_size in (
+        (DENSE_NOMIC_VECTOR, DENSE_NOMIC_DIMENSIONS),
+        (DENSE_OPENAI_VECTOR, DENSE_OPENAI_DIMENSIONS),
+        (LATE_INTERACTION_VECTOR, LATE_INTERACTION_DIMENSIONS),
+    ):
+        existing = vectors.get(name)
+        if existing is None:
+            raise CollectionSchemaMismatchError(
+                f"collection {collection!r} is missing the {name!r} vector"
+            )
+        if existing.size != expected_size:
+            raise CollectionSchemaMismatchError(
+                f"collection {collection!r} has {name!r} at {existing.size} dimensions, "
+                f"expected {expected_size}; re-dimensioning is a re-embedding migration"
+            )
+
+    sparse = (params.sparse_vectors or {}).get(SPARSE_VECTOR)
+    if sparse is None:
+        raise CollectionSchemaMismatchError(
+            f"collection {collection!r} is missing the {SPARSE_VECTOR!r} vector; sparse "
+            f"vectors cannot be added without recreating the collection"
+        )
+    if sparse.modifier != models.Modifier.IDF:
+        raise CollectionSchemaMismatchError(
+            f"collection {collection!r} has {SPARSE_VECTOR!r} with modifier "
+            f"{sparse.modifier!r}, expected 'idf'; without it Qdrant applies no IDF "
+            f"and lexical retrieval silently degrades (ADR-0005)"
+        )
+
+
+async def ensure_chunks_collection(client: AsyncQdrantClient, *, collection: str) -> bool:
+    """Create the `chunks` collection and its payload indexes if absent.
+
+    Idempotent: an existing collection is verified against the pinned schema
+    and left alone. Returns whether it created the collection.
+
+    Raises `CollectionSchemaMismatchError` if an existing collection diverges.
+    """
+    if await client.collection_exists(collection):
+        _verify_existing(collection, await client.get_collection(collection))
+        logger.info("qdrant.collection.verified", collection=collection)
+        # Payload indexes are additive and idempotent, so (re)creating them
+        # here is what lets an index be added to an already-live collection.
+        await _create_payload_indexes(client, collection=collection)
+        return False
+
+    await client.create_collection(
+        collection_name=collection,
+        vectors_config=_vectors_config(),
+        sparse_vectors_config=_sparse_vectors_config(),
+        # m=0 disables the global index; payload_m builds per-tenant graphs
+        # instead, per Qdrant's multitenant guidance (ADR-0001).
+        hnsw_config=models.HnswConfigDiff(m=0, payload_m=16),
+    )
+    logger.info("qdrant.collection.created", collection=collection)
+    await _create_payload_indexes(client, collection=collection)
+    return True
+
+
+async def _create_payload_indexes(client: AsyncQdrantClient, *, collection: str) -> None:
+    for field_name, field_schema in _PAYLOAD_INDEXES:
+        await client.create_payload_index(
+            collection_name=collection, field_name=field_name, field_schema=field_schema
+        )
+    logger.info(
+        "qdrant.collection.payload_indexes.ensured",
+        collection=collection,
+        fields=[name for name, _ in _PAYLOAD_INDEXES],
+    )

tests/integration/qdrant/conftest.py

@@ -0,0 +1,52 @@
+import uuid
+from collections.abc import AsyncIterator, Iterator
+
+import pytest
+import pytest_asyncio
+from qdrant_client import AsyncQdrantClient
+from testcontainers.community.qdrant import QdrantContainer
+
+from src.config import QdrantSettings
+from src.infrastructure.qdrant.client import create_client
+
+# Pinned to match the `qdrant-client` major/minor in pyproject.toml. The
+# testcontainers default image trails it far enough that the client emits an
+# incompatibility warning, and testing against a version we do not deploy is
+# the wrong signal anyway.
+_QDRANT_IMAGE = "qdrant/qdrant:v1.19.0"
+
+
+@pytest.fixture(scope="session")
+def qdrant_container() -> Iterator[QdrantContainer]:
+    with QdrantContainer(image=_QDRANT_IMAGE) as container:
+        yield container
+
+
+@pytest.fixture(scope="session")
+def qdrant_url(qdrant_container: QdrantContainer) -> str:
+    """The container's REST URL, pinned to IPv4.
+
+    Same gotcha as postgres_url/minio_settings (see their conftests):
+    testcontainers reports the host as `localhost`, which resolves to `::1`
+    first, but Docker publishes the mapped port on IPv4 only. The IPv6 SYN is
+    dropped rather than refused, so the client hangs until its timeout instead
+    of falling back to the second address -- the connection does not fail, it
+    hangs.
+    """
+    host = qdrant_container.get_container_host_ip().replace("localhost", "127.0.0.1")
+    return f"http://{host}:{qdrant_container.get_exposed_port(6333)}"
+
+
+@pytest.fixture
+def qdrant_settings(qdrant_url: str) -> QdrantSettings:
+    """Settings naming a collection unique to this test (ADR-0016 isolation)."""
+    return QdrantSettings(url=qdrant_url, collection=f"chunks_{uuid.uuid4().hex}")
+
+
+@pytest_asyncio.fixture(loop_scope="session")
+async def qdrant_client(qdrant_settings: QdrantSettings) -> AsyncIterator[AsyncQdrantClient]:
+    client = create_client(qdrant_settings)
+    try:
+        yield client
+    finally:
+        await client.close()

tests/integration/qdrant/test_collection.py

@@ -0,0 +1,114 @@
+"""`ensure_chunks_collection` against a real Qdrant (ADR-0001).
+
+The assertions that matter most here are the ones for schema properties that
+fail *silently* in production: the sparse `modifier=idf` and the pinned dense
+dimensions.
+"""
+
+import pytest
+from qdrant_client import AsyncQdrantClient, models
+
+from src.config import QdrantSettings
+from src.infrastructure.qdrant.collection import (
+    CollectionSchemaMismatchError,
+    ensure_chunks_collection,
+)
+
+pytestmark = [
+    pytest.mark.integration,
+    pytest.mark.qdrant,
+    pytest.mark.asyncio(loop_scope="session"),
+]
+
+
+async def test_ensure_chunks_collection_creates_all_four_named_vectors(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    created = await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+    assert created is True
+    info = await qdrant_client.get_collection(qdrant_settings.collection)
+    vectors = info.config.params.vectors
+    assert isinstance(vectors, dict)
+    assert vectors["dense_nomic"].size == 768
+    assert vectors["dense_openai"].size == 3072
+    assert vectors["late_interaction"].size == 128
+    assert vectors["late_interaction"].multivector_config is not None
+    assert vectors["late_interaction"].on_disk is True
+
+
+async def test_ensure_chunks_collection_sets_the_sparse_idf_modifier(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    """Without this, Qdrant applies no IDF and lexical retrieval silently
+    degrades -- no error, no warning (ADR-0005).
+    """
+    await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+    info = await qdrant_client.get_collection(qdrant_settings.collection)
+    sparse = info.config.params.sparse_vectors
+    assert sparse is not None
+    assert sparse["sparse"].modifier == models.Modifier.IDF
+
+
+async def test_ensure_chunks_collection_creates_the_payload_indexes(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+    info = await qdrant_client.get_collection(qdrant_settings.collection)
+    schema = info.payload_schema
+    assert set(schema) >= {
+        "tenant_id",
+        "domain",
+        "file_id",
+        "order_id",
+        "previous_chunk_id",
+        "next_chunk_id",
+    }
+    # order_id must be numeric: Qdrant's Range/order_by reject keyword payloads.
+    assert schema["order_id"].data_type == models.PayloadSchemaType.FLOAT
+    assert schema["tenant_id"].data_type == models.PayloadSchemaType.KEYWORD
+
+
+async def test_ensure_chunks_collection_is_idempotent(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    assert await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+    assert not await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+
+async def test_ensure_chunks_collection_rejects_a_mismatched_existing_collection(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    """A wrong-dimension collection must fail loudly, not be silently accepted."""
+    await qdrant_client.create_collection(
+        collection_name=qdrant_settings.collection,
+        vectors_config={
+            "dense_nomic": models.VectorParams(size=384, distance=models.Distance.COSINE),
+            "dense_openai": models.VectorParams(size=3072, distance=models.Distance.COSINE),
+            "late_interaction": models.VectorParams(size=128, distance=models.Distance.COSINE),
+        },
+        sparse_vectors_config={"sparse": models.SparseVectorParams(modifier=models.Modifier.IDF)},
+    )
+
+    with pytest.raises(CollectionSchemaMismatchError, match="768"):
+        await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
+
+
+async def test_ensure_chunks_collection_rejects_a_collection_without_the_idf_modifier(
+    qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
+) -> None:
+    await qdrant_client.create_collection(
+        collection_name=qdrant_settings.collection,
+        vectors_config={
+            "dense_nomic": models.VectorParams(size=768, distance=models.Distance.COSINE),
+            "dense_openai": models.VectorParams(size=3072, distance=models.Distance.COSINE),
+            "late_interaction": models.VectorParams(size=128, distance=models.Distance.COSINE),
+        },
+        sparse_vectors_config={"sparse": models.SparseVectorParams()},
+    )
+
+    with pytest.raises(CollectionSchemaMismatchError, match="idf"):
+        await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)