feat(qdrant): index content, is_active, and chunk_index on the chunks collection
Why: - ADR-0002's keyword search needs a full-text index on content, which collection.py deliberately deferred to plan 002. is_active and chunk_index were unindexed while ingestion was the only reader; every /v1/points read path filters on them. Changes: - content gets a TEXT index with the multilingual tokenizer, which segments Persian correctly where the word tokenizer mishandles ZWNJ-joined compounds. No stemmer or stopword list: content is already letter-folded by normalize_persian_text at ingest, and the ranked Farsi lexical path is the benchmarked BM25 sparse vector, not this index. - Tests assert content is TEXT rather than KEYWORD -- a keyword index would only match an entire chunk verbatim, which never happens and fails silently. - Adds a test that a missing index is added to an already-live collection. Impact: - Requires re-running `python -m src.cli.qdrant_bootstrap`. Payload indexes are additive, so no collection rebuild and no re-embedding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -80,14 +80,25 @@ def _sparse_vectors_config() -> dict[str, models.SparseVectorParams]:
|
||||
return {SPARSE_VECTOR: models.SparseVectorParams(modifier=models.Modifier.IDF)}
|
||||
|
||||
|
||||
# `content`'s full-text index backs ADR-0002's keyword search. The
|
||||
# `multilingual` tokenizer is the one that segments Persian correctly; `word`
|
||||
# splits on non-alphanumerics, which mis-handles ZWNJ-joined compounds. No
|
||||
# stemmer or stopword list is configured: `content` is already letter-folded by
|
||||
# `normalize_persian_text` at ingest (ADR-0018), and the *ranked* Farsi lexical
|
||||
# path is the benchmarked BM25 sparse vector, not this index. This one exists
|
||||
# for exact keyword/filter matching, which ADR-0002 keeps deliberately distinct
|
||||
# from retrieval.
|
||||
_CONTENT_INDEX = models.TextIndexParams(
|
||||
type=models.TextIndexType.TEXT,
|
||||
tokenizer=models.TokenizerType.MULTILINGUAL,
|
||||
lowercase=True,
|
||||
)
|
||||
|
||||
# (field name, schema). `order_id` is float because Qdrant's `Range` conditions
|
||||
# and `order_by` only support numeric/datetime payloads -- a keyword key could
|
||||
# only be sorted client-side after fetching every chunk (ADR-0001).
|
||||
#
|
||||
# The full-text index on `content` is deliberately absent: it belongs to plan
|
||||
# 002's keyword search, and payload indexes -- unlike vector config -- can be
|
||||
# added to a live collection later.
|
||||
_PAYLOAD_INDEXES: tuple[tuple[str, models.PayloadSchemaType | models.KeywordIndexParams], ...] = (
|
||||
_PayloadIndexSchema = models.PayloadSchemaType | models.KeywordIndexParams | models.TextIndexParams
|
||||
_PAYLOAD_INDEXES: tuple[tuple[str, _PayloadIndexSchema], ...] = (
|
||||
# `is_tenant` co-locates a tenant's vectors on disk for sequential reads,
|
||||
# which is the whole point of payload-partitioned multitenancy.
|
||||
(
|
||||
@@ -99,6 +110,13 @@ _PAYLOAD_INDEXES: tuple[tuple[str, models.PayloadSchemaType | models.KeywordInde
|
||||
("order_id", models.PayloadSchemaType.FLOAT),
|
||||
("previous_chunk_id", models.PayloadSchemaType.KEYWORD),
|
||||
("next_chunk_id", models.PayloadSchemaType.KEYWORD),
|
||||
("content", _CONTENT_INDEX),
|
||||
# Every read path filters on `is_active` (ADR-0002 implies `is_active: true`
|
||||
# unless the caller opts in), and the re-ingestion sweep and `/v1/points`
|
||||
# both range over `chunk_index`. Both were unindexed while ingestion was the
|
||||
# only reader; plan 002 makes them hot.
|
||||
("is_active", models.PayloadSchemaType.BOOL),
|
||||
("chunk_index", models.PayloadSchemaType.INTEGER),
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user