diff --git a/docs/adr/0002-chunk-crud-and-search-api.md b/docs/adr/0002-chunk-crud-and-search-api.md index 8309837..9c96ad2 100644 --- a/docs/adr/0002-chunk-crud-and-search-api.md +++ b/docs/adr/0002-chunk-crud-and-search-api.md @@ -75,6 +75,51 @@ retrieval used by the AI agent in ADR-0003; the two "search" concepts serve different callers (a human/admin managing chunks vs. an agent retrieving context) and should not be conflated in the API or in future discussion. +Two properties follow from the index being a *filter*: results carry no +relevance score, and their order is unspecified. The API therefore returns +neither a score field nor a ranked list, and callers must not read the array +order as relevance. A caller that wants ranking wants ADR-0003's path. + +#### The query is normalized the way ingested content was + +`normalize_persian_text` (ADR-0018) folds Arabic letterforms to their Persian +equivalents — U+064A to U+06CC, U+0643 to U+06A9 — on every text block before +chunking, so stored `content` is uniformly Persian-formed. A query string is +not chunk content and never passes through that path, so a term typed on an +Arabic keyboard reaches the index as a different codepoint sequence than the +document it should match. + +The service therefore applies the same folding to the query before matching. +Without it the endpoint fails in the worst available way: an exact-looking +query returns an empty result set, with no error, no warning, and nothing in +the logs to distinguish "no such term" from "the term is spelled with the +other yeh". Note this is a *query-side* transformation only — it changes what +is compared, never what is stored. + +This does not extend to stemming or synonyms. Qdrant's full-text index offers +neither, and adding a Farsi analyzer here would duplicate the benchmarked BM25 +sparse pipeline (ADR-0005) in a code path that is not benchmarked against +anything. + +#### Listing is scoped to one file, and paginates by `order_id` + +`GET /points?file_id=...` requires `file_id` rather than treating it as one +optional filter among several, and its pagination cursor is an `order_id` +value rather than an offset. Both follow from `order_id` being per-file: + +- A cursor is only meaningful against a totally ordered key. `order_id` orders + points within one file and says nothing across files, so an unscoped listing + has no stable sort to paginate along. +- An offset cursor is wrong even within one file. Insert, reorder, and delete + all shift positions, so a page-two request issued after a concurrent insert + ahead of the cursor would repeat a row already returned — silently. Ranging + on `order_id > cursor` is unaffected: the reader has passed that value, and a + point inserted behind it was already served. + +The second point depends on `order_id` being unique within a file, which the +gap-exhaustion rule below preserves by rejecting a reorder whose computed gap +would collapse onto a neighbour value. + ### Delete is soft by default `DELETE /points/{point_id}` and `DELETE /points?file_id=...` set