docs(adr): record query normalization and file-scoped pagination

Why:
- Both are contracts callers depend on, not implementation details, and neither
  was written down. This repo treats the ADR as the source of truth rather than
  letting code diverge from it silently.

Changes:
- Record that the search query is folded the same way ingested content was, and
  why the alternative fails in the worst available way: an exact-looking query
  returning nothing, with no error and nothing in the logs to distinguish it
  from a genuine miss.
- Record that listing requires file_id and paginates by order_id value, and why
  an offset cursor repeats an already-served row under a concurrent insert.
- State that results carry no relevance score and no ranked order, so callers
  cannot read array position as relevance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-22 15:09:07 +03:30
parent 3b9434faf4
commit 43932b6562

View File

@@ -75,6 +75,51 @@ retrieval used by the AI agent in ADR-0003; the two "search" concepts serve
different callers (a human/admin managing chunks vs. an agent retrieving different callers (a human/admin managing chunks vs. an agent retrieving
context) and should not be conflated in the API or in future discussion. context) and should not be conflated in the API or in future discussion.
Two properties follow from the index being a *filter*: results carry no
relevance score, and their order is unspecified. The API therefore returns
neither a score field nor a ranked list, and callers must not read the array
order as relevance. A caller that wants ranking wants ADR-0003's path.
#### The query is normalized the way ingested content was
`normalize_persian_text` (ADR-0018) folds Arabic letterforms to their Persian
equivalents — U+064A to U+06CC, U+0643 to U+06A9 — on every text block before
chunking, so stored `content` is uniformly Persian-formed. A query string is
not chunk content and never passes through that path, so a term typed on an
Arabic keyboard reaches the index as a different codepoint sequence than the
document it should match.
The service therefore applies the same folding to the query before matching.
Without it the endpoint fails in the worst available way: an exact-looking
query returns an empty result set, with no error, no warning, and nothing in
the logs to distinguish "no such term" from "the term is spelled with the
other yeh". Note this is a *query-side* transformation only — it changes what
is compared, never what is stored.
This does not extend to stemming or synonyms. Qdrant's full-text index offers
neither, and adding a Farsi analyzer here would duplicate the benchmarked BM25
sparse pipeline (ADR-0005) in a code path that is not benchmarked against
anything.
#### Listing is scoped to one file, and paginates by `order_id`
`GET /points?file_id=...` requires `file_id` rather than treating it as one
optional filter among several, and its pagination cursor is an `order_id`
value rather than an offset. Both follow from `order_id` being per-file:
- A cursor is only meaningful against a totally ordered key. `order_id` orders
points within one file and says nothing across files, so an unscoped listing
has no stable sort to paginate along.
- An offset cursor is wrong even within one file. Insert, reorder, and delete
all shift positions, so a page-two request issued after a concurrent insert
ahead of the cursor would repeat a row already returned — silently. Ranging
on `order_id > cursor` is unaffected: the reader has passed that value, and a
point inserted behind it was already served.
The second point depends on `order_id` being unique within a file, which the
gap-exhaustion rule below preserves by rejecting a reorder whose computed gap
would collapse onto a neighbour value.
### Delete is soft by default ### Delete is soft by default
`DELETE /points/{point_id}` and `DELETE /points?file_id=...` set `DELETE /points/{point_id}` and `DELETE /points?file_id=...` set