docs(adr): record query normalization and file-scoped pagination
Why: - Both are contracts callers depend on, not implementation details, and neither was written down. This repo treats the ADR as the source of truth rather than letting code diverge from it silently. Changes: - Record that the search query is folded the same way ingested content was, and why the alternative fails in the worst available way: an exact-looking query returning nothing, with no error and nothing in the logs to distinguish it from a genuine miss. - Record that listing requires file_id and paginates by order_id value, and why an offset cursor repeats an already-served row under a concurrent insert. - State that results carry no relevance score and no ranked order, so callers cannot read array position as relevance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -75,6 +75,51 @@ retrieval used by the AI agent in ADR-0003; the two "search" concepts serve
|
||||
different callers (a human/admin managing chunks vs. an agent retrieving
|
||||
context) and should not be conflated in the API or in future discussion.
|
||||
|
||||
Two properties follow from the index being a *filter*: results carry no
|
||||
relevance score, and their order is unspecified. The API therefore returns
|
||||
neither a score field nor a ranked list, and callers must not read the array
|
||||
order as relevance. A caller that wants ranking wants ADR-0003's path.
|
||||
|
||||
#### The query is normalized the way ingested content was
|
||||
|
||||
`normalize_persian_text` (ADR-0018) folds Arabic letterforms to their Persian
|
||||
equivalents — U+064A to U+06CC, U+0643 to U+06A9 — on every text block before
|
||||
chunking, so stored `content` is uniformly Persian-formed. A query string is
|
||||
not chunk content and never passes through that path, so a term typed on an
|
||||
Arabic keyboard reaches the index as a different codepoint sequence than the
|
||||
document it should match.
|
||||
|
||||
The service therefore applies the same folding to the query before matching.
|
||||
Without it the endpoint fails in the worst available way: an exact-looking
|
||||
query returns an empty result set, with no error, no warning, and nothing in
|
||||
the logs to distinguish "no such term" from "the term is spelled with the
|
||||
other yeh". Note this is a *query-side* transformation only — it changes what
|
||||
is compared, never what is stored.
|
||||
|
||||
This does not extend to stemming or synonyms. Qdrant's full-text index offers
|
||||
neither, and adding a Farsi analyzer here would duplicate the benchmarked BM25
|
||||
sparse pipeline (ADR-0005) in a code path that is not benchmarked against
|
||||
anything.
|
||||
|
||||
#### Listing is scoped to one file, and paginates by `order_id`
|
||||
|
||||
`GET /points?file_id=...` requires `file_id` rather than treating it as one
|
||||
optional filter among several, and its pagination cursor is an `order_id`
|
||||
value rather than an offset. Both follow from `order_id` being per-file:
|
||||
|
||||
- A cursor is only meaningful against a totally ordered key. `order_id` orders
|
||||
points within one file and says nothing across files, so an unscoped listing
|
||||
has no stable sort to paginate along.
|
||||
- An offset cursor is wrong even within one file. Insert, reorder, and delete
|
||||
all shift positions, so a page-two request issued after a concurrent insert
|
||||
ahead of the cursor would repeat a row already returned — silently. Ranging
|
||||
on `order_id > cursor` is unaffected: the reader has passed that value, and a
|
||||
point inserted behind it was already served.
|
||||
|
||||
The second point depends on `order_id` being unique within a file, which the
|
||||
gap-exhaustion rule below preserves by rejecting a reorder whose computed gap
|
||||
would collapse onto a neighbour value.
|
||||
|
||||
### Delete is soft by default
|
||||
|
||||
`DELETE /points/{point_id}` and `DELETE /points?file_id=...` set
|
||||
|
||||
Reference in New Issue
Block a user