chatbot_v3/docs/adr/0002-chunk-crud-and-search-api.md

0002. Chunk CRUD and payload/keyword search API

Status

Accepted

Route naming note: ADR-0008 owns the REST API surface and renames the indicative endpoint examples in this ADR from /chunks/... to /points/.... The CRUD semantics, Qdrant primitives, payload schema, soft-delete behavior, and tenant-isolation rules in this ADR remain accepted.

Context

Beyond bulk ingestion (0001), we need FastAPI endpoints for direct, fine-grained management of individual chunks: create a single chunk, update a chunk's content/vectors/payload, delete chunks (by ID or in bulk by filter), and list/search chunks by keyword and payload metadata (tenant, domain, file, etc.) — independent of the semantic hybrid retrieval used by the AI agent (ADR-0003).

This pipeline operates on the same collection and payload schema defined in ADR-0001, and Qdrant's Points API already provides the primitives needed — there's no need for a custom data layer on top.

Decision

Expose a thin FastAPI layer directly over Qdrant's Points API, operating on the chunks collection and payload schema from ADR-0001:

Operation	FastAPI endpoint (indicative)	Qdrant primitive
Create a point	POST /points	upsert (single point)
Update a point's vectors/content	PUT /points/{point_id}	upsert (update_only mode)
Partially update payload	PATCH /points/{point_id}/payload	set_payload / overwrite_payload
Delete one point	DELETE /points/{point_id}	delete by ID
Delete many points	DELETE /points?file_id=...	delete by filter
List/paginate points, in order	GET /points?file_id=...	scroll with filter + pagination, order_by: order_id
Reorder/insert a point	PATCH /points/{point_id}/order	set_payload on order_id only
Count points	GET /points/count	count
Keyword search	GET /points/search?q=...	full-text payload index on content (match, not semantic)
Bulk multi-op edits	POST /points/batch	Qdrant points/batch

Ordering and reordering

Chunks belonging to a file_id are listed via scroll ordered by the mutable order_id payload field (ADR-0001) — this is what lets the frontend display and let a backend user modify chunk order. Because order_id is a fractional float key rather than a sequential integer position, moving or inserting a chunk only requires assigning it a new value between its two new neighbors — no renumbering of siblings, and no effect on the chunk's stable point ID (which is derived from the immutable chunk_index, not order_id).

PATCH /points/{point_id}/order does more than set one field, though: since ADR-0001 also maintains previous_chunk_id/next_chunk_id pointers for O(1) adjacency lookups, a single reorder touches up to four chunks in one points/batch call:

1. The moved chunk: new order_id, new previous_chunk_id/next_chunk_id.
2. Its old neighbors: re-link their previous_chunk_id/next_chunk_id to skip over the moved chunk.
3. Its new neighbors: re-link them to point at the moved chunk.

The same fix-up applies to insert (new chunk) and delete (soft-deleted chunk — see below) operations: any operation that changes a chunk's position or removes it from the sequence must atomically update the neighbors' pointers in the same points/batch request, or previous_chunk_id/next_chunk_id go stale.

Keyword search is not semantic search

GET /points/search matches against the full-text payload index on content (and structured filters on tenant_id, domain, file_id, etc.) — it is filter/match-based keyword search, not embedding-based retrieval. This is deliberately distinct from the hybrid dense+sparse retrieval used by the AI agent in ADR-0003; the two "search" concepts serve different callers (a human/admin managing chunks vs. an agent retrieving context) and should not be conflated in the API or in future discussion.

Two properties follow from the index being a filter: results carry no relevance score, and their order is unspecified. The API therefore returns neither a score field nor a ranked list, and callers must not read the array order as relevance. A caller that wants ranking wants ADR-0003's path.

The query is normalized the way ingested content was

normalize_persian_text (ADR-0018) folds Arabic letterforms to their Persian equivalents — U+064A to U+06CC, U+0643 to U+06A9 — on every text block before chunking, so stored content is uniformly Persian-formed. A query string is not chunk content and never passes through that path, so a term typed on an Arabic keyboard reaches the index as a different codepoint sequence than the document it should match.

The service therefore applies the same folding to the query before matching. Without it the endpoint fails in the worst available way: an exact-looking query returns an empty result set, with no error, no warning, and nothing in the logs to distinguish "no such term" from "the term is spelled with the other yeh". Note this is a query-side transformation only — it changes what is compared, never what is stored.

This does not extend to stemming or synonyms. Qdrant's full-text index offers neither, and adding a Farsi analyzer here would duplicate the benchmarked BM25 sparse pipeline (ADR-0005) in a code path that is not benchmarked against anything.

Listing is scoped to one file, and paginates by order_id

GET /points?file_id=... requires file_id rather than treating it as one optional filter among several, and its pagination cursor is an order_id value rather than an offset. Both follow from order_id being per-file:

• A cursor is only meaningful against a totally ordered key. order_id orders points within one file and says nothing across files, so an unscoped listing has no stable sort to paginate along.
• An offset cursor is wrong even within one file. Insert, reorder, and delete all shift positions, so a page-two request issued after a concurrent insert ahead of the cursor would repeat a row already returned — silently. Ranging on order_id > cursor is unaffected: the reader has passed that value, and a point inserted behind it was already served.

The second point depends on order_id being unique within a file, which the gap-exhaustion rule below preserves by rejecting a reorder whose computed gap would collapse onto a neighbour value.

Delete is soft by default

DELETE /points/{point_id} and DELETE /points?file_id=... set is_active: false and deleted_at (via set_payload) rather than removing the point from Qdrant — consistent with ADR-0001's soft-delete fields. This keeps deleted points available for audit and lets GET /points and GET /points/search filter them out by default (is_active: true implied unless the caller explicitly asks to include inactive chunks). A hard delete (actual point removal, e.g. points/delete) is available separately for compliance-driven purges, not as the default CRUD behavior. Either way, the deleted chunk's previous_chunk_id/next_chunk_id neighbors are relinked to point at each other in the same points/batch call, so context-window expansion (ADR-0003) never walks into a deactivated or removed chunk.

Tenant isolation

Every endpoint (read and write) has its tenant_id filter injected server-side from the authenticated request context — never accepted as client-supplied input in the request body or query string. This is non-negotiable: it's the same isolation boundary ADR-0001 relies on for multitenancy, and it must hold for every code path that touches the collection, not just ingestion.

Concurrency

Updates use the version payload field (from ADR-0001) together with Qdrant's update_filter, giving an optimistic-concurrency-style guard against races between a concurrent ingestion re-run (ADR-0001) and a manual edit through this API.

Re-ingestion versus manual edits

A file can be re-uploaded after someone has hand-edited one of its points through this API. The newly ingested file wins. Ingestion is authoritative for the content of the file it ingested; a manual edit is a correction that survives only until the source document is replaced.

Concretely:

• A point that still exists in the new version (same file_id + chunk_index, hence the same deterministic point ID) is overwritten in place. Ingestion performs a read-check-write so version is incremented from whatever the manual edit left it at, rather than reset to 1.
• A point from the previous ingestion that is absent from the new version is flagged is_active: false with deleted_at set. It is never removed from Qdrant — the soft-delete rule above applies to re-ingestion exactly as it applies to DELETE.
• A manually created point (POST /points) is assigned a chunk_index past the ingested range, so the same sweep deactivates it on the next upload of its file. This is the intended consequence of "the new file wins", not an accident of the sweep's bounds.

Because the point ID is derived from the immutable chunk_index, an overwritten point cannot hold both the manual edit and the new file's content. The clobbered content is therefore recorded in point_audit_events (ADR-0009) as a reingest_overwrite operation carrying before_version, so the edit is recoverable from the audit trail even though it is no longer a live point.

Rejected alternative: preserving manual edits by having ingestion skip points with version > 1. It breaks the guarantee that a successful upload leaves Qdrant matching the uploaded document, and it needs a second, separate rule for edited points that no longer exist in the new version — two divergent notions of authority over one file.

Re-embedding on content edit

PUT /points/{point_id} can change content, which leaves the stored vectors stale unless they're recomputed. When content changes, the point is re-embedded inline, reusing the same async embedding ports and batch/semaphore bounds ingestion uses (0017), for parity between the two write paths. When content is unchanged, the edit applies only the supplied vector/payload fields and skips re-embedding entirely. Failure modes on this path reuse ingestion's status codes: 502 on embedder failure, 504 if the edit's embedding step exceeds the same timeout budget class as ingestion. The version guard (update_filter) still applies to the write — re-embedding happens before the guarded write, not instead of it, so a stale-version edit still fails with 409 rather than re-embedding for nothing.

Rejected alternatives: requiring the caller to supply vectors when content changes (pushes model knowledge onto the client, and is easy to get subtly wrong); marking the point stale for background re-embedding later (needs background work, which ADR-0017 currently rules out for this slice).

order_id gap exhaustion

Repeatedly inserting into the same gap between two neighbors eventually exhausts float precision (ADR-0001's known limitation). This slice does not ship a renormalize endpoint. Instead, any operation that assigns a new fractional order_id between two neighbors (insert, reorder) computes the resulting gap and:

• logs a structured warning (points.order_id.gap_low) with file_id and the two neighbor point IDs once the gap falls under a defined safety threshold, so the condition is observable before it becomes uninsertable;
• rejects the write with 409 and a distinct error code if the computed gap is no longer representable (would collapse to one of the two neighbor values), instead of silently applying an imprecise value.

Recovering from an exhausted gap is a manual data-fix operation covered by the operator runbook, not an endpoint this slice builds — deferring the renormalize primitive is acceptable, silently producing an unrepresentable gap is not.

POST /points/batch semantics

Batch requests are all-or-nothing, capped at 100 operations per request. The service layer validates every operation's version precondition before applying any of them; if any operation's precondition fails, the whole request is rejected with 409 and nothing is applied — no partially-applied batch ever reaches Qdrant. This follows directly from the version-guard rule above applied at the batch level, and from the pointer-relinking rule (a reorder/insert/delete's neighbor updates must land in the same points/batch call, and a partial relink is a defect): partial application of a batch is exactly the failure mode that would produce a stale pointer chain. The 100-operation cap is independent of ADR-0001's 64–256-point bulk-ingestion batch sizing — that number is about upload throughput; this one bounds an admin/manual edit request to something that comfortably finishes inside a normal request timeout.

Consequences

Positive

• Reuses the exact collection/schema from ADR-0001 — no parallel data model to keep in sync.
• Thin mapping to Qdrant primitives keeps the API predictable and easy to extend as new filter/sort needs arise.
• Clear separation from ADR-0003's semantic search avoids API consumers confusing "find chunks matching this filter" with "retrieve chunks relevant to this query."

Negative

• Keyword search here is limited to what Qdrant's full-text payload index supports — no stemming/synonym handling beyond what that index offers.
• Optimistic concurrency via version requires every writer (ingestion and this API) to consistently read-check-write; a writer that skips this can silently clobber concurrent edits.
• Inline re-embedding puts embedder latency and 502/504 failure modes on an admin content edit, not just on ingestion — an edit that only intended to fix a typo pays the same embedding cost as a fresh chunk.
• Deferring the order_id renormalize endpoint means a file whose gaps are genuinely exhausted has no automated recovery in this slice; an operator must intervene by hand until that endpoint exists.
• All-or-nothing batch semantics mean one stale operation in a 100-operation batch fails the entire request, even when the other 99 operations are independent and would have succeeded on their own.

Alternatives Considered

• Separate keyword-search infrastructure (e.g. Elasticsearch): rejected — Qdrant's native full-text payload index already covers filter-style keyword search on content, and adding a second search system would duplicate data and infrastructure for no clear benefit at current scale.
• Client-supplied tenant_id in request body: rejected — trusting client input for the isolation boundary is a direct multitenancy security risk; it must come from server-side auth context.
• Caller-supplied vectors on content edit: rejected — pushes embedding model knowledge onto the client and makes it easy to silently desync vectors from content.
• Mark-stale-and-re-embed-later on content edit: rejected for this slice — needs background work, which ADR-0017 currently rules out.
• Renormalize order_id automatically within this slice: rejected — nothing in current scope has hit gap exhaustion; building the primitive now is speculative. Revisit if the logged warning starts firing in practice.
• Partial-success batch semantics (per-operation status): rejected — a partially-applied batch is exactly the failure mode that leaves the pointer chain (previous_chunk_id/next_chunk_id) inconsistent, which this ADR treats as a defect, not a degraded-but-acceptable outcome.