docs(architecture): adopt inline synchronous ingestion (ADR-0017)
Why: - ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path. Changes: - new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it. Impact: - no broker, outbox, or worker code; rabbitmq test marker removed.
This commit is contained in:
@@ -105,6 +105,62 @@ Qdrant's `update_filter`, giving an optimistic-concurrency-style guard
|
||||
against races between a concurrent ingestion re-run (ADR-0001) and a manual
|
||||
edit through this API.
|
||||
|
||||
### Re-embedding on content edit
|
||||
|
||||
`PUT /points/{point_id}` can change `content`, which leaves the stored
|
||||
vectors stale unless they're recomputed. When `content` changes, the point
|
||||
is **re-embedded inline**, reusing the same async embedding ports and
|
||||
batch/semaphore bounds ingestion uses ([0017](0017-synchronous-ingestion-in-the-request-path.md)),
|
||||
for parity between the two write paths. When `content` is unchanged, the
|
||||
edit applies only the supplied vector/payload fields and skips re-embedding
|
||||
entirely. Failure modes on this path reuse ingestion's status codes: `502`
|
||||
on embedder failure, `504` if the edit's embedding step exceeds the same
|
||||
timeout budget class as ingestion. The `version` guard (`update_filter`)
|
||||
still applies to the write — re-embedding happens before the guarded write,
|
||||
not instead of it, so a stale-version edit still fails with `409` rather
|
||||
than re-embedding for nothing.
|
||||
|
||||
Rejected alternatives: requiring the caller to supply vectors when content
|
||||
changes (pushes model knowledge onto the client, and is easy to get subtly
|
||||
wrong); marking the point stale for background re-embedding later (needs
|
||||
background work, which ADR-0017 currently rules out for this slice).
|
||||
|
||||
### `order_id` gap exhaustion
|
||||
|
||||
Repeatedly inserting into the same gap between two neighbors eventually
|
||||
exhausts float precision (ADR-0001's known limitation). This slice does
|
||||
**not** ship a renormalize endpoint. Instead, any operation that assigns a
|
||||
new fractional `order_id` between two neighbors (insert, reorder) computes
|
||||
the resulting gap and:
|
||||
|
||||
- logs a structured warning (`points.order_id.gap_low`) with `file_id` and
|
||||
the two neighbor point IDs once the gap falls under a defined safety
|
||||
threshold, so the condition is observable before it becomes uninsertable;
|
||||
- **rejects** the write with `409` and a distinct error code if the
|
||||
computed gap is no longer representable (would collapse to one of the two
|
||||
neighbor values), instead of silently applying an imprecise value.
|
||||
|
||||
Recovering from an exhausted gap is a manual data-fix operation covered by
|
||||
the operator runbook, not an endpoint this slice builds — deferring the
|
||||
renormalize primitive is acceptable, silently producing an unrepresentable
|
||||
gap is not.
|
||||
|
||||
### `POST /points/batch` semantics
|
||||
|
||||
Batch requests are **all-or-nothing**, capped at **100 operations per
|
||||
request**. The service layer validates every operation's `version`
|
||||
precondition before applying any of them; if any operation's precondition
|
||||
fails, the whole request is rejected with `409` and nothing is applied — no
|
||||
partially-applied batch ever reaches Qdrant. This follows directly from the
|
||||
`version`-guard rule above applied at the batch level, and from the
|
||||
pointer-relinking rule (a reorder/insert/delete's neighbor updates must land
|
||||
in the same `points/batch` call, and a partial relink is a defect): partial
|
||||
application of a batch is exactly the failure mode that would produce a
|
||||
stale pointer chain. The 100-operation cap is independent of ADR-0001's
|
||||
64–256-point bulk-ingestion batch sizing — that number is about upload
|
||||
throughput; this one bounds an admin/manual edit request to something that
|
||||
comfortably finishes inside a normal request timeout.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
@@ -122,6 +178,15 @@ edit through this API.
|
||||
- Optimistic concurrency via `version` requires every writer (ingestion and
|
||||
this API) to consistently read-check-write; a writer that skips this can
|
||||
silently clobber concurrent edits.
|
||||
- Inline re-embedding puts embedder latency and `502`/`504` failure modes on
|
||||
an admin content edit, not just on ingestion — an edit that only intended
|
||||
to fix a typo pays the same embedding cost as a fresh chunk.
|
||||
- Deferring the `order_id` renormalize endpoint means a file whose gaps are
|
||||
genuinely exhausted has no automated recovery in this slice; an operator
|
||||
must intervene by hand until that endpoint exists.
|
||||
- All-or-nothing batch semantics mean one stale operation in a 100-operation
|
||||
batch fails the entire request, even when the other 99 operations are
|
||||
independent and would have succeeded on their own.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
@@ -132,3 +197,16 @@ edit through this API.
|
||||
- **Client-supplied `tenant_id` in request body**: rejected — trusting
|
||||
client input for the isolation boundary is a direct multitenancy security
|
||||
risk; it must come from server-side auth context.
|
||||
- **Caller-supplied vectors on content edit**: rejected — pushes embedding
|
||||
model knowledge onto the client and makes it easy to silently desync
|
||||
vectors from content.
|
||||
- **Mark-stale-and-re-embed-later on content edit**: rejected for this
|
||||
slice — needs background work, which ADR-0017 currently rules out.
|
||||
- **Renormalize `order_id` automatically within this slice**: rejected —
|
||||
nothing in current scope has hit gap exhaustion; building the primitive
|
||||
now is speculative. Revisit if the logged warning starts firing in
|
||||
practice.
|
||||
- **Partial-success batch semantics (per-operation status)**: rejected —
|
||||
a partially-applied batch is exactly the failure mode that leaves the
|
||||
pointer chain (`previous_chunk_id`/`next_chunk_id`) inconsistent, which
|
||||
this ADR treats as a defect, not a degraded-but-acceptable outcome.
|
||||
|
||||
Reference in New Issue
Block a user