docs(adr): record the re-ingestion rule and close plan 002's open decisions
Why: - ADR-0002 already answered three of the four questions plan 002 listed as "decisions needed"; the fourth -- what happens to a manually edited point when its file is re-uploaded -- was left for a Phase 6 test to force. Deciding it in code rather than in the ADR would invert this repo's rule. Changes: - ADR-0002 gains "Re-ingestion versus manual edits": the new file wins, surviving points are overwritten with an incremented version, absent points are flagged inactive rather than removed, and manually created points sit past the ingested chunk_index range so the existing sweep covers them. - Plan 002's stale decisions section becomes a pointer table; its audit scope is pinned to both ADR-0009 tables. Impact: - Clobbered edits are recoverable from point_audit_events, not from Qdrant: the deterministic point ID cannot hold both versions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -105,6 +105,41 @@ Qdrant's `update_filter`, giving an optimistic-concurrency-style guard
|
|||||||
against races between a concurrent ingestion re-run (ADR-0001) and a manual
|
against races between a concurrent ingestion re-run (ADR-0001) and a manual
|
||||||
edit through this API.
|
edit through this API.
|
||||||
|
|
||||||
|
### Re-ingestion versus manual edits
|
||||||
|
|
||||||
|
A file can be re-uploaded after someone has hand-edited one of its points
|
||||||
|
through this API. **The newly ingested file wins.** Ingestion is authoritative
|
||||||
|
for the content of the file it ingested; a manual edit is a correction that
|
||||||
|
survives only until the source document is replaced.
|
||||||
|
|
||||||
|
Concretely:
|
||||||
|
|
||||||
|
- A point that still exists in the new version (same `file_id` +
|
||||||
|
`chunk_index`, hence the same deterministic point ID) is **overwritten in
|
||||||
|
place**. Ingestion performs a read-check-write so `version` is incremented
|
||||||
|
from whatever the manual edit left it at, rather than reset to `1`.
|
||||||
|
- A point from the previous ingestion that is **absent** from the new version
|
||||||
|
is flagged `is_active: false` with `deleted_at` set. It is never removed
|
||||||
|
from Qdrant — the soft-delete rule above applies to re-ingestion exactly as
|
||||||
|
it applies to `DELETE`.
|
||||||
|
- A manually created point (`POST /points`) is assigned a `chunk_index` past
|
||||||
|
the ingested range, so the same sweep deactivates it on the next upload of
|
||||||
|
its file. This is the intended consequence of "the new file wins", not an
|
||||||
|
accident of the sweep's bounds.
|
||||||
|
|
||||||
|
Because the point ID is derived from the immutable `chunk_index`, an
|
||||||
|
overwritten point cannot hold both the manual edit and the new file's content.
|
||||||
|
The clobbered content is therefore recorded in `point_audit_events`
|
||||||
|
(ADR-0009) as a `reingest_overwrite` operation carrying `before_version`, so
|
||||||
|
the edit is recoverable from the audit trail even though it is no longer a
|
||||||
|
live point.
|
||||||
|
|
||||||
|
Rejected alternative: preserving manual edits by having ingestion skip points
|
||||||
|
with `version > 1`. It breaks the guarantee that a successful upload leaves
|
||||||
|
Qdrant matching the uploaded document, and it needs a second, separate rule
|
||||||
|
for edited points that no longer exist in the new version — two divergent
|
||||||
|
notions of authority over one file.
|
||||||
|
|
||||||
### Re-embedding on content edit
|
### Re-embedding on content edit
|
||||||
|
|
||||||
`PUT /points/{point_id}` can change `content`, which leaves the stored
|
`PUT /points/{point_id}` can change `content`, which leaves the stored
|
||||||
|
|||||||
@@ -15,8 +15,9 @@ are; this document defines order, scope, and verification criteria.
|
|||||||
|
|
||||||
## Prerequisite
|
## Prerequisite
|
||||||
|
|
||||||
Plan 001 must be complete through **Phase 5** before Phase 3 of this plan
|
Plan 001 is complete through Phase 6, so this prerequisite is satisfied. It
|
||||||
starts. Specifically this plan depends on: the `chunks` collection and its
|
required plan 001 through **Phase 5** before Phase 3 of this plan starts.
|
||||||
|
Specifically this plan depends on: the `chunks` collection and its
|
||||||
payload indexes actually existing, API-key authentication and `AuthContext`
|
payload indexes actually existing, API-key authentication and `AuthContext`
|
||||||
tenant derivation, the application-lifetime Qdrant client from the FastAPI
|
tenant derivation, the application-lifetime Qdrant client from the FastAPI
|
||||||
lifespan, and the request-lifetime `AsyncSession` wiring. Phases 1–2 below
|
lifespan, and the request-lifetime `AsyncSession` wiring. Phases 1–2 below
|
||||||
@@ -71,7 +72,9 @@ project owner accepts them, and update the ADR rather than diverging silently.
|
|||||||
(bulk soft delete of a file's points), from ADR-0008.
|
(bulk soft delete of a file's points), from ADR-0008.
|
||||||
- Soft delete as the default for every delete path, with neighbor relinking.
|
- Soft delete as the default for every delete path, with neighbor relinking.
|
||||||
- Optimistic concurrency on every mutating path via the `version` payload field.
|
- Optimistic concurrency on every mutating path via the `version` payload field.
|
||||||
- Audit rows in Postgres for mutating operations.
|
- Audit rows in Postgres for mutating operations: both ADR-0009 tables,
|
||||||
|
`api_request_logs` (one row per API call, written from the request middleware)
|
||||||
|
and `point_audit_events` with the real `api_request_log_id` foreign key.
|
||||||
- Automated tests for tenant isolation, pointer integrity, concurrency
|
- Automated tests for tenant isolation, pointer integrity, concurrency
|
||||||
conflicts, and pagination.
|
conflicts, and pagination.
|
||||||
|
|
||||||
@@ -113,40 +116,22 @@ project owner accepts them, and update the ADR rather than diverging silently.
|
|||||||
9. Routers contain no Qdrant SDK calls and no filter construction. The Qdrant
|
9. Routers contain no Qdrant SDK calls and no filter construction. The Qdrant
|
||||||
client is injected from the lifespan (ADR-0012).
|
client is injected from the lifespan (ADR-0012).
|
||||||
|
|
||||||
## Decisions needed before the affected phase
|
## Decisions resolved before implementation
|
||||||
|
|
||||||
### Re-embedding on content edit (blocks Phase 4)
|
An earlier revision of this plan listed three open decisions here. All are now
|
||||||
|
settled, and one further question this plan deferred to a Phase 6 test has been
|
||||||
|
settled too. They are recorded in the ADRs — these lines are a pointer, not a
|
||||||
|
second source of truth.
|
||||||
|
|
||||||
`PUT /v1/points/{point_id}` can change `content`. The stored vectors then no
|
| Question | Resolution | Recorded in |
|
||||||
longer match the text. Three options, in order of preference:
|
|---|---|---|
|
||||||
|
| Re-embedding on content edit | Re-embed inline, reusing ingestion's ports and bounds and its `502`/`504` codes. The re-embed happens *before* the version-guarded write, so a stale edit still `409`s rather than re-embedding for nothing. | ADR-0002, "Re-embedding on content edit" |
|
||||||
|
| Fractional-key exhaustion | No renormalize endpoint in this slice. Log `points.order_id.gap_low` under a safety threshold; reject with `409` and a distinct error code if the gap would collapse onto a neighbor value. Recovery is a runbook operation. | ADR-0002, "`order_id` gap exhaustion" |
|
||||||
|
| Batch semantics | All-or-nothing, capped at 100 operations. Every operation's `version` precondition is validated before any is applied; one failure rejects the whole request and nothing reaches Qdrant. | ADR-0002, "`POST /points/batch` semantics" |
|
||||||
|
| Re-ingestion versus manual edits | The newly uploaded file wins. Surviving points are overwritten in place with an incremented `version`; points absent from the new version are flagged inactive, never removed; manually created points sit past the ingested `chunk_index` range and are swept by the same rule. Clobbered content is recorded in `point_audit_events` as `reingest_overwrite`. | ADR-0002, "Re-ingestion versus manual edits" |
|
||||||
|
|
||||||
1. **Re-embed inline** on content change, reusing plan 001's embedding ports and
|
Phase 6's cross-slice end-to-end test therefore *verifies* the re-ingestion rule
|
||||||
bounds. Consistent, but puts embedder latency and `502`/`504` failure modes
|
rather than forcing the decision.
|
||||||
on an admin edit path.
|
|
||||||
2. **Require caller-supplied vectors** when content changes, and reject the edit
|
|
||||||
otherwise. Simple and honest, but pushes model knowledge to the client.
|
|
||||||
3. **Mark the point stale** (a payload flag) and re-embed later. Needs
|
|
||||||
background work, which ADR-0017 currently rules out.
|
|
||||||
|
|
||||||
Default to (1) for parity with ingestion, with the same batch/semaphore bounds
|
|
||||||
and the same status codes. Record whichever is chosen in ADR-0002 before
|
|
||||||
implementing Phase 4 — this is a real behavioral contract, not an
|
|
||||||
implementation detail.
|
|
||||||
|
|
||||||
### Fractional-key exhaustion
|
|
||||||
|
|
||||||
ADR-0001 notes float keys eventually need renormalization. Decide now whether
|
|
||||||
this slice ships a renormalize path (an internal operation rewriting a file's
|
|
||||||
`order_id` values to `1000, 2000, 3000, ...`) or explicitly defers it with a
|
|
||||||
logged warning when the gap between neighbors falls under a threshold. Deferring
|
|
||||||
is acceptable; silently producing unrepresentable gaps is not.
|
|
||||||
|
|
||||||
### Batch semantics
|
|
||||||
|
|
||||||
`POST /v1/points/batch` must define, in the API schema and the tests: whether
|
|
||||||
operations are all-or-nothing, what happens when operation 3 of 5 fails a
|
|
||||||
version check, and the maximum operation count per request. Decide before
|
|
||||||
Phase 5; do not let the answer be "whatever Qdrant happened to do."
|
|
||||||
|
|
||||||
## Build order
|
## Build order
|
||||||
|
|
||||||
@@ -251,8 +236,8 @@ of mutations.
|
|||||||
ordering behavior, isolated per test by unique collection or tenant keys.
|
ordering behavior, isolated per test by unique collection or tenant keys.
|
||||||
2. An end-to-end test crossing plan 001 and this slice: ingest a CSV, list its
|
2. An end-to-end test crossing plan 001 and this slice: ingest a CSV, list its
|
||||||
points, reorder one, soft-delete another, re-upload the same file, and assert
|
points, reorder one, soft-delete another, re-upload the same file, and assert
|
||||||
the manual edits interact with re-ingestion exactly as ADR-0001/0002 specify.
|
the manual edits interact with re-ingestion exactly as ADR-0002's
|
||||||
If that interaction is not yet decided, this test is what forces the decision.
|
"Re-ingestion versus manual edits" specifies.
|
||||||
3. Structured logging at the mutation boundary with stable event names
|
3. Structured logging at the mutation boundary with stable event names
|
||||||
(`points.updated`, `points.reordered`, `points.soft_deleted`) carrying
|
(`points.updated`, `points.reordered`, `points.soft_deleted`) carrying
|
||||||
`request_id`, `tenant_id`, `file_id`, and the resulting version.
|
`request_id`, `tenant_id`, `file_id`, and the resulting version.
|
||||||
|
|||||||
Reference in New Issue
Block a user