docs(architecture): adopt inline synchronous ingestion (ADR-0017)

Why:
- ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path.

Changes:
- new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it.

Impact:
- no broker, outbox, or worker code; rabbitmq test marker removed.
This commit is contained in:
2026-08-16 11:53:59 +03:30
parent e2322a2909
commit fd70ad01af
9 changed files with 542 additions and 154 deletions

View File

@@ -164,16 +164,20 @@ metadata such as `domain`. File validation is server-side:
- Derive `tenant_id`, `created_by`, and `updated_by` from `AuthContext`, not
from form fields.
Ingestion may be slow because it parses, chunks, embeds, and writes many
Qdrant points. The REST contract is job-shaped even if the first
implementation runs inline:
Ingestion parses, chunks, embeds, and writes many Qdrant points.
[ADR-0017](0017-synchronous-ingestion-in-the-request-path.md) supersedes the
job-shaped contract originally specified here: ingestion runs inline and the
response is terminal.
```text
202 Accepted -> { file_id, ingestion_job_id, status: "queued" | "running" }
201 Created -> { file_id, ingestion_job_id, status: "succeeded", chunks_indexed }
```
A durable worker/job queue can be added later without changing the API
contract.
`ingestion_job_id` is retained so the attempt stays inspectable via
`GET /v1/files/{file_id}`, and so a future move back to a queued `202 Accepted`
contract (ADR-0014) is additive for clients that already read it. Ingestion
failures are HTTP failures: `400` unparseable, `413` too large, `502` embedder
failure, `503` at capacity, `504` past the ingestion timeout.
### Point endpoints replace the older `/chunks` sketches
@@ -228,7 +232,7 @@ Important status codes:
| Status | Use |
|---|---|
| `202 Accepted` | Ingestion accepted as a job. |
| `201 Created` | Ingestion completed inline (ADR-0017). |
| `400 Bad Request` | Invalid domain/filter combinations or unsupported file type. |
| `401 Unauthorized` | Missing/invalid API key. |
| `403 Forbidden` | Valid key without required scope. |
@@ -252,8 +256,8 @@ and Qdrant operations can be correlated.
records — while preserving the chunk payload schema underneath.
- `/threads/{thread_id}/runs` remains compatible with the LangGraph thread/run
model already chosen in ADR-0007.
- Job-shaped file ingestion lets the first implementation be simple while
keeping room for a durable worker without breaking clients.
- Inline file ingestion (ADR-0017) gives callers a terminal result in one
request, with failures surfaced as ordinary HTTP errors.
- Router-level dependencies and typed FastAPI dependencies keep auth, tenant
resolution, sessions, and scopes reusable instead of repeated per endpoint.
@@ -266,8 +270,9 @@ and Qdrant operations can be correlated.
public product API.
- API-key auth in Postgres adds a database lookup to every request unless
short-lived caching is introduced. Caching must preserve revocation semantics.
- A job-shaped ingestion contract needs a job status store even if the initial
implementation processes inline.
- Inline ingestion ties the upload's duration to proxy/client timeouts, and
moving back to a queued `202` contract later is a breaking change for clients
(see ADR-0017's trigger list). The `ingestion_jobs` store is kept either way.
- The REST layer now depends on the tenant/API-key, ingestion-job, audit, and
usage tables defined in [ADR-0009](0009-postgres-sqlalchemy-alembic-schema.md).
@@ -288,10 +293,12 @@ and Qdrant operations can be correlated.
- **Expose one generic `/v1/qdrant/*` proxy**: rejected. It would leak Qdrant's
full API surface, bypass tenant/scoping rules too easily, and couple clients
to storage operations the service should hide.
- **Synchronous file ingestion only**: rejected as the contract. It is simpler
to implement, but embedding and late-interaction vector generation can be
slow enough to exceed HTTP timeouts. The job-shaped response gives the
implementation room to evolve.
- **Synchronous file ingestion only**: originally rejected here on the grounds
that embedding and late-interaction vector generation can exceed HTTP timeouts;
**adopted** by ADR-0017 for the first slice. Dense embedding is async network
I/O that batches and runs concurrently, and late-interaction vectors are not
populated at ingest yet — which is what made the original objection decisive
and is now the named trigger for reverting to a job-shaped contract.
- **Create threads with `POST /v1/threads`**: rejected for now. The main
backend owns conversation/session records, and LangGraph can create a
checkpoint sequence on the first run for a `thread_id`. A create endpoint