docs(architecture): adopt inline synchronous ingestion (ADR-0017)
Why: - ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path. Changes: - new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it. Impact: - no broker, outbox, or worker code; rabbitmq test marker removed.
This commit is contained in:
@@ -164,16 +164,20 @@ metadata such as `domain`. File validation is server-side:
|
||||
- Derive `tenant_id`, `created_by`, and `updated_by` from `AuthContext`, not
|
||||
from form fields.
|
||||
|
||||
Ingestion may be slow because it parses, chunks, embeds, and writes many
|
||||
Qdrant points. The REST contract is job-shaped even if the first
|
||||
implementation runs inline:
|
||||
Ingestion parses, chunks, embeds, and writes many Qdrant points.
|
||||
[ADR-0017](0017-synchronous-ingestion-in-the-request-path.md) supersedes the
|
||||
job-shaped contract originally specified here: ingestion runs inline and the
|
||||
response is terminal.
|
||||
|
||||
```text
|
||||
202 Accepted -> { file_id, ingestion_job_id, status: "queued" | "running" }
|
||||
201 Created -> { file_id, ingestion_job_id, status: "succeeded", chunks_indexed }
|
||||
```
|
||||
|
||||
A durable worker/job queue can be added later without changing the API
|
||||
contract.
|
||||
`ingestion_job_id` is retained so the attempt stays inspectable via
|
||||
`GET /v1/files/{file_id}`, and so a future move back to a queued `202 Accepted`
|
||||
contract (ADR-0014) is additive for clients that already read it. Ingestion
|
||||
failures are HTTP failures: `400` unparseable, `413` too large, `502` embedder
|
||||
failure, `503` at capacity, `504` past the ingestion timeout.
|
||||
|
||||
### Point endpoints replace the older `/chunks` sketches
|
||||
|
||||
@@ -228,7 +232,7 @@ Important status codes:
|
||||
|
||||
| Status | Use |
|
||||
|---|---|
|
||||
| `202 Accepted` | Ingestion accepted as a job. |
|
||||
| `201 Created` | Ingestion completed inline (ADR-0017). |
|
||||
| `400 Bad Request` | Invalid domain/filter combinations or unsupported file type. |
|
||||
| `401 Unauthorized` | Missing/invalid API key. |
|
||||
| `403 Forbidden` | Valid key without required scope. |
|
||||
@@ -252,8 +256,8 @@ and Qdrant operations can be correlated.
|
||||
records — while preserving the chunk payload schema underneath.
|
||||
- `/threads/{thread_id}/runs` remains compatible with the LangGraph thread/run
|
||||
model already chosen in ADR-0007.
|
||||
- Job-shaped file ingestion lets the first implementation be simple while
|
||||
keeping room for a durable worker without breaking clients.
|
||||
- Inline file ingestion (ADR-0017) gives callers a terminal result in one
|
||||
request, with failures surfaced as ordinary HTTP errors.
|
||||
- Router-level dependencies and typed FastAPI dependencies keep auth, tenant
|
||||
resolution, sessions, and scopes reusable instead of repeated per endpoint.
|
||||
|
||||
@@ -266,8 +270,9 @@ and Qdrant operations can be correlated.
|
||||
public product API.
|
||||
- API-key auth in Postgres adds a database lookup to every request unless
|
||||
short-lived caching is introduced. Caching must preserve revocation semantics.
|
||||
- A job-shaped ingestion contract needs a job status store even if the initial
|
||||
implementation processes inline.
|
||||
- Inline ingestion ties the upload's duration to proxy/client timeouts, and
|
||||
moving back to a queued `202` contract later is a breaking change for clients
|
||||
(see ADR-0017's trigger list). The `ingestion_jobs` store is kept either way.
|
||||
- The REST layer now depends on the tenant/API-key, ingestion-job, audit, and
|
||||
usage tables defined in [ADR-0009](0009-postgres-sqlalchemy-alembic-schema.md).
|
||||
|
||||
@@ -288,10 +293,12 @@ and Qdrant operations can be correlated.
|
||||
- **Expose one generic `/v1/qdrant/*` proxy**: rejected. It would leak Qdrant's
|
||||
full API surface, bypass tenant/scoping rules too easily, and couple clients
|
||||
to storage operations the service should hide.
|
||||
- **Synchronous file ingestion only**: rejected as the contract. It is simpler
|
||||
to implement, but embedding and late-interaction vector generation can be
|
||||
slow enough to exceed HTTP timeouts. The job-shaped response gives the
|
||||
implementation room to evolve.
|
||||
- **Synchronous file ingestion only**: originally rejected here on the grounds
|
||||
that embedding and late-interaction vector generation can exceed HTTP timeouts;
|
||||
**adopted** by ADR-0017 for the first slice. Dense embedding is async network
|
||||
I/O that batches and runs concurrently, and late-interaction vectors are not
|
||||
populated at ingest yet — which is what made the original objection decisive
|
||||
and is now the named trigger for reverting to a job-shaped contract.
|
||||
- **Create threads with `POST /v1/threads`**: rejected for now. The main
|
||||
backend owns conversation/session records, and LangGraph can create a
|
||||
checkpoint sequence on the first run for a `thread_id`. A create endpoint
|
||||
|
||||
Reference in New Issue
Block a user