docs(architecture): adopt inline synchronous ingestion (ADR-0017)

Why:
- ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path.

Changes:
- new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it.

Impact:
- no broker, outbox, or worker code; rabbitmq test marker removed.
This commit is contained in:
2026-08-16 11:53:59 +03:30
parent e2322a2909
commit fd70ad01af
9 changed files with 542 additions and 154 deletions

View File

@@ -4,17 +4,24 @@
Proposed
> Amended by [ADR-0017](0017-synchronous-ingestion-in-the-request-path.md):
> there is no broker, outbox, or queue, so the `rabbitmq` marker and
> `tests/integration/rabbitmq/` are not carried until ADR-0014 is adopted.
> Ingestion is exercised through the upload request itself, which now returns a
> terminal result. Every reliability invariant below still applies — retrying an
> upload stands in for redelivery.
## Context
The project has ADRs for tenant-scoped ingestion, explicit resource ownership,
MinIO object storage, transactional outbox dispatch, RabbitMQ workers,
MinIO object storage, inline request-path ingestion,
Qdrant indexing, and a modular monolith. It has no test runner, test fixtures,
or executable test suite yet.
The first CSV ingestion slice has correctness properties that cannot be left to
manual testing: Alembic is the only schema-management path; tenant identity is
trusted server-side context; a file, job, and outbox event commit atomically;
workers are safe under at-least-once delivery; and generated Qdrant points are
trusted server-side context; the source file and its job row commit
atomically; job execution is safe under at-least-once semantics; and generated Qdrant points are
idempotent and tenant-filtered. ADR-0015 already reserves a test layout by
boundary, while ADR-0012 requires explicit dependencies and resource lifetimes
that should make tests practical without import-time client patching.
@@ -23,8 +30,11 @@ Tests need to give fast feedback during implementation without replacing
integration coverage with mocks or making routine development depend on Docker,
provider credentials, live models, or Langfuse availability.
ADR-0014's transactional-outbox decision controls ingestion dispatch. The upload
path records durable dispatch intent; a separate outbox publisher publishes it.
ADR-0017 controls ingestion: `POST /v1/files` parses, chunks, embeds, and
indexes inline, then returns a terminal `201`. `ingestion_jobs` records the
attempt. There is no outbox, queue, or publisher to test — but the request's
bounds (size, timeout, capacity) and its two-transaction shape are testable
contracts.
## Decision
@@ -42,8 +52,8 @@ ADR-0012. `FastAPI.TestClient` is not the default project test client.
Register these markers:
- one primary boundary marker per test: `unit`, `integration`, or `e2e`;
- `postgres`, `minio`, `rabbitmq`, or `qdrant` for the real service used by an
integration test;
- `postgres`, `minio`, or `qdrant` for the real service used by an integration
test (a `rabbitmq` marker returns with ADR-0014);
- `slow` only where a test materially exceeds the normal integration feedback
target;
- `live_provider` for an opt-in, credential-gated external-provider smoke test.
@@ -68,7 +78,6 @@ tests/
├── integration/
│ ├── postgres/
│ ├── minio/
│ ├── rabbitmq/
│ └── qdrant/
└── e2e/
```
@@ -84,7 +93,7 @@ tests/
tests.
Hand-written fakes and spies implement narrow application-owned ports, not
MinIO, RabbitMQ, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder,
MinIO, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder,
retrieval, clock, and UUID fakes make normal test runs deterministic.
### Apply pragmatic TDD
@@ -106,7 +115,7 @@ real-adapter integration test before declaring that boundary complete.
### Use disposable real infrastructure in integration tests
Use Testcontainers as the standard automated integration-test resource mechanism
for Postgres, MinIO, RabbitMQ, and Qdrant.
for Postgres, MinIO, and Qdrant.
- Tests never connect to a developer's local services or Langfuse-owned storage
and credentials.
@@ -119,8 +128,8 @@ for Postgres, MinIO, RabbitMQ, and Qdrant.
adapters through their normal constructors.
Docker Compose remains the mechanism for manual local validation and a later,
serialized operational smoke test where web, outbox-publisher, and worker run as
independent processes. It is not the default pytest fixture mechanism.
serialized operational smoke test of the running web process, which performs
ingestion inline under ADR-0017. It is not the default pytest fixture mechanism.
### Treat invariants as reusable contracts
@@ -128,21 +137,25 @@ Test the following requirements at the applicable application, adapter, and E2E
boundaries:
- Tenant identity comes from server-side authenticated context. Request payloads,
query parameters, object metadata, and broker messages cannot override it.
query parameters, object metadata, and job payloads cannot override it.
- Cross-tenant access does not disclose tenant-owned data. Public routes normally
return `404` for inaccessible resources.
- Alembic creates the schema from an empty database. Tests never use
`Base.metadata.create_all()`, and FastAPI startup performs readiness checks only,
never DDL.
- The upload transaction records `source_files`, a queued `ingestion_jobs` row,
and an unpublished `outbox_events` row atomically. The HTTP route does not
directly publish the ingestion event.
- Broker messages contain durable identifiers and correlation metadata only. The
worker reloads job and source-file records from Postgres before tenant-scoped
side effects.
- Worker acknowledgement follows durable progress or terminal-state persistence.
Duplicate publication and redelivery do not regress terminal jobs, inflate
counters, or create duplicate logical chunks.
- The first upload transaction records `source_files` and a `running`
`ingestion_jobs` row atomically, and commits before any parse/embed work; no
session or transaction is held open across that work.
- Every terminating path — success, parse failure, embedder failure, timeout —
writes a terminal job status and its `ingestion_job_events` row. A job is
never left in `running` by a handled failure.
- Each bound maps to its status code: oversized upload `413`, capacity `503`,
timeout `504`, embedder failure `502`.
- Retrying an upload does not regress terminal jobs, inflate counters, or create
duplicate logical chunks; identical content is recognized rather than
re-ingested.
- Embedding is batched and concurrency-bounded rather than serial per chunk, and
blocking work is executed off the event loop under an explicit limiter.
- MinIO keys are server-derived internal paths. Qdrant reads and mutations use a
server-derived tenant filter, deterministic point IDs, and upsert semantics.
@@ -176,8 +189,8 @@ ratcheting threshold rather than encouraging low-value coverage.
- Unit tests provide fast, deterministic TDD feedback for core application
behavior.
- Real-service tests cover the behaviors least safe to simulate: Alembic
migrations, object storage, RabbitMQ acknowledgements/redelivery, and Qdrant
filtering/upserts.
migrations, object storage, transaction boundaries under real sessions, and
Qdrant filtering/upserts.
- Explicit fakes reinforce the dependency direction and resource ownership rules
from ADR-0012 and ADR-0015.
- The ingestion path has concrete tenant-isolation and reliability contracts,
@@ -197,8 +210,8 @@ ratcheting threshold rather than encouraging low-value coverage.
## Alternatives Considered
- **Mock all external SDKs**: rejected. Mocks cannot prove migrations, real
RabbitMQ acknowledgement/redelivery behavior, MinIO semantics, or Qdrant
tenant filtering.
transaction/connection behavior, MinIO semantics, or Qdrant tenant
filtering.
- **Use full-stack Compose tests only**: rejected. They are slow and opaque for
the default development loop and make failures difficult to localize.
- **Run all integration containers on every pytest invocation**: rejected. Test