docs(architecture): adopt inline synchronous ingestion (ADR-0017)
Why: - ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path. Changes: - new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it. Impact: - no broker, outbox, or worker code; rabbitmq test marker removed.
This commit is contained in:
@@ -4,17 +4,24 @@
|
||||
|
||||
Proposed
|
||||
|
||||
> Amended by [ADR-0017](0017-synchronous-ingestion-in-the-request-path.md):
|
||||
> there is no broker, outbox, or queue, so the `rabbitmq` marker and
|
||||
> `tests/integration/rabbitmq/` are not carried until ADR-0014 is adopted.
|
||||
> Ingestion is exercised through the upload request itself, which now returns a
|
||||
> terminal result. Every reliability invariant below still applies — retrying an
|
||||
> upload stands in for redelivery.
|
||||
|
||||
## Context
|
||||
|
||||
The project has ADRs for tenant-scoped ingestion, explicit resource ownership,
|
||||
MinIO object storage, transactional outbox dispatch, RabbitMQ workers,
|
||||
MinIO object storage, inline request-path ingestion,
|
||||
Qdrant indexing, and a modular monolith. It has no test runner, test fixtures,
|
||||
or executable test suite yet.
|
||||
|
||||
The first CSV ingestion slice has correctness properties that cannot be left to
|
||||
manual testing: Alembic is the only schema-management path; tenant identity is
|
||||
trusted server-side context; a file, job, and outbox event commit atomically;
|
||||
workers are safe under at-least-once delivery; and generated Qdrant points are
|
||||
trusted server-side context; the source file and its job row commit
|
||||
atomically; job execution is safe under at-least-once semantics; and generated Qdrant points are
|
||||
idempotent and tenant-filtered. ADR-0015 already reserves a test layout by
|
||||
boundary, while ADR-0012 requires explicit dependencies and resource lifetimes
|
||||
that should make tests practical without import-time client patching.
|
||||
@@ -23,8 +30,11 @@ Tests need to give fast feedback during implementation without replacing
|
||||
integration coverage with mocks or making routine development depend on Docker,
|
||||
provider credentials, live models, or Langfuse availability.
|
||||
|
||||
ADR-0014's transactional-outbox decision controls ingestion dispatch. The upload
|
||||
path records durable dispatch intent; a separate outbox publisher publishes it.
|
||||
ADR-0017 controls ingestion: `POST /v1/files` parses, chunks, embeds, and
|
||||
indexes inline, then returns a terminal `201`. `ingestion_jobs` records the
|
||||
attempt. There is no outbox, queue, or publisher to test — but the request's
|
||||
bounds (size, timeout, capacity) and its two-transaction shape are testable
|
||||
contracts.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -42,8 +52,8 @@ ADR-0012. `FastAPI.TestClient` is not the default project test client.
|
||||
Register these markers:
|
||||
|
||||
- one primary boundary marker per test: `unit`, `integration`, or `e2e`;
|
||||
- `postgres`, `minio`, `rabbitmq`, or `qdrant` for the real service used by an
|
||||
integration test;
|
||||
- `postgres`, `minio`, or `qdrant` for the real service used by an integration
|
||||
test (a `rabbitmq` marker returns with ADR-0014);
|
||||
- `slow` only where a test materially exceeds the normal integration feedback
|
||||
target;
|
||||
- `live_provider` for an opt-in, credential-gated external-provider smoke test.
|
||||
@@ -68,7 +78,6 @@ tests/
|
||||
├── integration/
|
||||
│ ├── postgres/
|
||||
│ ├── minio/
|
||||
│ ├── rabbitmq/
|
||||
│ └── qdrant/
|
||||
└── e2e/
|
||||
```
|
||||
@@ -84,7 +93,7 @@ tests/
|
||||
tests.
|
||||
|
||||
Hand-written fakes and spies implement narrow application-owned ports, not
|
||||
MinIO, RabbitMQ, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder,
|
||||
MinIO, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder,
|
||||
retrieval, clock, and UUID fakes make normal test runs deterministic.
|
||||
|
||||
### Apply pragmatic TDD
|
||||
@@ -106,7 +115,7 @@ real-adapter integration test before declaring that boundary complete.
|
||||
### Use disposable real infrastructure in integration tests
|
||||
|
||||
Use Testcontainers as the standard automated integration-test resource mechanism
|
||||
for Postgres, MinIO, RabbitMQ, and Qdrant.
|
||||
for Postgres, MinIO, and Qdrant.
|
||||
|
||||
- Tests never connect to a developer's local services or Langfuse-owned storage
|
||||
and credentials.
|
||||
@@ -119,8 +128,8 @@ for Postgres, MinIO, RabbitMQ, and Qdrant.
|
||||
adapters through their normal constructors.
|
||||
|
||||
Docker Compose remains the mechanism for manual local validation and a later,
|
||||
serialized operational smoke test where web, outbox-publisher, and worker run as
|
||||
independent processes. It is not the default pytest fixture mechanism.
|
||||
serialized operational smoke test of the running web process, which performs
|
||||
ingestion inline under ADR-0017. It is not the default pytest fixture mechanism.
|
||||
|
||||
### Treat invariants as reusable contracts
|
||||
|
||||
@@ -128,21 +137,25 @@ Test the following requirements at the applicable application, adapter, and E2E
|
||||
boundaries:
|
||||
|
||||
- Tenant identity comes from server-side authenticated context. Request payloads,
|
||||
query parameters, object metadata, and broker messages cannot override it.
|
||||
query parameters, object metadata, and job payloads cannot override it.
|
||||
- Cross-tenant access does not disclose tenant-owned data. Public routes normally
|
||||
return `404` for inaccessible resources.
|
||||
- Alembic creates the schema from an empty database. Tests never use
|
||||
`Base.metadata.create_all()`, and FastAPI startup performs readiness checks only,
|
||||
never DDL.
|
||||
- The upload transaction records `source_files`, a queued `ingestion_jobs` row,
|
||||
and an unpublished `outbox_events` row atomically. The HTTP route does not
|
||||
directly publish the ingestion event.
|
||||
- Broker messages contain durable identifiers and correlation metadata only. The
|
||||
worker reloads job and source-file records from Postgres before tenant-scoped
|
||||
side effects.
|
||||
- Worker acknowledgement follows durable progress or terminal-state persistence.
|
||||
Duplicate publication and redelivery do not regress terminal jobs, inflate
|
||||
counters, or create duplicate logical chunks.
|
||||
- The first upload transaction records `source_files` and a `running`
|
||||
`ingestion_jobs` row atomically, and commits before any parse/embed work; no
|
||||
session or transaction is held open across that work.
|
||||
- Every terminating path — success, parse failure, embedder failure, timeout —
|
||||
writes a terminal job status and its `ingestion_job_events` row. A job is
|
||||
never left in `running` by a handled failure.
|
||||
- Each bound maps to its status code: oversized upload `413`, capacity `503`,
|
||||
timeout `504`, embedder failure `502`.
|
||||
- Retrying an upload does not regress terminal jobs, inflate counters, or create
|
||||
duplicate logical chunks; identical content is recognized rather than
|
||||
re-ingested.
|
||||
- Embedding is batched and concurrency-bounded rather than serial per chunk, and
|
||||
blocking work is executed off the event loop under an explicit limiter.
|
||||
- MinIO keys are server-derived internal paths. Qdrant reads and mutations use a
|
||||
server-derived tenant filter, deterministic point IDs, and upsert semantics.
|
||||
|
||||
@@ -176,8 +189,8 @@ ratcheting threshold rather than encouraging low-value coverage.
|
||||
- Unit tests provide fast, deterministic TDD feedback for core application
|
||||
behavior.
|
||||
- Real-service tests cover the behaviors least safe to simulate: Alembic
|
||||
migrations, object storage, RabbitMQ acknowledgements/redelivery, and Qdrant
|
||||
filtering/upserts.
|
||||
migrations, object storage, transaction boundaries under real sessions, and
|
||||
Qdrant filtering/upserts.
|
||||
- Explicit fakes reinforce the dependency direction and resource ownership rules
|
||||
from ADR-0012 and ADR-0015.
|
||||
- The ingestion path has concrete tenant-isolation and reliability contracts,
|
||||
@@ -197,8 +210,8 @@ ratcheting threshold rather than encouraging low-value coverage.
|
||||
## Alternatives Considered
|
||||
|
||||
- **Mock all external SDKs**: rejected. Mocks cannot prove migrations, real
|
||||
RabbitMQ acknowledgement/redelivery behavior, MinIO semantics, or Qdrant
|
||||
tenant filtering.
|
||||
transaction/connection behavior, MinIO semantics, or Qdrant tenant
|
||||
filtering.
|
||||
- **Use full-stack Compose tests only**: rejected. They are slow and opaque for
|
||||
the default development loop and make failures difficult to localize.
|
||||
- **Run all integration containers on every pytest invocation**: rejected. Test
|
||||
|
||||
Reference in New Issue
Block a user