chatbot_v3/docs/adr/0016-testing-strategy-and-quality-gates.md

0016. Testing strategy and quality gates

Status

Proposed

Context

The project has ADRs for tenant-scoped ingestion, explicit resource ownership, MinIO object storage, transactional outbox dispatch, RabbitMQ workers, Qdrant indexing, and a modular monolith. It has no test runner, test fixtures, or executable test suite yet.

The first CSV ingestion slice has correctness properties that cannot be left to manual testing: Alembic is the only schema-management path; tenant identity is trusted server-side context; a file, job, and outbox event commit atomically; workers are safe under at-least-once delivery; and generated Qdrant points are idempotent and tenant-filtered. ADR-0015 already reserves a test layout by boundary, while ADR-0012 requires explicit dependencies and resource lifetimes that should make tests practical without import-time client patching.

Tests need to give fast feedback during implementation without replacing integration coverage with mocks or making routine development depend on Docker, provider credentials, live models, or Langfuse availability.

ADR-0014's transactional-outbox decision controls ingestion dispatch. The upload path records durable dispatch intent; a separate outbox publisher publishes it.

Decision

Use pytest with explicit async support

Use pytest as the test runner. Add pytest, pytest-asyncio, httpx, asgi-lifespan, testcontainers, pytest-timeout, and pytest-cov as development dependencies.

Configure pytest-asyncio in strict mode. Async tests and fixtures must be explicit. FastAPI tests use httpx.AsyncClient, ASGITransport, and LifespanManager so they exercise application startup and shutdown according to ADR-0012. FastAPI.TestClient is not the default project test client.

Register these markers:

• one primary boundary marker per test: unit, integration, or e2e;
• postgres, minio, rabbitmq, or qdrant for the real service used by an integration test;
• slow only where a test materially exceeds the normal integration feedback target;
• live_provider for an opt-in, credential-gated external-provider smoke test.

Name tests test_<unit>_<scenario>_<outcome> and use Arrange–Act–Assert.

Test by architectural boundary

Use the test layout reserved by ADR-0015, with shared support for fixtures and assertions:

tests/
├── conftest.py
├── fakes.py
├── support/
│   ├── factories.py
│   └── assertions.py
├── unit/
│   ├── application/
│   └── agent/
├── integration/
│   ├── postgres/
│   ├── minio/
│   ├── rabbitmq/
│   └── qdrant/
└── e2e/

• Unit tests are the default development feedback loop. They cover pure application policy, validation, lifecycle transitions, tenant propagation, deterministic IDs, CSV chunking, event construction, and LangGraph control-flow policy.
• Integration tests validate a production infrastructure adapter against its real backing service and its deployment-relevant behavior.
• End-to-end tests prove a small vertical-slice acceptance contract through the real application composition. They do not replace faster unit or adapter tests.

Hand-written fakes and spies implement narrow application-owned ports, not MinIO, RabbitMQ, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder, retrieval, clock, and UUID fakes make normal test runs deterministic.

Apply pragmatic TDD

For application behavior, HTTP contracts, database migrations, reliability rules, and defects, first write a focused failing test that describes the observable requirement. Make the smallest change that passes it, then refactor while the relevant suite is green.

TDD applies to behavior and regressions, not as an artificial ritual for pure refactors or configuration-only changes with no observable behavior change. Those changes must preserve and extend existing relevant coverage as needed.

Before introducing a concrete adapter, write tests for the consuming application port. Before each Alembic migration, write the empty-database migration test or extend the existing migration test. Add the corresponding real-adapter integration test before declaring that boundary complete.

Use disposable real infrastructure in integration tests

Use Testcontainers as the standard automated integration-test resource mechanism for Postgres, MinIO, RabbitMQ, and Qdrant.

• Tests never connect to a developer's local services or Langfuse-owned storage and credentials.
• Start containers at suite or session scope, then isolate data per test with unique data, object-key prefixes, exchange/queue/binding names, and collection names.
• Disable parallel integration execution until fixture isolation and cleanup are proven worker-safe.
• Fixtures expose typed settings or connection values. Tests create production adapters through their normal constructors.

Docker Compose remains the mechanism for manual local validation and a later, serialized operational smoke test where web, outbox-publisher, and worker run as independent processes. It is not the default pytest fixture mechanism.

Treat invariants as reusable contracts

Test the following requirements at the applicable application, adapter, and E2E boundaries:

• Tenant identity comes from server-side authenticated context. Request payloads, query parameters, object metadata, and broker messages cannot override it.
• Cross-tenant access does not disclose tenant-owned data. Public routes normally return 404 for inaccessible resources.
• Alembic creates the schema from an empty database. Tests never use Base.metadata.create_all(), and FastAPI startup performs readiness checks only, never DDL.
• The upload transaction records source_files, a queued ingestion_jobs row, and an unpublished outbox_events row atomically. The HTTP route does not directly publish the ingestion event.
• Broker messages contain durable identifiers and correlation metadata only. The worker reloads job and source-file records from Postgres before tenant-scoped side effects.
• Worker acknowledgement follows durable progress or terminal-state persistence. Duplicate publication and redelivery do not regress terminal jobs, inflate counters, or create duplicate logical chunks.
• MinIO keys are server-derived internal paths. Qdrant reads and mutations use a server-derived tenant filter, deterministic point IDs, and upsert semantics.

Keep correctness tests separate from model-quality evaluation

Pytest verifies deterministic application behavior: graph policy, schemas, redaction, correlation metadata, retry budgets, and graceful observability failure handling. Normal pytest runs never call a paid or live model provider.

Langfuse datasets and experiments evaluate prompts, model behavior, retrieval, citations, and response quality. They are promotion evidence, not a replacement for application correctness tests. Live-provider smoke tests, if introduced, are opt-in, credential-gated, rate-limited, and excluded from ordinary local and pull-request runs.

Establish phased quality gates

Initially require Ruff format checking, Ruff linting, Ty type checking, and unit tests. Require Docker-capable integration tests as their adapters are implemented. Run the separate-process Compose E2E smoke as a serialized pre-release or scheduled gate until it is reliable enough for every pull request.

Collect coverage reports but do not set a percentage threshold before the first vertical slice has meaningful implementation. Later introduce a scoped, ratcheting threshold rather than encouraging low-value coverage.

Consequences

Positive

• Unit tests provide fast, deterministic TDD feedback for core application behavior.
• Real-service tests cover the behaviors least safe to simulate: Alembic migrations, object storage, RabbitMQ acknowledgements/redelivery, and Qdrant filtering/upserts.
• Explicit fakes reinforce the dependency direction and resource ownership rules from ADR-0012 and ADR-0015.
• The ingestion path has concrete tenant-isolation and reliability contracts, rather than only a happy-path demonstration.
• Live model quality can improve through Langfuse experiments without making application tests nondeterministic or expensive.

Negative

• Docker is required for integration and E2E suites.
• Testcontainers add setup time and require careful fixture cleanup.
• Maintaining real-service coverage and test data isolation adds engineering effort.
• E2E tests do not prove semantic quality of LLM responses; that remains an evaluation responsibility.

Alternatives Considered

• Mock all external SDKs: rejected. Mocks cannot prove migrations, real RabbitMQ acknowledgement/redelivery behavior, MinIO semantics, or Qdrant tenant filtering.
• Use full-stack Compose tests only: rejected. They are slow and opaque for the default development loop and make failures difficult to localize.
• Run all integration containers on every pytest invocation: rejected. Test boundaries should be selected deliberately for fast local feedback.
• Use live model providers in routine tests: rejected because they are nondeterministic, costly, slow, credential-dependent, and hard to assert.
• Use Langfuse as the primary regression-test runner: rejected. Langfuse is the quality/evaluation plane; pytest remains the deterministic application test framework.
• Create schemas with Base.metadata.create_all() in fixtures: rejected. It bypasses the production Alembic migration path.
• Require strict test-first work for every non-behavioral refactor: rejected. TDD should protect observable behavior and regressions, not add ceremony where no behavior changes.