chatbot_v3/docs/adr/0016-testing-strategy-and-quality-gates.md

0016. Testing strategy and quality gates

Status

Proposed

Amended by ADR-0017: there is no broker, outbox, or queue, so the rabbitmq marker and tests/integration/rabbitmq/ are not carried until ADR-0014 is adopted. Ingestion is exercised through the upload request itself, which now returns a terminal result. Every reliability invariant below still applies — retrying an upload stands in for redelivery.

Context

The project has ADRs for tenant-scoped ingestion, explicit resource ownership, MinIO object storage, inline request-path ingestion, Qdrant indexing, and a modular monolith. It has no test runner, test fixtures, or executable test suite yet.

The first CSV ingestion slice has correctness properties that cannot be left to manual testing: Alembic is the only schema-management path; tenant identity is trusted server-side context; the source file and its job row commit atomically; job execution is safe under at-least-once semantics; and generated Qdrant points are idempotent and tenant-filtered. ADR-0015 already reserves a test layout by boundary, while ADR-0012 requires explicit dependencies and resource lifetimes that should make tests practical without import-time client patching.

Tests need to give fast feedback during implementation without replacing integration coverage with mocks or making routine development depend on Docker, provider credentials, live models, or Langfuse availability.

ADR-0017 controls ingestion: POST /v1/files parses, chunks, embeds, and indexes inline, then returns a terminal 201. ingestion_jobs records the attempt. There is no outbox, queue, or publisher to test — but the request's bounds (size, timeout, capacity) and its two-transaction shape are testable contracts.

Decision

Use pytest with explicit async support

Use pytest as the test runner. Add pytest, pytest-asyncio, httpx, asgi-lifespan, testcontainers, pytest-timeout, and pytest-cov as development dependencies.

Configure pytest-asyncio in strict mode. Async tests and fixtures must be explicit. FastAPI tests use httpx.AsyncClient, ASGITransport, and LifespanManager so they exercise application startup and shutdown according to ADR-0012. FastAPI.TestClient is not the default project test client.

Register these markers:

• one primary boundary marker per test: unit, integration, or e2e;
• postgres, minio, or qdrant for the real service used by an integration test (a rabbitmq marker returns with ADR-0014);
• slow only where a test materially exceeds the normal integration feedback target;
• live_provider for an opt-in, credential-gated external-provider smoke test.

Name tests test_<unit>_<scenario>_<outcome> and use Arrange–Act–Assert.

Test by architectural boundary

Use the test layout reserved by ADR-0015, with shared support for fixtures and assertions:

tests/
├── conftest.py
├── fakes.py
├── support/
│   ├── factories.py
│   └── assertions.py
├── unit/
│   ├── application/
│   └── agent/
├── integration/
│   ├── postgres/
│   ├── minio/
│   └── qdrant/
└── e2e/

• Unit tests are the default development feedback loop. They cover pure application policy, validation, lifecycle transitions, tenant propagation, deterministic IDs, CSV chunking, event construction, and LangGraph control-flow policy.
• Integration tests validate a production infrastructure adapter against its real backing service and its deployment-relevant behavior.
• End-to-end tests prove a small vertical-slice acceptance contract through the real application composition. They do not replace faster unit or adapter tests.

Hand-written fakes and spies implement narrow application-owned ports, not MinIO, Qdrant, or model SDK-shaped interfaces. Scripted model, embedder, retrieval, clock, and UUID fakes make normal test runs deterministic.

Apply pragmatic TDD

For application behavior, HTTP contracts, database migrations, reliability rules, and defects, first write a focused failing test that describes the observable requirement. Make the smallest change that passes it, then refactor while the relevant suite is green.

TDD applies to behavior and regressions, not as an artificial ritual for pure refactors or configuration-only changes with no observable behavior change. Those changes must preserve and extend existing relevant coverage as needed.

Before introducing a concrete adapter, write tests for the consuming application port. Before each Alembic migration, write the empty-database migration test or extend the existing migration test. Add the corresponding real-adapter integration test before declaring that boundary complete.

Use disposable real infrastructure in integration tests

Use Testcontainers as the standard automated integration-test resource mechanism for Postgres, MinIO, and Qdrant.

• Tests never connect to a developer's local services or Langfuse-owned storage and credentials.
• Start containers at suite or session scope, then isolate data per test with unique data, object-key prefixes, exchange/queue/binding names, and collection names.
• Disable parallel integration execution until fixture isolation and cleanup are proven worker-safe.
• Fixtures expose typed settings or connection values. Tests create production adapters through their normal constructors.

Docker Compose remains the mechanism for manual local validation and a later, serialized operational smoke test of the running web process, which performs ingestion inline under ADR-0017. It is not the default pytest fixture mechanism.

Treat invariants as reusable contracts

Test the following requirements at the applicable application, adapter, and E2E boundaries:

• Tenant identity comes from server-side authenticated context. Request payloads, query parameters, object metadata, and job payloads cannot override it.
• Cross-tenant access does not disclose tenant-owned data. Public routes normally return 404 for inaccessible resources.
• Alembic creates the schema from an empty database. Tests never use Base.metadata.create_all(), and FastAPI startup performs readiness checks only, never DDL.
• The first upload transaction records source_files and a running ingestion_jobs row atomically, and commits before any parse/embed work; no session or transaction is held open across that work.
• Every terminating path — success, parse failure, embedder failure, timeout — writes a terminal job status and its ingestion_job_events row. A job is never left in running by a handled failure.
• Each bound maps to its status code: oversized upload 413, capacity 503, timeout 504, embedder failure 502.
• Retrying an upload does not regress terminal jobs, inflate counters, or create duplicate logical chunks; identical content is recognized rather than re-ingested.
• Embedding is batched and concurrency-bounded rather than serial per chunk, and blocking work is executed off the event loop under an explicit limiter.
• MinIO keys are server-derived internal paths. Qdrant reads and mutations use a server-derived tenant filter, deterministic point IDs, and upsert semantics.

Keep correctness tests separate from model-quality evaluation

Pytest verifies deterministic application behavior: graph policy, schemas, redaction, correlation metadata, retry budgets, and graceful observability failure handling. Normal pytest runs never call a paid or live model provider.

Langfuse datasets and experiments evaluate prompts, model behavior, retrieval, citations, and response quality. They are promotion evidence, not a replacement for application correctness tests. Live-provider smoke tests, if introduced, are opt-in, credential-gated, rate-limited, and excluded from ordinary local and pull-request runs.

Establish phased quality gates

Initially require Ruff format checking, Ruff linting, Ty type checking, and unit tests. Require Docker-capable integration tests as their adapters are implemented. Run the separate-process Compose E2E smoke as a serialized pre-release or scheduled gate until it is reliable enough for every pull request.

Collect coverage reports but do not set a percentage threshold before the first vertical slice has meaningful implementation. Later introduce a scoped, ratcheting threshold rather than encouraging low-value coverage.

Consequences

Positive

• Unit tests provide fast, deterministic TDD feedback for core application behavior.
• Real-service tests cover the behaviors least safe to simulate: Alembic migrations, object storage, transaction boundaries under real sessions, and Qdrant filtering/upserts.
• Explicit fakes reinforce the dependency direction and resource ownership rules from ADR-0012 and ADR-0015.
• The ingestion path has concrete tenant-isolation and reliability contracts, rather than only a happy-path demonstration.
• Live model quality can improve through Langfuse experiments without making application tests nondeterministic or expensive.

Negative

• Docker is required for integration and E2E suites.
• Testcontainers add setup time and require careful fixture cleanup.
• Maintaining real-service coverage and test data isolation adds engineering effort.
• E2E tests do not prove semantic quality of LLM responses; that remains an evaluation responsibility.

Alternatives Considered

• Mock all external SDKs: rejected. Mocks cannot prove migrations, real transaction/connection behavior, MinIO semantics, or Qdrant tenant filtering.
• Use full-stack Compose tests only: rejected. They are slow and opaque for the default development loop and make failures difficult to localize.
• Run all integration containers on every pytest invocation: rejected. Test boundaries should be selected deliberately for fast local feedback.
• Use live model providers in routine tests: rejected because they are nondeterministic, costly, slow, credential-dependent, and hard to assert.
• Use Langfuse as the primary regression-test runner: rejected. Langfuse is the quality/evaluation plane; pytest remains the deterministic application test framework.
• Create schemas with Base.metadata.create_all() in fixtures: rejected. It bypasses the production Alembic migration path.
• Require strict test-first work for every non-behavioral refactor: rejected. TDD should protect observable behavior and regressions, not add ceremony where no behavior changes.