Compare commits

..

30 Commits

Author SHA1 Message Date
0b1932f716 docs(claude): record plan 002 phase 3 in the project status
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 17:13:05 +03:30
73bdac0da2 test(points): cover soft delete, relinking, and the file sweep
Why:
- The failure modes worth testing are races, and they are cheap to force
  against the fake and expensive to observe anywhere else.

Changes:
- Unit tests for both boundaries, the repeat delete, a missing neighbour, the
  traversal property after several deletes, and — via a repository that bumps a
  rival's version before each apply — both the partial-apply repair and the
  unconvergent 409.
- One new shared contract scenario (a multi-point batch applies every patch,
  including nulling a pointer) so it runs against the fake and real Qdrant.
- HTTP tests against real Postgres and Qdrant for relinking, soft-not-hard
  delete, cross-tenant 404, and scope enforcement on both routes.
- create_source_file factory for tests addressing a file without uploading.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 17:12:56 +03:30
b58f4630f3 feat(points): add soft delete with neighbour relinking
Why:
- Plan 002 Phase 3. ADR-0002 makes delete soft by default and treats a partial
  neighbour relink as a defect, since ADR-0003's context-window expansion walks
  the previous/next pointer chain.

Changes:
- relinking.py computes the patches still missing between the state just read
  and the desired end state, so a normal delete, a repeat delete, and recovery
  from a half-applied batch are one path.
- deletion.py re-plans and re-applies up to three times, verifying by read-back,
  because Qdrant has no multi-point transaction and reports success for a
  filtered set_payload that matched nothing; exhausting the retries raises
  PointVersionConflictError (409).
- DELETE /v1/points/{point_id} and DELETE /v1/files/{file_id}, both on
  points:write. The file route sweeps points first, then marks the source_files
  row soft_deleted in its own short transaction.
- Log events carry ADR-0011 duration_ms plus rounds.

Impact:
- A deleted file's source_files row leaves 'active', so re-uploading the same
  bytes now re-ingests instead of matching the duplicate path.
- No migration; no point is ever removed from Qdrant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 17:12:43 +03:30
b25c15fefa docs(claude): record plan 002 phase 2 in the project status
Why:
- The status section described plan 002 as Phase 1 only. The /v1/points read
  routes, the query service, and the points scopes had shipped but were listed
  under "not built yet", so a fresh session would start from a wrong map of the
  codebase.

Changes:
- Record the five read routes, the points:read gating, and the 404-not-403
  mapping.
- Call out the three route-level rules that fail quietly when broken: /count and
  /search must precede /{point_id}, file_id is required on the listing, and the
  search query is Persian-folded before matching.
- Point at tests/support/point_contract.py as the place new repository behaviour
  belongs, since it runs against both the fake and real Qdrant.
- Narrow the "not built yet" list to Phases 3-6.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 16:04:59 +03:30
ba7921dd4e test(points): cover the read paths and the query service
Why:
- The isolation and pagination guarantees are the ones that fail silently, so
  they need tests that would actually notice.

Changes:
- API tests over real Postgres and Qdrant together, because the invariant worth
  testing spans both: the tenant Postgres derived is the only one Qdrant is ever
  queried with.
- Pagination holds when a point is inserted behind the cursor mid-listing --
  the defect an offset cursor would have.
- A route-order guard, since /{point_id} declared first turns count into a 422
  and nothing else in the suite would catch it.
- Unit tests for the query service against the fake, including the Arabic to
  Persian letterform fold and raising rather than returning None.

Impact:
- Suite goes to 325 passed, 3 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 15:09:31 +03:30
802bae429d docs(claude): record plan 002 phase 1 in the project status 2026-08-22 15:09:18 +03:30
43932b6562 docs(adr): record query normalization and file-scoped pagination
Why:
- Both are contracts callers depend on, not implementation details, and neither
  was written down. This repo treats the ADR as the source of truth rather than
  letting code diverge from it silently.

Changes:
- Record that the search query is folded the same way ingested content was, and
  why the alternative fails in the worst available way: an exact-looking query
  returning nothing, with no error and nothing in the logs to distinguish it
  from a genuine miss.
- Record that listing requires file_id and paginates by order_id value, and why
  an offset cursor repeats an already-served row under a concurrent insert.
- State that results carry no relevance score and no ranked order, so callers
  cannot read array position as relevance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 15:09:07 +03:30
3b9434faf4 feat(points): add the /v1/points read paths
Why:
- Ingestion writes points in bulk but nothing could read one back. Plan 002
  Phase 2 opens the read surface an admin frontend needs.

Changes:
- GET /v1/points/{point_id}, /v1/points?file_id=..., /v1/points/count,
  /v1/points/search, and /v1/files/{file_id}/points, all under points:read --
  the scope follows the data, so an upload key does not become a way to read
  every chunk of every file.
- The keyword query is Persian-normalized before matching, because ingestion
  letter-folds content at ingest and an unfolded query would return an empty
  result set silently rather than an error.
- file_id is required on the listing: the cursor is an order_id value and
  order_id is only unique within one file.
- PointNotFoundError maps to 404, never 403, so a cross-tenant point id is
  indistinguishable from a nonexistent one.
- Route order is load-bearing: /count and /search precede /{point_id}, or
  "count" is parsed as a UUID and fails 422.

Impact:
- Requires the content/is_active/chunk_index payload indexes, so a deployed
  environment needs qdrant_bootstrap re-run before search works.
- Keyword search returns no relevance score and no ranked order; callers must
  not read array position as relevance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 15:08:54 +03:30
5251990444 feat(tenant): grant points scopes on provisioned keys
Why:
- points:read and points:write are named in ADR-0009 but were absent from
  DEFAULT_SCOPES, so a provisioned key could not reach the read paths that
  follow. The runbook documented only files:write and domains:write, which
  understated what a default key can now do.

Changes:
- Add points:read and points:write to DEFAULT_SCOPES.
- Document the full scope table in the runbook, calling out that points:read
  grants the text of every chunk of every file -- so an upload-only key gets
  files:write alone.

Impact:
- Keys issued before this change keep their existing scopes; provisioning does
  not backfill. Reissue or widen an existing key explicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 15:08:39 +03:30
062fdd7ac1 refactor(test): share the Qdrant point-seeding helper
Why:
- The helper turning a SeedSpec into an upsertable ChunkPoint lived inside one
  integration test. A second suite now needs to seed real Qdrant the same way,
  and a copy would let the two drift.

Changes:
- Move `_chunk_point` into tests/support/point_contract.py as `chunk_point_for`,
  beside the `build_point` read model it derives its payload from.

Impact:
- Pure move. No behaviour change; enables the API suite that follows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 15:08:25 +03:30
923ac8e5d6 test(points): hold the fake and the Qdrant adapter to one shared contract
Why:
- Two parallel test files let a fake drift more permissive than the store it
  stands in for, so unit tests stay green while production diverges. Plan 002
  Phase 1's exit criterion is precisely that the two agree.

Changes:
- One scenario suite in tests/support/point_contract.py, run against
  FakePointRepository (unit) and QdrantPointRepository (integration). A
  divergence fails one of the two runs rather than hiding.
- The fake models the behaviours services branch on: the implied is_active read
  filter, value-based cursor pagination, and a stale version guard that matches
  nothing rather than raising -- the no-op Qdrant's filtered set_payload actually
  has, and the reason a service must read back to know its write landed.
- Patched points are re-validated rather than model_copy'd, so the fake holds a
  datetime where a read from real Qdrant returns one.
- The seeded corpus gives each tenant its own file: point IDs derive from
  file_id plus chunk_index alone, so two tenants in one file would collide on a
  single ID and the fixture would assert an impossible state.

Impact:
- 15 scenarios pass against both implementations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 13:09:50 +03:30
4da30f9983 feat(points): add the point read/edit port and its Qdrant adapter
Why:
- PointStorage is deliberately the two bulk operations ingestion performs. Reads,
  single-point edits, and keyword search have a different caller, a different
  failure vocabulary, and a different tenant-filter obligation, so they get their
  own port rather than accreting onto the ingestion one.

Changes:
- tenant_id is a required keyword argument on every port method, making a
  forgotten tenant filter a type error rather than a review question.
- Reads go through scroll with a HasIdCondition, not retrieve: retrieve takes no
  filter and would push the tenant check into Python after Qdrant already
  answered -- the shape ADR-0002's isolation rule exists to prevent.
- Ordered listing paginates by order_id value, not offset. Qdrant returns no page
  offset under order_by, and an offset cursor skips or repeats rows when a
  concurrent insert shifts positions underneath the reader.
- Point.from_payload takes a Mapping, not a dict: dict is invariant in its value
  type, so the SDK's concrete vector union is not a dict[str, object].
- Request schemas forbid extra keys and omit server-owned fields, so a client
  sending tenant_id or version gets 422 rather than having it silently ignored.

Impact:
- No route uses this yet; the /v1/points surface is Phase 2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 13:09:37 +03:30
5e935e5895 feat(qdrant): index content, is_active, and chunk_index on the chunks collection
Why:
- ADR-0002's keyword search needs a full-text index on content, which
  collection.py deliberately deferred to plan 002. is_active and chunk_index
  were unindexed while ingestion was the only reader; every /v1/points read path
  filters on them.

Changes:
- content gets a TEXT index with the multilingual tokenizer, which segments
  Persian correctly where the word tokenizer mishandles ZWNJ-joined compounds.
  No stemmer or stopword list: content is already letter-folded by
  normalize_persian_text at ingest, and the ranked Farsi lexical path is the
  benchmarked BM25 sparse vector, not this index.
- Tests assert content is TEXT rather than KEYWORD -- a keyword index would only
  match an entire chunk verbatim, which never happens and fails silently.
- Adds a test that a missing index is added to an already-live collection.

Impact:
- Requires re-running `python -m src.cli.qdrant_bootstrap`. Payload indexes are
  additive, so no collection rebuild and no re-embedding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 13:09:26 +03:30
ac3810182d docs(adr): record the re-ingestion rule and close plan 002's open decisions
Why:
- ADR-0002 already answered three of the four questions plan 002 listed as
  "decisions needed"; the fourth -- what happens to a manually edited point when
  its file is re-uploaded -- was left for a Phase 6 test to force. Deciding it in
  code rather than in the ADR would invert this repo's rule.

Changes:
- ADR-0002 gains "Re-ingestion versus manual edits": the new file wins,
  surviving points are overwritten with an incremented version, absent points
  are flagged inactive rather than removed, and manually created points sit past
  the ingested chunk_index range so the existing sweep covers them.
- Plan 002's stale decisions section becomes a pointer table; its audit scope is
  pinned to both ADR-0009 tables.

Impact:
- Clobbered edits are recoverable from point_audit_events, not from Qdrant: the
  deterministic point ID cannot hold both versions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 13:09:15 +03:30
Ali Zarinkolah
3d9269e54f docs(ops): add the operator runbook and record Phase 6 as complete
Why:
- the slice had no operator documentation: no tuning guidance, no failure
  procedure, no statement of the proxy timeout requirement.

Changes:
- add docs/runbook.md: startup, deployment steps, provisioning, the INGESTION_*
  tuning table with each bound's status code, the proxy read-timeout rule,
  /healthz vs /readyz, failure investigation by real event name plus job/event
  SQL, retry semantics, and alert thresholds
- link it from the README and note provisioning there
- mark plan 001 Phase 6 done and refresh CLAUDE.md's status paragraph

Impact:
- documentation only

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:58 +03:30
Ali Zarinkolah
7c1fe79f1c test(e2e): add the Compose smoke test of the running web process
Why:
- ADR-0016 reserves Compose for a serialized smoke test of the running web
  process. Nothing else exercises the deployment steps, a real HTTP server, or
  the real logging configuration, which every in-process test no-ops.

Changes:
- scripts/smoke.sh brings up Compose, runs both bootstrap steps, provisions a
  throwaway tenant, starts uvicorn, and drives the test against it
- the test skips unless SMOKE_BASE_URL is set, so `uv run pytest` never invokes
  Compose; it asserts the ADR-0011 JSON log sink

Impact:
- a pre-release gate, not a per-PR one

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:47 +03:30
Ali Zarinkolah
1b873e5a6f test(e2e): cover the ingestion slice against real Postgres, MinIO, and Qdrant
Why:
- the slice's reliability invariants (ADR-0016) had no end-to-end coverage.

Changes:
- 13 Testcontainers-based tests: duplicate upload, retry after a failed job,
  cross-tenant 404, unregistered domain, capacity 503, timeout 504, parse 400,
  real Qdrant 502, missing scope 403, and both readiness states
- only the dense embedders are faked (ADR-0016 bars live providers); they
  return the pinned 768/3072 dimensions
- the capacity test uses a committing sessionmaker, since the shared-connection
  fixture cannot serve concurrent sessions

Impact:
- runs in the default `uv run pytest`; needs Docker, like every integration test

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:36 +03:30
Ali Zarinkolah
133f565704 feat(tenant): add operator provisioning for tenants, API keys, and domains
Why:
- nothing over HTTP could create the first tenant: every /v1 route needs an API
  key, and a key cannot exist before its tenant. The service was unusable by a
  human without hand-written SQL.

Changes:
- add `provision_tenant`, owning tenant reuse-or-create, key generation and
  hashing, and domain registration in one transaction
- expose it as `python -m src.cli.provision_tenant`, alongside
  `alembic upgrade head` and `qdrant_bootstrap`
- add `tenants.get_by_slug`/`create` and `api_keys.create`
- log `tenant.provisioned` / `api_key.provisioned` with the key prefix only

Impact:
- a third deployment step; the plaintext key is printed once and never logged
  or stored (ADR-0011, ADR-0009)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:25 +03:30
Ali Zarinkolah
c7a5b69c0a fix(api): use the non-deprecated 422 status constant
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:12 +03:30
Ali Zarinkolah
2c72688440 refactor(test): share container fixtures across the integration and e2e suites
Why:
- tests/e2e/ cannot reach fixtures defined in a per-boundary conftest, and
  `pytest_plugins` is only honoured in the root conftest.

Changes:
- move the Postgres/MinIO/Qdrant container fixtures into
  tests/support/containers.py and register it as a root plugin
- fold MinIO bucket creation into `minio_settings`; an autouse fixture in a
  globally registered plugin would pull a container into unit runs
- add a `postgres_settings` fixture so a component can be built from it directly

Impact:
- no behavior change; `pytest -m unit` still needs no Docker

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:26:02 +03:30
Ali Zarinkolah
012b44d5f2 feat(observability): backfill logging for upload, auth, and domain services
Why:
- resolve_auth_context() runs on every authenticated request and logged
  nothing; four distinct rejection reasons (malformed/unknown/inactive/
  expired key, inactive tenant) were all invisible.
- The domain-allowlist rejection in upload_source_file() happens before any
  ingestion_jobs row exists, so it wasn't covered by the job-level
  ingestion.job.failed event either -- a rejected upload left zero trace.
- Four of upload_source_file()'s five failure branches (parse_failed,
  chunk_limit_exceeded, embedding_failed, index_failed) called
  _mark_job_failed(), which wrote to Postgres but never logged; only
  storage_failed and timeout had an ad-hoc logger.warning duplicated at their
  own call sites.

Changes:
- auth/service.py: auth.succeeded / auth.failed (with a reason field per
  rejection type), matching ADR-0011's own event catalog.
- domains/service.py: domain.rejected on the allowlist check;
  domain.created / domain.updated / domain.status_changed on the three
  mutations.
- files/upload.py: centralized failure logging inside _mark_job_failed
  (every failure branch already calls it, so logging there once closes all
  five branches instead of duplicating a log call at each site) as
  ingestion.job.failed; added ingestion.job.started; renamed the ad-hoc
  files.upload.succeeded to ingestion.job.completed for catalog consistency.

Impact:
- None to request/response behavior -- log events only.
2026-08-20 19:24:39 +03:30
Ali Zarinkolah
ac779dec7e fix(tests): isolate structlog global state from lifespan-triggering fixtures
Why:
- Any test using the client/api_client fixtures runs the app's real lifespan,
  which calls the production configure_logging() -- setting
  cache_logger_on_first_use=True (ADR-0011). That permanently monkeypatches
  the .bind method on whichever module-level
  logger = structlog.get_logger(__name__) instance is used first.
  structlog.reset_defaults() only resets *global* config, not that
  per-instance mutation, so once triggered, structlog.testing.capture_logs()
  silently stops intercepting events in every test that runs afterward in the
  same pytest process -- order-dependent flakiness with no useful failure
  message (assertions just see an empty list).

Changes:
- Added two autouse fixtures: one no-ops configure_logging for tests that
  spin up the app via LifespanManager (they test HTTP behavior, not logging
  output, so they don't need the real thing), one resets structlog defaults
  after every test as defense in depth.

Impact:
- Test-only; makes capture_logs()-based assertions reliable regardless of
  test execution order.
2026-08-20 19:21:56 +03:30
Ali Zarinkolah
9e8987968c feat(observability): add dual local logging sinks and static environment context
Why:
- Wanted human-readable console output while developing locally, without
  losing a machine-parseable log for later grepping/parsing. A single
  renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
  (build/region/instance) independent of any one request.

Changes:
- configure_logging() now builds two independent handlers: console (always
  on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
  (LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
  both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
  Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
  would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
  LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
  sinks locally"; "bind process-level environment context once at startup").

Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
  both call sites (lifespan, qdrant_bootstrap CLI) updated.
2026-08-20 19:20:27 +03:30
Ali Zarinkolah
e9e83b3a26 feat(tenant): add tenant_domains allowlist and /v1/domains management API
Why:
- Domain values are denormalized into every Qdrant point payload. Without
  validation, an unregistered or typo'd domain (e.g. "fier" for "fire")
  silently creates a new partition that retrieval never queries — the file
  ends up invisible rather than rejected. Tenants also need independently
  sized domain sets (one may run 14 insurance lines, another 6), which rules
  out an enum.

Changes:
- tenant_domains table (migration 41335d162de8) + repository, unique on
  (tenant_id, domain).
- src/application/domains/: ensure_domain_allowed() is the strict-allowlist
  check now run inside upload_source_file()'s first transaction, before any
  MinIO object, job row, or Qdrant point is written.
- /v1/domains (list/create/patch/disable/enable) gated on its own
  domains:read/domains:write scopes, deliberately separate from files:write
  so an upload key cannot create partitions. domain itself is immutable
  (denormalized into every point payload); only display_name is editable.
  Disable blocks new uploads without touching already-indexed points.

Impact:
- BREAKING: POST /v1/files now rejects any domain without an active
  tenant_domains row (400, unknown_domain). A domain must be created via
  POST /v1/domains before the first upload to it.
2026-08-20 18:20:24 +03:30
Ali Zarinkolah
fa933b08ff fix(qdrant): verify collection existence in the readiness check
Why:
- The chunks collection is now created by an explicit deployment step
  (qdrant_bootstrap), not at startup, which means a process can boot against
  a healthy Qdrant that has no collection at all. /readyz's previous check
  only called get_collections(), so it reported ready in that state — the
  misconfiguration stayed invisible until the first upload failed with a 502
  after already paying for the MinIO write and embedding round trips.

Changes:
- ping_qdrant() now checks collection_exists(collection) instead of just
  reachability.
2026-08-20 18:17:56 +03:30
Ali Zarinkolah
cc915f0f1a feat(ingestion): index embedded chunks into Qdrant on upload
Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
  — chunks were parsed and embedded but never written to Qdrant, so nothing
  was actually searchable after upload.

Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
  same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
  (error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
  chunks_indexed now reflect the real indexing result instead of a hardcoded
  zero.
- Wired PointStorage through AppResources/lifespan/the files router.

Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
2026-08-20 18:17:39 +03:30
Ali Zarinkolah
d00d436e5c feat(qdrant): add tenant-scoped point storage for ingestion
Why:
- Ingested chunks need to become searchable Qdrant points before the upload
  response returns, with tenant/domain isolation and a safe re-ingestion
  story per ADR-0001/0017.

Changes:
- src/application/points/: index_chunks() is the sole entry point, owning
  payload construction, batched/bounded-concurrency upserts
  (upsert_concurrency semaphore), and a soft-delete sweep for points a
  shorter re-ingestion leaves behind. The sweep runs only after every upsert
  in the attempt succeeds, so a failed attempt can leave a stale prefix but
  never removes content from a working index.
- PointStorage port (application/ports/) + QdrantPointStorage adapter
  (infrastructure/qdrant/points.py), keeping the qdrant_client SDK out of
  application code per ADR-0015.
- FakePointStorage test double for exercising the ordering/idempotency
  guarantees without a real Qdrant.
2026-08-20 18:16:35 +03:30
Ali Zarinkolah
58ca6109d1 feat(embedding): expose model_version on dense and sparse embedder ports
Why:
- ADR-0001's Qdrant payload records embedding_model_version so a future model
  swap can identify which chunks need re-embedding. The embedder is what
  knows which model produced its vectors, so it reports this rather than the
  call site reconstructing it from settings.

Changes:
- DenseEmbedder/SparseEmbedder protocols gain a model_version: str attribute.
- OpenAICompatibleEmbedder reports its configured model; Bm25SparseEmbedder
  reports its analyzer (bm25-<analyzer>).
2026-08-20 18:15:38 +03:30
Ali Zarinkolah
5e0addcc55 feat(qdrant): provision the chunks collection as an explicit deployment step
Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
  sparse, late_interaction) and payload indexes defined at creation time per
  ADR-0001; sparse/multivector fields cannot be added to an existing
  collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
  ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
  setup(), so it is a deployment step instead.

Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
  idempotent and schema-verifying (raises on dimension/modifier mismatch
  rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
  (python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.

Impact:
- Deployments must run the new bootstrap command before the first upload;
  see ADR-0001's new "Collection provisioning" section.
2026-08-20 18:15:21 +03:30
Ali Zarinkolah
e8fb41af87 test(ci): scope pytest timeout to test bodies, not fixture setup
Why:
- Testcontainers' session-scoped container startup (~25s on a cold Docker
  cache) was charged against the global 10s pytest-timeout budget, causing
  every integration test to fail regardless of its own runtime.

Changes:
- Set timeout_func_only = true so the budget applies to the test function
  only, not fixture setup.
2026-08-20 18:14:45 +03:30
99 changed files with 9241 additions and 276 deletions

View File

@@ -10,10 +10,18 @@
# Application
APP_ENV=local
APP_READINESS_CHECK_TIMEOUT_SECONDS=2.0
# Set by CI/CD at build/deploy time; never computed at runtime.
APP_SERVICE_VERSION=dev
# Logging
LOG_LEVEL=INFO
LOG_JSON_FORMAT=false
# Optional second sink, always JSON regardless of LOG_JSON_FORMAT. Local dev
# only -- leave unset in production, where stdout/stderr collection is
# preferred over an in-container log file.
# LOG_FILE_PATH=logs/app.log
LOG_FILE_MAX_BYTES=10485760
LOG_FILE_BACKUP_COUNT=5
# Postgres (application database, separate from Langfuse's Postgres)
# Use 127.0.0.1 rather than localhost: some environments resolve localhost to
@@ -44,6 +52,9 @@ INGESTION_EMBED_CONCURRENCY=4
# Qdrant
QDRANT_URL=http://127.0.0.1:6343
QDRANT_API_KEY=
QDRANT_COLLECTION=chunks
QDRANT_UPSERT_BATCH_SIZE=128
QDRANT_UPSERT_CONCURRENCY=4
# Dense embedders (ADR-0001). Both speak an OpenAI-compatible /embeddings
# endpoint, so one adapter serves both. Models and endpoints are the ones the

159
CLAUDE.md
View File

@@ -18,10 +18,116 @@ the chunk-count ceiling (`413`). The embedding configuration is **ported from
the `emet` evaluation lab** (`~/code/talie/emet`), which benchmarked these
models and analyzers on the real Farsi corpus — the analyzer and BM25 weights
are verified token-for-token against it, so treat them as a measured artifact
and re-benchmark rather than tune them in place (ADR-0005). Not built yet: Qdrant collection bootstrap,
Qdrant point upserts (so uploaded chunks are parsed/embedded but not yet
searchable), and `src/agent/`. That maps to plan 001 Phases 1-4 done, Phase 5
not started.
and re-benchmark rather than tune them in place (ADR-0005). Also working: the
`chunks` collection bootstrap (`src/infrastructure/qdrant/collection.py`, run as
a deployment step via `uv run python -m src.cli.qdrant_bootstrap` — never at
startup) and tenant-scoped point upserts (`src/application/points/` behind the
`PointStorage` port), so an upload is searchable by the time `201` returns.
Also working: `tenant_domains` plus `/v1/domains` (`src/application/domains/`),
a strict per-tenant allowlist — `POST /v1/files` rejects an unregistered or
disabled `domain` with `400` before anything is written, and domain management
sits behind its own `domains:read`/`domains:write` scopes, never `files:write`.
Also working: the operator runbook (`docs/runbook.md`), tenant/API-key/domain
provisioning (`uv run python -m src.cli.provision_tenant` — the third deployment
step, since nothing over HTTP can create the first tenant), a Testcontainers
e2e suite in the default pytest run (`tests/e2e/test_ingestion_slice.py`:
duplicate upload, retry after failure, tenant isolation, capacity, timeout,
parse and Qdrant failure), and the one Compose-based test — `scripts/smoke.sh`
driving `tests/e2e/test_compose_smoke.py` against a real uvicorn process, which
skips itself unless `SMOKE_BASE_URL` is set. That maps to plan 001 Phases 1-6
done.
Plan 002 (`/v1/points` CRUD and keyword search) is **Phases 1-3 done**. Phase 1
landed the `PointRepository` port (`src/application/ports/point_repository.py`)
with its `Point` read model (`src/application/points/point.py`), the Qdrant adapter
(`src/infrastructure/qdrant/point_repository.py`), request/response schemas
(`src/api/schemas/points.py`), and lifespan wiring. This port is **separate from
`PointStorage`**, which stays exactly the two bulk operations ingestion
performs — reads, single-point edits, and keyword search have a different caller
and a different tenant-filter obligation, so do not accrete them onto the
ingestion port. `tenant_id` is a required keyword argument on every
`PointRepository` method by design; keep it that way, because it is what turns a
forgotten tenant filter into a type error. The `chunks` collection also gained
full-text `content`, `is_active`, and `chunk_index` payload indexes, so a
deployed environment needs `qdrant_bootstrap` re-run (indexes are additive — no
rebuild, no re-embedding).
Two adapter mechanics there are load-bearing and easy to "simplify" into bugs:
reads go through `scroll` with a `HasIdCondition` rather than `retrieve` (which
takes no filter, and would move the tenant check to *after* Qdrant answered),
and ordered listing paginates by `order_id` value rather than offset (Qdrant
returns no page offset under `order_by`, and an offset cursor skips or repeats
rows under a concurrent insert).
Phase 2 added the **read routes**: `GET /v1/points/{point_id}`,
`GET /v1/points?file_id=...`, `GET /v1/points/count`, `GET /v1/points/search`,
and `GET /v1/files/{file_id}/points`, over `src/application/points/queries.py`
(`src/api/routers/points.py`). All are gated on `points:read`, which — with
`points:write` — is now in `DEFAULT_SCOPES`; `GET /v1/files/{file_id}/points`
uses `points:read` rather than `files:write`, so the scope follows the data
rather than the URL prefix. `PointNotFoundError` maps to `404` in
`src/api/errors.py`, never `403`. Three route-level rules are load-bearing:
`/count` and `/search` are declared **before** `/{point_id}` (FastAPI matches in
declaration order, so reordering them makes `/v1/points/count` a `422`),
`file_id` is **required** on the listing (the cursor is an `order_id` value and
`order_id` is unique only within one file), and `search_points` folds the query
with `normalize_persian_text` before matching, because ingestion letter-folds
content and an unfolded Arabic-keyboard query would return an empty result set
silently rather than erroring (ADR-0002).
Phase 3 added **soft delete**: `DELETE /v1/points/{point_id}` and
`DELETE /v1/files/{file_id}`, over `src/application/points/deletion.py` (with
the pure relinking primitive in `src/application/points/relinking.py`) and
`src/application/files/deletion.py`. Both are gated on `points:write` — the
file route included, since the data it destroys is points. Nothing is ever
removed from Qdrant.
Four rules there are load-bearing, and three of them look like complications
until the concurrency is taken seriously:
- `patches_for_removal` computes **what is still missing between the state just
read and the desired end state**, not "the patches a delete implies". That is
what makes a normal delete, a second delete of an already-inactive point (a
no-op success, never `404`), and recovery from a half-applied batch one code
path. Rewriting it as a straight-line "deactivate, patch prev, patch next"
breaks all three.
- Qdrant has no multi-point transaction and reports success for a filtered
`set_payload` that matched nothing, so a batch whose second operation loses a
version race applies its first anyway. `soft_delete_point` therefore re-plans
and re-applies up to three times, verifying by read-back, and only then raises
`PointVersionConflictError` (`409`). A single-shot delete would be able to
leave a stale pointer, which ADR-0002 calls a defect.
- A soft-deleted point **keeps its own** `previous_chunk_id`/`next_chunk_id`;
only the surviving neighbours are rewritten. Those pointers are unreachable
rather than stale, they are the only record of where the point sat, and the
retry re-plans from them. The whole-file sweep follows from the same rule:
every point leaves at once, so no survivor can dangle and no pointer is
touched at all.
- `DELETE /v1/files/{file_id}` marks the `source_files` row `soft_deleted`
**after** the point sweep, in its own short transaction (no session is held
across the Qdrant work). Order matters: a half-finished sweep leaves the row
`active` and a retried `DELETE` finishes it, and retiring the row is what
makes a later re-upload of the same bytes re-ingest instead of matching
`find_active_by_content_hash` and returning a file whose points are gone.
Audit rows are still Phase 4/6 work; Phase 3 emits log events only
(`points.soft_deleted`, `files.soft_deleted`, `points.relink.neighbour_missing`,
and the two `*.conflict` warnings). The completion and conflict events carry
ADR-0011's `duration_ms` plus `rounds`, and the pair is what makes them
diagnostic: relinking itself is O(1) (that is what the adjacency pointers buy),
so a single-point delete costs a fixed ~5 Qdrant round trips and a `rounds`
above 1 means contention, not a slow store. The whole-file sweep is the one
whose cost scales — two round trips per 100-point page.
Also worth knowing before touching the points tests: `tests/support/point_contract.py`
holds **one** scenario suite run against both `FakePointRepository` (unit) and
`QdrantPointRepository` (integration), so new repository behaviour belongs there
rather than in one of the two runners — that is what keeps the fake from drifting
more permissive than the real store.
Not built yet: plan 002 Phases 4-6 — create/replace/patch, reorder and batch,
the `api_request_logs`/`point_audit_events` tables, and the runbook section on
inspecting and repairing a file's pointer chain — and `src/agent/`.
Architecture decisions live in `docs/adr/` (18 ADRs plus the 0000 template;
0001–0004 are `Accepted` — 0004 amended by 0018; 0014 is `Superseded by 0017`;
@@ -140,9 +246,13 @@ as its only caller-facing entry point. It dispatches on source type and owns
the `anyio.to_thread.run_sync` + `CapacityLimiter` offload ADR-0017 requires;
`parse_docx`/`parse_csv`/`parse_xlsx`/`chunk_document` stay in the package,
exported mainly for their own tests, not for outside callers to reach for
directly. Follow this pattern in `application/` as new packages are added
there — `points/`, `retrieval/`, `threads/` — rather than exposing their
internals as the primary surface.
directly. `src/application/points/` follows the same shape: `index_chunks` is
the only caller-facing entry point, owning payload construction, batching,
the `upsert_concurrency` semaphore, and the ordering rule that the soft-delete
sweep runs only after every upsert succeeds; `build_chunk_payload` stays
internal. Follow this pattern in `application/` as new packages are added
there — `retrieval/`, `threads/` — rather than exposing their internals as the
primary surface.
### Resource lifetime rules (ADR-0012)
@@ -221,6 +331,11 @@ content_sha256)` idempotency, no terminal job returning to `running`.
### Postgres conventions (ADR-0009)
`domain` is never free-form: it must match an `active` `tenant_domains` row for
the authenticated tenant (ADR-0009). Domain sets are per-tenant and vary in
size. The key itself is immutable — it is denormalized into every Qdrant point
payload and into `source_files`, so renaming it is a migration, not an edit.
UUID primary keys (app-generated), `timestamptz` for all timestamps,
`Numeric(18, 8)` for money (never floats), `JSONB` for flexible metadata but
typed/indexed columns for query-critical fields, string status columns with
@@ -237,7 +352,24 @@ Postgres remains system of record for tenants, API keys, audit, jobs,
`graph_runs`, `llm_calls`/`llm_pricing`. Correlate the two via `request_id`,
`tenant_id`, `thread_id`, `run_id`. Use `structlog` with stable event names
and structured fields (`logger.info("graph.run.completed", ...)`), not
interpolated prose; JSON logs by default in production.
interpolated prose; JSON logs by default in production, plus an optional
local-only JSON file sink independent of the console renderer (`LOG_FILE_PATH`).
**Add logging in the same change that adds the code, not as a follow-up.**
When you add a new service-level entry point (an `application/` function a
route calls directly, an ingestion phase, a mutation) or a new failure branch
inside one, add its `logger.*` event in that same diff, using ADR-0011's
level/event-naming table. Deferring it means re-deriving the failure modes and
field names later from code that no longer has them in working memory — as
happened with `src/application/files/upload.py`, where four failure branches
(`parse_failed`, `chunk_limit_exceeded`, `embedding_failed`, `index_failed`)
shipped with no log event and had to be retrofitted.
This does not mean logging every function. Pure functions, models, schemas,
and repositories (`infrastructure/postgres/repositories/`) stay silent by
convention — the caller that turns their result into a business-meaningful
outcome (job succeeded, upload rejected, domain disabled) is where the event
belongs, not the row-level function underneath it.
## Testing (ADR-0016)
@@ -251,9 +383,14 @@ interpolated prose; JSON logs by default in production.
- Test naming: `test_<unit>_<scenario>_<outcome>`, Arrange–Act–Assert.
- Layout mirrors architecture: `tests/unit/{application,agent}`,
`tests/integration/{postgres,minio,qdrant}`, `tests/e2e/`.
- Integration tests use **Testcontainers** (never a developer's local
services or Langfuse-owned storage/credentials) — this is the standard
automated mechanism, not Docker Compose. Isolate data per test via unique
- Integration **and e2e** tests use **Testcontainers** (never a developer's
local services or Langfuse-owned storage/credentials) — this is the standard
automated mechanism, not Docker Compose. Compose is reserved for exactly one
thing: the serialized operational smoke test of the *running web process*
(`scripts/smoke.sh`), which is gated out of `uv run pytest`. Shared container
fixtures live in `tests/support/containers.py`, registered from the root
`tests/conftest.py` via `pytest_plugins` (a non-root conftest cannot declare
it). Isolate data per test via unique
keys/queue/collection names; parallel integration execution is disabled
until fixture isolation is proven safe.
- Pytest never calls a live/paid model provider in routine runs — that's

View File

@@ -2,6 +2,49 @@
Architecture decisions live in [`docs/adr`](docs/adr). The first implementation
milestone is documented in the [ingestion vertical-slice plan](docs/plans/001-ingestion-vertical-slice.md).
Day-to-day operation — tuning the ingestion bounds, the proxy timeout
requirement, and how to investigate or retry a failed upload — is the
[operator runbook](docs/runbook.md).
## Provisioning the datastores
Both schema steps run as explicit deployment steps. The application performs no
DDL at startup — not for Postgres (ADR-0009) and not for Qdrant (ADR-0001,
"Collection provisioning").
```bash
docker compose up -d # Postgres, MinIO, Qdrant
uv run alembic upgrade head # Postgres schema
uv run python -m src.cli.qdrant_bootstrap # the `chunks` collection
uv run fastapi dev src/main.py
```
Nothing over HTTP can create the first tenant — every `/v1` route needs an API
key, and a key cannot exist before its tenant. One command issues both, plus any
domains, printing the key once (only its hash is stored):
```bash
uv run python -m src.cli.provision_tenant --slug acme --domain fire
```
Before a tenant can upload, its domains must be registered — `POST /v1/files`
rejects an unregistered or disabled `domain` with `400`. The calling backend
manages them over `/v1/domains` using a key with the `domains:write` scope:
```bash
curl -X POST http://localhost:8000/v1/domains \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{"domain": "fire", "display_name": "Fire insurance"}'
```
Both bootstrap commands are idempotent and safe to re-run. `qdrant_bootstrap` verifies an
existing collection against the pinned schema and exits non-zero on a mismatch,
rather than leaving a silently degraded sparse index in place.
`./scripts/smoke.sh` verifies the whole path — Compose up, both deployment
steps, provisioning, an upload through the running web process to indexed Qdrant
points. See the [runbook](docs/runbook.md#12-verifying-a-deployment).
## Local Langfuse

View File

@@ -0,0 +1,48 @@
"""create tenant_domains
Revision ID: 41335d162de8
Revises: bfc6c81c2542
Create Date: 2026-08-20 17:48:29.443293
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision: str = '41335d162de8'
down_revision: Union[str, Sequence[str], None] = 'bfc6c81c2542'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('tenant_domains',
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('tenant_id', sa.Uuid(), nullable=False),
sa.Column('domain', sa.String(length=80), nullable=False),
sa.Column('display_name', sa.String(length=200), nullable=False),
sa.Column('status', sa.String(length=20), server_default='active', nullable=False),
sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), server_default='{}', nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('disabled_at', sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint("status IN ('active', 'disabled')", name='ck_tenant_domains_status'),
sa.ForeignKeyConstraint(['tenant_id'], ['tenants.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('tenant_id', 'domain', name='uq_tenant_domains_tenant_id_domain')
)
op.create_index(op.f('ix_tenant_domains_tenant_id'), 'tenant_domains', ['tenant_id'], unique=False)
# ### end Alembic commands ###
def downgrade() -> None:
"""Downgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_tenant_domains_tenant_id'), table_name='tenant_domains')
op.drop_table('tenant_domains')
# ### end Alembic commands ###

View File

@@ -108,6 +108,38 @@ measurement rather than assumption:
- Payload index on `previous_chunk_id` / `next_chunk_id`: keyword index,
used for O(1) adjacency retrieval (see below).
### Collection provisioning
The collection is created by an explicit **deployment step**, not by application
startup and not lazily on first write:
uv run python -m src.cli.qdrant_bootstrap
Creating a collection is DDL, and this project already keeps DDL out of the boot
and request paths: [0009](0009-postgres-sqlalchemy-alembic-schema.md) requires
Alembic for Postgres schema and forbids `create_all()` at startup, and
[0012](0012-application-resource-lifetime-and-dependency-ownership.md) makes
LangGraph's `.setup()` a deployment step for the same reason. Neither ADR named
Qdrant explicitly; this section closes that gap rather than letting the placement
be decided by whichever code happened to need it first.
Doing it in the FastAPI lifespan was rejected: it couples process boot to Qdrant
being reachable (which is `/readyz`'s job, not boot's), races across replicas,
and turns a misconfigured collection into a silent skip. Doing it lazily on first
upsert was rejected for putting DDL on a user request and hiding the
misconfiguration until traffic arrives.
`ensure_chunks_collection` is idempotent and **verifying**: against an existing
collection it compares the dense dimensions and the sparse `modifier` to the
pinned values and fails loudly on divergence. That check is the point of making
the step explicit — both properties degrade silently in production if wrong (a
missing `modifier="idf"` produces no error, just unweighted lexical retrieval).
Payload indexes are (re)created on every run, since unlike vector configuration
they can be added to a live collection. The full-text index on `content` is
therefore deferred to the keyword-search work in
[0002](0002-chunk-crud-and-search-api.md), not created here.
### Payload schema
This schema is now decided for the fields below. Additional document-context
@@ -124,7 +156,7 @@ involves format-specific tradeoffs not yet made.
| `chunk_id` | keyword | stable identifier for a single chunk |
| `content_type` | keyword | classification of the chunk's content; exact value set (e.g. `paragraph`, `table_row`, `heading`) to be finalized alongside the chunking-strategy ADR |
| `source_filename` | keyword | original uploaded filename |
| `source_type` | keyword (`docx` \| `csv`) | which parser produced this chunk |
| `source_type` | keyword (`docx` \| `xlsx` \| `csv`) | which parser produced this chunk — `xlsx` added by [0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md) |
| `order_id` | float (see below) | chunk's *display* position within the file; mutable so the backend can reorder/insert chunks |
| `chunk_index` | integer | chunk's *original ingestion* ordinal — immutable, used to derive the deterministic point ID below (kept separate from `order_id` precisely because `order_id` can change) |
| `previous_chunk_id` | keyword, nullable | `chunk_id` of the preceding chunk in display order (`null` for the first chunk in a file) — O(1) adjacency pointer for context-window expansion in ADR-0003 |
@@ -136,7 +168,7 @@ involves format-specific tradeoffs not yet made.
| `updated_at` | datetime | last modification timestamp |
| `created_by` | keyword | user/service that created the chunk |
| `updated_by` | keyword | user/service that last modified the chunk |
| `version` | integer | optimistic-concurrency counter, used in ADR-0002 |
| `version` | integer | optimistic-concurrency counter, used in ADR-0002. Ingestion currently writes `1` unconditionally: the read-check-write that makes the guard meaningful costs one read per point and belongs with the `/v1/points` write paths, so plan 002 owns it. Safe while ingestion is the only writer of a file's points; it would clobber a concurrent manual edit's counter once `/v1/points` ships. |
| `content_hash` | keyword | hash of the chunk's raw text; lets re-ingestion detect unchanged content and skip re-embedding it |
| `embedding_model_version` | keyword | identifies which embedding model(s) produced this chunk's vectors; needed to know which chunks require re-embedding after a future model swap |

View File

@@ -75,6 +75,51 @@ retrieval used by the AI agent in ADR-0003; the two "search" concepts serve
different callers (a human/admin managing chunks vs. an agent retrieving
context) and should not be conflated in the API or in future discussion.
Two properties follow from the index being a *filter*: results carry no
relevance score, and their order is unspecified. The API therefore returns
neither a score field nor a ranked list, and callers must not read the array
order as relevance. A caller that wants ranking wants ADR-0003's path.
#### The query is normalized the way ingested content was
`normalize_persian_text` (ADR-0018) folds Arabic letterforms to their Persian
equivalents — U+064A to U+06CC, U+0643 to U+06A9 — on every text block before
chunking, so stored `content` is uniformly Persian-formed. A query string is
not chunk content and never passes through that path, so a term typed on an
Arabic keyboard reaches the index as a different codepoint sequence than the
document it should match.
The service therefore applies the same folding to the query before matching.
Without it the endpoint fails in the worst available way: an exact-looking
query returns an empty result set, with no error, no warning, and nothing in
the logs to distinguish "no such term" from "the term is spelled with the
other yeh". Note this is a *query-side* transformation only — it changes what
is compared, never what is stored.
This does not extend to stemming or synonyms. Qdrant's full-text index offers
neither, and adding a Farsi analyzer here would duplicate the benchmarked BM25
sparse pipeline (ADR-0005) in a code path that is not benchmarked against
anything.
#### Listing is scoped to one file, and paginates by `order_id`
`GET /points?file_id=...` requires `file_id` rather than treating it as one
optional filter among several, and its pagination cursor is an `order_id`
value rather than an offset. Both follow from `order_id` being per-file:
- A cursor is only meaningful against a totally ordered key. `order_id` orders
points within one file and says nothing across files, so an unscoped listing
has no stable sort to paginate along.
- An offset cursor is wrong even within one file. Insert, reorder, and delete
all shift positions, so a page-two request issued after a concurrent insert
ahead of the cursor would repeat a row already returned — silently. Ranging
on `order_id > cursor` is unaffected: the reader has passed that value, and a
point inserted behind it was already served.
The second point depends on `order_id` being unique within a file, which the
gap-exhaustion rule below preserves by rejecting a reorder whose computed gap
would collapse onto a neighbour value.
### Delete is soft by default
`DELETE /points/{point_id}` and `DELETE /points?file_id=...` set
@@ -105,6 +150,41 @@ Qdrant's `update_filter`, giving an optimistic-concurrency-style guard
against races between a concurrent ingestion re-run (ADR-0001) and a manual
edit through this API.
### Re-ingestion versus manual edits
A file can be re-uploaded after someone has hand-edited one of its points
through this API. **The newly ingested file wins.** Ingestion is authoritative
for the content of the file it ingested; a manual edit is a correction that
survives only until the source document is replaced.
Concretely:
- A point that still exists in the new version (same `file_id` +
`chunk_index`, hence the same deterministic point ID) is **overwritten in
place**. Ingestion performs a read-check-write so `version` is incremented
from whatever the manual edit left it at, rather than reset to `1`.
- A point from the previous ingestion that is **absent** from the new version
is flagged `is_active: false` with `deleted_at` set. It is never removed
from Qdrant — the soft-delete rule above applies to re-ingestion exactly as
it applies to `DELETE`.
- A manually created point (`POST /points`) is assigned a `chunk_index` past
the ingested range, so the same sweep deactivates it on the next upload of
its file. This is the intended consequence of "the new file wins", not an
accident of the sweep's bounds.
Because the point ID is derived from the immutable `chunk_index`, an
overwritten point cannot hold both the manual edit and the new file's content.
The clobbered content is therefore recorded in `point_audit_events`
(ADR-0009) as a `reingest_overwrite` operation carrying `before_version`, so
the edit is recoverable from the audit trail even though it is no longer a
live point.
Rejected alternative: preserving manual edits by having ingestion skip points
with `version > 1`. It breaks the guarantee that a successful upload leaves
Qdrant matching the uploaded document, and it needs a second, separate rule
for edited points that no longer exist in the new version — two divergent
notions of authority over one file.
### Re-embedding on content edit
`PUT /points/{point_id}` can change `content`, which leaves the stored

View File

@@ -98,13 +98,16 @@ One row per customer/tenant.
| `slug` | Stable short name, unique, human-readable. |
| `name` | Display name. |
| `status` | `active` \| `suspended` \| `deleted`. Suspended tenants authenticate to a clear error but cannot run work. |
| `settings` | JSONB for tenant-level feature flags/limits (max upload size, enabled file types, allowed domains, etc.). |
| `settings` | JSONB for tenant-level feature flags/limits (max upload size, enabled file types, etc.). Allowed domains were previously listed here as well; they live in `tenant_domains` instead, per this ADR's own rule that query-critical fields get typed columns — `domain` is validated on every upload and filtered on every query. |
| `created_at`, `updated_at`, `deleted_at` | Audit/soft-delete timestamps. |
#### `tenant_domains`
Optional but recommended. Validates the `domain` values used throughout Qdrant
payloads (`car`, `fire`, etc.) per tenant.
**Required.** (Previously "optional but recommended"; implemented and made
mandatory alongside `/v1/domains`.) Validates the `domain` values used
throughout Qdrant payloads (`car`, `fire`, etc.) per tenant. Domain sets are
per-tenant and differ in size — one tenant may run 14 insurance lines and
another 6 — so this is data, not an enum.
| Column | Notes |
|---|---|
@@ -116,7 +119,37 @@ payloads (`car`, `fire`, etc.) per tenant.
| `metadata` | JSONB for domain-specific ingestion/retrieval settings. |
This prevents arbitrary caller-supplied domains from silently creating new
partitions in Qdrant.
partitions in Qdrant. The failure it guards against is quiet: a typo such as
`fier` for `fire` produces no error anywhere — the file is stored, parsed,
embedded, and indexed into a partition retrieval never queries, so it is
invisible rather than failed.
##### Enforcement and management
- **Strict allowlist.** `POST /v1/files` rejects a domain with no `active` row
for the tenant (`400`, error code `unknown_domain`). There is no auto-create
on first use: that would record the typo rather than prevent it. The check
runs inside the upload's first transaction, before any MinIO object, job row,
or Qdrant point is written.
- **Managed over the API, not by an operator.** `/v1/domains` (list, create,
update, disable, enable) is the surface the calling backend uses. Domains are
created by an explicit, scoped call rather than as a side effect of an upload
— that distinction, not who makes the call, is what "strict" means here.
- **Its own scope.** `domains:read`/`domains:write`, deliberately separate from
`files:write`. Folding domain creation into the upload scope would let an
upload key create partitions again, which is the exact hole this closes.
`api_keys.scopes` is already a free JSONB list, so this needs no schema change.
- **`tenant_id` stays derived from the API key.** One key per tenant; nothing
request-suppliable. A platform key acting across tenants would need a real
actor model and is not adopted.
- **`domain` is immutable; `display_name` is not.** The key is denormalized into
every Qdrant point payload and into `source_files`, so renaming it means
rewriting all of them — a migration, not a `PATCH`. The update schema
therefore has no `domain` field.
- **Disable is not delete.** `status='disabled'` blocks new uploads and hides
the domain from listings, leaving already-indexed points intact and
retrievable. Actual removal needs the retention/erasure workflow this ADR and
plan 001 defer.
#### `api_keys`

View File

@@ -70,17 +70,30 @@ logger.info(
Do not build log messages by interpolating operational metadata into prose.
Prefer fields over long strings because fields are queryable.
### Emit JSON logs by default in production
### Emit JSON logs by default in production; console and file are independent sinks locally
Production logs are JSON on stdout so process managers, container runtimes, and
log collectors can ingest them directly. Local development may use a colored
console renderer controlled by configuration.
log collectors can ingest them directly. This does not change.
File logging is optional and mainly for local development. If enabled, it must
use explicit rotation settings such as `maxBytes` and `backupCount`. Do not rely
on a default `RotatingFileHandler` with no rotation parameters. In containerized
production, stdout/stderr collection is preferred over writing `logs/app.log`
inside the application container.
Locally, stdout and an optional file are two **independent, simultaneous**
handlers on the same logger, not a single renderer chosen by a flag — the same
structlog event fans out to both:
- **Console handler**: always on, `structlog.dev.ConsoleRenderer(colors=True)`.
This is what a developer reads while the process runs, so it stays
human-readable regardless of whether file logging is also enabled.
- **File handler**: off by default, enabled by setting `LOG_FILE_PATH`. Always
renders JSON (`structlog.processors.JSONRenderer()`), independent of the
console handler's renderer, so a saved log is machine-parseable even though
the terminal output next to it is not. Must use explicit rotation
(`RotatingFileHandler` with `maxBytes`/`backupCount` — never an unrotated
handler).
In containerized production, stdout/stderr collection remains preferred over
writing `logs/app.log` inside the application container, so `LOG_FILE_PATH` is
expected to be unset there; the file handler exists for local development,
where reading a colored terminal *and* keeping a JSON trail to grep/parse later
are both useful at once.
### Configure stdlib and structlog together
@@ -188,6 +201,36 @@ Notes:
- `structlog.contextvars.merge_contextvars` ensures request-bound fields appear
on both structlog and stdlib logs processed through the formatter.
### Bind process-level environment context once at startup
Deployment identity — which build is running, in which environment, on which
instance — answers a different question than request correlation: "is this
issue specific to one deployment / one region / one instance?" rather than "is
this issue specific to one request?" It does not vary per request, so it must
not go through `structlog.contextvars`, which `RequestIdMiddleware` clears on
every request; a value bound there before the first request would be wiped the
moment that middleware runs.
Instead, add a static structlog **processor** — a plain closure over values read
once at `configure_logging()` time — so it runs on every event regardless of
request context:
```python
def _bind_environment(settings: AppLimitSettings):
def processor(logger, method_name, event_dict):
event_dict["env"] = settings.env
event_dict["service_version"] = settings.service_version
return event_dict
return processor
```
`service_version` should be the deployed commit SHA or release tag (e.g. from a
`GIT_SHA`/`APP_VERSION` build-time env var — not computed at runtime by
shelling out to `git`). This makes "is this only happening on the new
deployment?" answerable directly from logs, without cross-referencing a
separate deployment record.
### Bind request context with contextvars
At FastAPI ingress, clear stale context, bind request identifiers, and return the

View File

@@ -159,8 +159,25 @@ retry, and phase 2 has no transaction protecting it:
return the existing file/job rather than re-ingesting (plan 001).
- `tenant_id` comes from `AuthContext`, never from the request body.
- A terminal job is never transitioned back to `running`.
- Qdrant points from a failed attempt do not replace the previous successful
index; replacement happens only after a successful attempt.
- A failed attempt never *removes* content from a working index. The
soft-delete sweep that retires a shortened file's leftover points runs only
after every upsert in the attempt has succeeded.
This is deliberately weaker than "replacement happens only after a successful
attempt", which an earlier revision of this ADR claimed. That guarantee is not
achievable alongside ADR-0001's deterministic point ids: those ids are exactly
what makes a retry idempotent, and they also mean a re-ingestion overwrites
points **in place**, so a crash partway through leaves a prefix updated and the
remainder still on the old content. Buying literal atomicity would mean
generation-suffixed ids and an activation flip, which contradicts ADR-0001 and
ADR-0002's stable point ids. Staging the new points as `is_active=false` and
flipping them on success is strictly worse — the in-place overwrite would
deactivate the previously live points, silently emptying a working index if the
attempt were interrupted.
What holds instead: the index is never emptied, never partially deleted, and a
retry converges — deterministic ids rewrite every point and the sweep re-runs,
reaching the exact correct state.
### Failures are HTTP failures

View File

@@ -125,9 +125,12 @@ them:
- Use `(tenant_id, domain, content_sha256)` to recognize identical uploads.
- An identical active upload should return the existing source-file/job reference
rather than create a duplicate ingestion.
- A changed upload creates a new ingestion job. Existing active Qdrant points are
replaced only after the new job completes successfully, so a failed re-ingestion
does not remove a working index.
- A changed upload creates a new ingestion job. A failed re-ingestion never
removes a working index: the soft-delete sweep for a shortened file runs only
after every upsert has succeeded. Because ADR-0001's point ids are
deterministic, upserts overwrite in place, so an interrupted attempt can leave
a prefix updated — it cannot empty or partially delete the index, and a retry
converges. See ADR-0017, "Re-running an ingestion stays safe".
- Preserve the original filename in Postgres metadata. MinIO object keys remain
internal ID-based paths.
@@ -272,6 +275,11 @@ code and a terminal job row.
tenant-filtered upserts, terminal state persistence, retrying an upload, and
parser/Qdrant failure handling.
The `chunks` collection itself is provisioned by a deployment step —
`uv run python -m src.cli.qdrant_bootstrap` — not by FastAPI startup, for the
same reason ADR-0009 keeps Alembic out of startup and ADR-0012 makes LangGraph's
`.setup()` a deployment step. See ADR-0001, "Collection provisioning".
**Exit criteria:** a successful upload returns `201` with a terminal status, and
its points are retrievable only under the owning tenant's Qdrant filter. A forced
failure mid-ingestion produces a `failed` job and the right HTTP status, and
@@ -281,17 +289,28 @@ retrying the upload produces a correct final state without duplicate chunks.
1. Add an operator runbook covering local startup, migrations, MinIO bucket
setup, the run command, ingestion-bound tuning, the proxy/client timeout
requirement, and how to retry a failed ingestion.
requirement, and how to retry a failed ingestion. — `docs/runbook.md`.
2. Add a serialized Compose-based operational smoke test covering upload through
indexed points against the running web process. Testcontainers remains the
standard pytest mechanism for individual adapter integration tests.
standard pytest mechanism for individual adapter integration tests. —
`scripts/smoke.sh` driving `tests/e2e/test_compose_smoke.py`, which skips
itself unless `SMOKE_BASE_URL` is set so `uv run pytest` never invokes
Compose.
3. Add end-to-end tests for duplicate upload, retrying a failed upload, tenant
isolation, capacity/timeout rejection, and failed parser/Qdrant behavior.
isolation, capacity/timeout rejection, and failed parser/Qdrant behavior. —
`tests/e2e/test_ingestion_slice.py`, on Testcontainers, in the default suite.
4. Add health/readiness checks that distinguish process health from dependency
readiness.
readiness. — `/healthz` and `/readyz`; `/readyz` additionally requires the
`chunks` collection to exist, since a reachable but unbootstrapped Qdrant
would `502` on the first upload.
5. Update the README with local-start instructions and links to ADRs, this plan,
and the operations runbook.
Provisioning a tenant and its first API key turned out to be a prerequisite for
1 and 2 rather than a separate milestone: nothing over HTTP can create the first
tenant, so `src/cli/provision_tenant.py` was added alongside the other two
deployment-step commands.
**Exit criteria:** a new developer can start the stack, apply migrations, upload a
a document, observe the job through completion, and understand how to investigate or
retry a failure.

View File

@@ -15,8 +15,9 @@ are; this document defines order, scope, and verification criteria.
## Prerequisite
Plan 001 must be complete through **Phase 5** before Phase 3 of this plan
starts. Specifically this plan depends on: the `chunks` collection and its
Plan 001 is complete through Phase 6, so this prerequisite is satisfied. It
required plan 001 through **Phase 5** before Phase 3 of this plan starts.
Specifically this plan depends on: the `chunks` collection and its
payload indexes actually existing, API-key authentication and `AuthContext`
tenant derivation, the application-lifetime Qdrant client from the FastAPI
lifespan, and the request-lifetime `AsyncSession` wiring. Phases 1–2 below
@@ -71,7 +72,9 @@ project owner accepts them, and update the ADR rather than diverging silently.
(bulk soft delete of a file's points), from ADR-0008.
- Soft delete as the default for every delete path, with neighbor relinking.
- Optimistic concurrency on every mutating path via the `version` payload field.
- Audit rows in Postgres for mutating operations.
- Audit rows in Postgres for mutating operations: both ADR-0009 tables,
`api_request_logs` (one row per API call, written from the request middleware)
and `point_audit_events` with the real `api_request_log_id` foreign key.
- Automated tests for tenant isolation, pointer integrity, concurrency
conflicts, and pagination.
@@ -113,40 +116,22 @@ project owner accepts them, and update the ADR rather than diverging silently.
9. Routers contain no Qdrant SDK calls and no filter construction. The Qdrant
client is injected from the lifespan (ADR-0012).
## Decisions needed before the affected phase
## Decisions resolved before implementation
### Re-embedding on content edit (blocks Phase 4)
An earlier revision of this plan listed three open decisions here. All are now
settled, and one further question this plan deferred to a Phase 6 test has been
settled too. They are recorded in the ADRs — these lines are a pointer, not a
second source of truth.
`PUT /v1/points/{point_id}` can change `content`. The stored vectors then no
longer match the text. Three options, in order of preference:
| Question | Resolution | Recorded in |
|---|---|---|
| Re-embedding on content edit | Re-embed inline, reusing ingestion's ports and bounds and its `502`/`504` codes. The re-embed happens *before* the version-guarded write, so a stale edit still `409`s rather than re-embedding for nothing. | ADR-0002, "Re-embedding on content edit" |
| Fractional-key exhaustion | No renormalize endpoint in this slice. Log `points.order_id.gap_low` under a safety threshold; reject with `409` and a distinct error code if the gap would collapse onto a neighbor value. Recovery is a runbook operation. | ADR-0002, "`order_id` gap exhaustion" |
| Batch semantics | All-or-nothing, capped at 100 operations. Every operation's `version` precondition is validated before any is applied; one failure rejects the whole request and nothing reaches Qdrant. | ADR-0002, "`POST /points/batch` semantics" |
| Re-ingestion versus manual edits | The newly uploaded file wins. Surviving points are overwritten in place with an incremented `version`; points absent from the new version are flagged inactive, never removed; manually created points sit past the ingested `chunk_index` range and are swept by the same rule. Clobbered content is recorded in `point_audit_events` as `reingest_overwrite`. | ADR-0002, "Re-ingestion versus manual edits" |
1. **Re-embed inline** on content change, reusing plan 001's embedding ports and
bounds. Consistent, but puts embedder latency and `502`/`504` failure modes
on an admin edit path.
2. **Require caller-supplied vectors** when content changes, and reject the edit
otherwise. Simple and honest, but pushes model knowledge to the client.
3. **Mark the point stale** (a payload flag) and re-embed later. Needs
background work, which ADR-0017 currently rules out.
Default to (1) for parity with ingestion, with the same batch/semaphore bounds
and the same status codes. Record whichever is chosen in ADR-0002 before
implementing Phase 4 — this is a real behavioral contract, not an
implementation detail.
### Fractional-key exhaustion
ADR-0001 notes float keys eventually need renormalization. Decide now whether
this slice ships a renormalize path (an internal operation rewriting a file's
`order_id` values to `1000, 2000, 3000, ...`) or explicitly defers it with a
logged warning when the gap between neighbors falls under a threshold. Deferring
is acceptable; silently producing unrepresentable gaps is not.
### Batch semantics
`POST /v1/points/batch` must define, in the API schema and the tests: whether
operations are all-or-nothing, what happens when operation 3 of 5 fails a
version check, and the maximum operation count per request. Decide before
Phase 5; do not let the answer be "whatever Qdrant happened to do."
Phase 6's cross-slice end-to-end test therefore *verifies* the re-ingestion rule
rather than forcing the decision.
## Build order
@@ -251,8 +236,8 @@ of mutations.
ordering behavior, isolated per test by unique collection or tenant keys.
2. An end-to-end test crossing plan 001 and this slice: ingest a CSV, list its
points, reorder one, soft-delete another, re-upload the same file, and assert
the manual edits interact with re-ingestion exactly as ADR-0001/0002 specify.
If that interaction is not yet decided, this test is what forces the decision.
the manual edits interact with re-ingestion exactly as ADR-0002's
"Re-ingestion versus manual edits" specifies.
3. Structured logging at the mutation boundary with stable event names
(`points.updated`, `points.reordered`, `points.soft_deleted`) carrying
`request_id`, `tenant_id`, `file_id`, and the resulting version.

303
docs/runbook.md Normal file
View File

@@ -0,0 +1,303 @@
# Operator runbook
How to start this service, configure its ingestion bounds, and investigate or
retry a failed upload. Architecture rationale lives in [`docs/adr/`](adr/); the
implementation milestone is
[plan 001](plans/001-ingestion-vertical-slice.md). This document covers
operating what those describe.
The service is a **single process with no background work**. `POST /v1/files`
parses, chunks, embeds, and indexes inline and returns a terminal result
(ADR-0017). There is no queue, no worker, and no automatic retry — the caller
owns the retry decision, which makes the request's duration a deployment
constraint. That fact drives most of this document.
## 1. Prerequisites and local startup
Docker, and [`uv`](https://docs.astral.sh/uv/) with Python 3.13.
```bash
cp .env.example .env # non-secret local defaults; .env is gitignored
uv sync
docker compose up -d --wait
```
`docker-compose.yml` runs Postgres (`127.0.0.1:5433`), MinIO
(`127.0.0.1:9100`, console `9101`), and Qdrant (`127.0.0.1:6343`). It is the
local development stack and says so in its header — it is not a production
deployment.
## 2. Deployment steps
Two schema steps run **before** the application, never at startup: FastAPI
performs no DDL, for Postgres (ADR-0009) or for Qdrant (ADR-0001, "Collection
provisioning"). Both commands and their reasoning are in the README's
[Provisioning the datastores](../README.md#provisioning-the-datastores)
section:
```bash
uv run alembic upgrade head # Postgres schema
uv run python -m src.cli.qdrant_bootstrap # the `chunks` collection
```
Both are idempotent. `qdrant_bootstrap` verifies an existing collection against
the pinned schema and **exits non-zero on a mismatch** rather than leaving a
silently degraded sparse index in place — the `sparse` vector's
`modifier="idf"` and the pinned dense dimensions (768 / 3072) fail silently if
wrong, which is why they are checked rather than assumed.
Run both again after every deploy that ships a migration or a collection-schema
change.
## 3. MinIO bucket
Under Compose the bucket already exists: the `app-minio` service's entrypoint
runs `mkdir -p /data/${MINIO_BUCKET:-chatbot-source-files}` before starting the
server, so first boot creates it. Nothing else needs to be done locally.
Outside Compose, create the bucket named by `MINIO_BUCKET` before the first
upload — the application never creates it. It must stay **private**; ADR-0013
keeps source bytes non-public and this slice ships no download API or presigned
URLs.
## 4. Provisioning a tenant, an API key, and its domains
Nothing over HTTP can bootstrap a tenant: every `/v1` route needs an API key,
and a key cannot exist before its tenant. So the first key is issued by an
operator command:
```bash
uv run python -m src.cli.provision_tenant \
--slug acme --domain fire --domain life --scopes files:write,domains:read
```
It prints `api_key=sk_...` **once**. Postgres stores only its SHA-256 hash
(ADR-0009), so a lost key is reissued by re-running the command, never
recovered. Structured logs carry only the non-secret `key_prefix` — a plaintext
key must never reach a log sink (ADR-0011).
Re-running with the same `--slug` reuses the tenant and any domains it already
has, and issues an **additional** key. Both keys stay valid; this adds a key, it
does not rotate one.
Scopes are the security boundary between uploading, reading chunks, and
managing the allowlist. Give an upload client `files:write` only. `domains:write`
lets its holder create new domains, which is exactly what the allowlist exists to
prevent an upload key from doing, and `points:read` lets its holder read the text
of every chunk of every file — so an upload-only key gets neither.
| Scope | Grants |
|---|---|
| `files:write` | Upload a document and read its ingestion status. |
| `points:read` | Read, list, count, and keyword-search this tenant's points, including `GET /v1/files/{file_id}/points`. |
| `points:write` | Create, edit, reorder, and soft-delete points (plan 002 Phases 3-5; no route uses it yet). |
| `domains:read` / `domains:write` | Inspect and manage the domain allowlist. |
| `admin` | Satisfies every scope check. |
The command's `--scopes` default issues all of the above except `admin`, which
suits a first operator key; narrow it explicitly for per-client keys.
### Domains after the first one
`POST /v1/files` rejects an unregistered or disabled `domain` with `400`
(`unknown_domain`) before anything is written. Ongoing domain management is the
`/v1/domains` API, under `domains:read` / `domains:write`:
```bash
curl -X POST http://localhost:8000/v1/domains \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"domain": "fire", "display_name": "Fire insurance"}'
```
The `domain` key itself is immutable — it is denormalized into every Qdrant
point payload and into `source_files`, so renaming it is a migration, not an
edit (ADR-0009). Disabling a domain blocks new uploads; it does not delete
existing points.
## 5. Running the service
```bash
uv run fastapi dev src/main.py # local, reload
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000 # deployed shape
```
Run more than one worker/replica only after reading §6: ingestion bounds are
**per process**, so `INGESTION_MAX_CONCURRENCY` multiplies by the number of
processes.
## 6. Ingestion bounds and tuning
Every bound is enforced server-side and maps to a status code. All are in
`.env.example`. Ingestion is CPU- and network-bound in the request, so these are
the numbers that decide whether the service degrades gracefully or falls over.
| Setting | Bounds | On breach | Size it against |
|---|---|---|---|
| `INGESTION_MAX_CONCURRENCY` | Ingestions in flight **per process** | `503` + `Retry-After` | Memory per in-flight upload (whole file plus its chunks and vectors are resident) and the embedder's capacity. Rejecting is deliberate: ADR-0017 refuses rather than queues. |
| `INGESTION_THREAD_POOL_SIZE` | Threads for blocking work (parse, chunk, hash, BM25, the sync `minio` SDK) | — (waits) | CPU cores. It exists to stop ingestion exhausting Starlette's own thread pool, so keep it below the total thread budget. |
| `INGESTION_TIMEOUT_SECONDS` | The whole work phase | `504`, job marked `failed` | The slowest legitimate document, plus headroom. See §7 — this must stay under every read timeout in front of it. |
| `INGESTION_MAX_UPLOAD_SIZE_MB` | Bytes accepted | `413` | Memory: the upload is read fully into the process before any work starts. |
| `INGESTION_MAX_CHUNKS_PER_FILE` | Chunks per file, checked before embedding | `413` | Embedder cost/time per chunk × `INGESTION_TIMEOUT_SECONDS`. This is the real defence against one pathological file eating a slot. |
| `INGESTION_EMBED_BATCH_SIZE` | Texts per embedder request | `502` on embedder failure | The provider's per-request limits. Batch before parallelizing. |
| `INGESTION_EMBED_CONCURRENCY` | Concurrent embed batches | `502` | Provider rate limits and the self-hosted embedder's throughput. Never unbounded. |
| `QDRANT_UPSERT_BATCH_SIZE` / `_CONCURRENCY` | Points per upsert and concurrent upserts | `502` (`index_error`) | Qdrant's ingest capacity; the batch size stays in ADR-0001's 64–256 band. |
Two settings that look like tuning knobs but are not:
- **`EMBEDDING_NOMIC_KEEP_ALIVE`** holds the self-hosted model resident. A cold
load of `nomic-embed-text-v2-moe` takes over 150 s — longer than any sane
`INGESTION_TIMEOUT_SECONDS` — so an idle period followed by an upload would
otherwise `504`. The lifespan also warms both dense embedders at startup for
the same reason.
- **The BM25 analyzer and weights** (`EMBEDDING_SPARSE_*`) are a measured
artifact ported from the `emet` evaluation lab, verified token-for-token
against it (ADR-0005). Re-benchmark; do not tune them in place.
## 7. The proxy and client read-timeout requirement
**Every read timeout in front of this service must exceed
`INGESTION_TIMEOUT_SECONDS`.** That includes the reverse proxy / ingress, any
load balancer, and the calling backend's own HTTP client.
If a proxy times out first, the client gets that proxy's error, the upload keeps
running in the process, and the caller learns nothing about the outcome from the
response. The job row still reaches a terminal status, so
`GET /v1/files/{file_id}` remains the way to find out what happened — but the
response contract is broken for that request. ADR-0017 names this the main cost
of inline ingestion.
A workable local ordering: client read timeout > proxy read timeout >
`INGESTION_TIMEOUT_SECONDS`.
## 8. Health and readiness
| Endpoint | Question it answers | Use for |
|---|---|---|
| `GET /healthz` | Is the process alive? | Liveness probes / restart policy. Never depends on Postgres, MinIO, or Qdrant. |
| `GET /readyz` | Can it actually serve? | Load-balancer admission and post-deploy gating. `200` with each dependency `true`, `503` if any is `false`. |
`/readyz` checks Postgres, MinIO, and Qdrant reachability **and** that the
`chunks` collection exists. A reachable-but-unbootstrapped Qdrant reports
`{"qdrant": false}` on purpose: uploads to it would fail with `502`, so it is
not ready, and this is how a skipped `qdrant_bootstrap` surfaces at deploy time
instead of on a user's first upload.
## 9. Investigating a failure
Logs are structured (`structlog`, JSON in production) with stable event names —
grep the event name, not prose (ADR-0011). Set `LOG_FILE_PATH` for a local
JSON file sink alongside the console renderer; leave it unset in production,
where stdout collection is preferred.
**Correlate by `request_id`.** Every request has one, echoed in the
`X-Request-Id` response header and included in every error envelope, and bound
into every log line emitted while handling that request. A client reporting a
failed upload should quote it. `tenant_id`, `file_id`, and `ingestion_job_id`
are the other join keys.
Events worth knowing:
| Event | Level | Means |
|---|---|---|
| `ingestion.job.started` | info | Txn A committed; work phase beginning. Carries `tenant_id`, `ingestion_job_id`, `file_id`, `domain`, `source_type`. |
| `ingestion.job.completed` | info | Terminal success, with `chunks_parsed`, `points_upserted`, `points_soft_deleted`. |
| `ingestion.job.failed` | warning | Terminal failure. **`error_code` says which stage**: `storage_upload_failed`, `parse_failed`, `chunk_limit_exceeded`, `embedding_failed`, `index_failed`, `timeout`. |
| `files.upload.duplicate` | info | Identical content already ingested; the existing file/job was returned and nothing was re-ingested. |
| `domain.rejected` | warning | Upload refused before any row was written; `reason` is `unregistered` or `disabled`. |
| `auth.failed` | warning | `reason` is `malformed_key`, `unknown_key`, `key_inactive`, `key_expired`, or `tenant_inactive`. Never contains key material. |
| `auth.succeeded` | info | Carries `tenant_id`, `api_key_id`, `actor_type`. |
| `lifespan.embedder.warm_failed` | warning | An embedder was unreachable at startup. Boot continues by design — `/readyz` and the first upload are where this bites. |
| `qdrant.bootstrap.schema_mismatch` | error | The existing collection diverges from the pinned schema. The bootstrap exits non-zero; do not start the app against it. |
| `api.unhandled_exception` | error | A bug: an exception with no mapping to the error envelope. Always worth a look. |
A `503` (`ingestion_at_capacity`) is rejected before a job row exists, so it
appears in the access log and metrics, not in `ingestion_jobs`.
### The durable record
Logs may roll; `ingestion_jobs` and `ingestion_job_events` do not. For one file:
```sql
SELECT id, status, error_code, error_message, points_created, points_soft_deleted,
created_at, updated_at
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND source_file_id = :file_id
ORDER BY created_at DESC;
SELECT stage, level, message, details, created_at
FROM ingestion_job_events
WHERE tenant_id = :tenant_id AND ingestion_job_id = :ingestion_job_id
ORDER BY created_at;
```
Recent failures across a tenant:
```sql
SELECT error_code, count(*), max(created_at)
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND status = 'failed' AND created_at > now() - interval '1 day'
GROUP BY error_code ORDER BY 2 DESC;
```
`GET /v1/files/{file_id}` reports the same terminal status over HTTP, scoped to
the owning tenant — a file belonging to another tenant returns `404`, not `403`.
## 10. Retrying a failed ingestion
**Re-upload the same bytes.** There is no retry endpoint and no automatic retry;
the client owns that decision (ADR-0017).
What that guarantees:
- Identical content with a **succeeded** job is recognized by
`(tenant_id, domain, content_sha256)` and returned as-is with `200` — no
re-ingestion, no duplicate points.
- Identical content whose last job **failed** starts a fresh job against the
same `source_files` row. A terminal job is never moved back to `running`.
- Point ids are deterministic from `file_id` + `chunk_index` (ADR-0001), so the
retry **overwrites in place** — it cannot duplicate chunks.
- A failed attempt never empties or partially deletes a working index: the
soft-delete sweep that retires a shortened file's leftover points runs only
after every upsert has succeeded. An interrupted attempt can leave a prefix
updated; a retry converges (ADR-0017, "Re-running an ingestion stays safe").
Fix the cause first — the `error_code` says where to look:
| `error_code` | Usual cause |
|---|---|
| `parse_failed` | The file is corrupt or is not really the type its extension claims. Retrying identical bytes will fail identically. |
| `chunk_limit_exceeded` | The file is genuinely too large for one inline ingestion. Split it, or raise `INGESTION_MAX_CHUNKS_PER_FILE` knowing what §6 says about the timeout. |
| `embedding_failed` | The embedder is down, rate-limiting, or unauthenticated. Fix it, then retry — this one usually succeeds unchanged. |
| `index_failed` | Qdrant is down, or the collection is missing (run `qdrant_bootstrap`). |
| `timeout` | The work exceeded `INGESTION_TIMEOUT_SECONDS`. Check whether the embedder was cold (see `lifespan.embedder.warm_failed` and `KEEP_ALIVE`) before raising the bound. |
| `storage_upload_failed` | MinIO is unreachable or the bucket is missing (§3). |
## 11. What to alert on
ADR-0017's own triggers for moving ingestion back off the request path. These
are the numbers that say the inline design has stopped fitting:
- **p95 ingestion duration** approaching `INGESTION_TIMEOUT_SECONDS`.
- **`503` and `504` rates** ceasing to be negligible.
- **Jobs stuck in `running` past the timeout** — every handled failure writes a
terminal status, so a non-zero count here means the process died mid-request:
```sql
SELECT count(*) FROM ingestion_jobs
WHERE status = 'running' AND created_at < now() - interval '5 minutes';
```
Also worth alerting: any `qdrant.bootstrap.schema_mismatch`, a sustained
`/readyz` `503`, and any `api.unhandled_exception`.
## 12. Verifying a deployment
```bash
./scripts/smoke.sh
```
Brings up Compose, runs both deployment steps, provisions a throwaway tenant,
starts the web process, and drives an upload through to indexed Qdrant points
against the **running process** — including asserting the structured log output
from §9. It is the only Compose-based test; everything else runs on
Testcontainers under `uv run pytest` (ADR-0016). Run it before a release.

View File

@@ -37,6 +37,11 @@ dev = [
testpaths = ["tests"]
asyncio_mode = "strict"
timeout = 10
# Bound the test function only, not fixture setup. Testcontainers' container
# startup is charged to whichever test first pulls a session-scoped container
# fixture; on a cold Docker cache that is ~25s and would trip the 10s budget
# for every integration test, regardless of how fast the test itself is.
timeout_func_only = true
markers = [
"unit: fast tests with no external services",
"integration: tests against a real disposable service",

98
scripts/smoke.sh Executable file
View File

@@ -0,0 +1,98 @@
#!/usr/bin/env bash
# Serialized operational smoke test of the running web process (ADR-0016, plan
# 001 Phase 6).
#
# ./scripts/smoke.sh
#
# Brings up the Compose stack, runs both deployment steps for real, provisions
# a tenant, starts uvicorn, and drives `tests/e2e/test_compose_smoke.py`
# against it over a socket. This is the only Compose-based test: every other
# test uses Testcontainers and an in-process ASGI transport, which is exactly
# what makes this one worth having -- it is the only thing that exercises the
# deployment steps, the real logging configuration, and a real HTTP server.
#
# Not part of `uv run pytest`: the smoke test skips itself unless SMOKE_BASE_URL
# is set, so this script is the only way it runs. Run it before a release.
#
# Leaves the Compose stack running (it is the local dev stack); only the uvicorn
# process and the temporary log file are cleaned up.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
PORT="${SMOKE_PORT:-8021}"
SLUG="smoke-$(date +%s)"
DOMAIN="smoke"
LOG_FILE="$(mktemp -t smoke-app-log.XXXXXX.jsonl)"
CONSOLE_LOG="$(mktemp -t smoke-app-console.XXXXXX.log)"
APP_PID=""
cleanup() {
if [[ -n "${APP_PID}" ]] && kill -0 "${APP_PID}" 2>/dev/null; then
kill "${APP_PID}" 2>/dev/null || true
wait "${APP_PID}" 2>/dev/null || true
fi
rm -f "${LOG_FILE}" "${CONSOLE_LOG}"
}
trap cleanup EXIT
if [[ ! -f .env ]]; then
echo "no .env found; copy .env.example first (see docs/runbook.md)" >&2
exit 1
fi
echo "==> starting Postgres, MinIO, Qdrant"
docker compose up -d --wait
echo "==> applying deployment steps"
uv run alembic upgrade head
uv run python -m src.cli.qdrant_bootstrap
echo "==> provisioning tenant '${SLUG}'"
PROVISION_OUTPUT="$(uv run python -m src.cli.provision_tenant \
--slug "${SLUG}" --domain "${DOMAIN}" --scopes files:write 2>/dev/null)"
API_KEY="$(printf '%s\n' "${PROVISION_OUTPUT}" | sed -n 's/^api_key=//p')"
if [[ -z "${API_KEY}" ]]; then
echo "provisioning did not return an api_key" >&2
exit 1
fi
echo "==> starting the web process on port ${PORT}"
# JSON to a file sink, because the smoke test asserts the real ADR-0011 log
# output -- the one thing no in-process test can check.
LOG_JSON_FORMAT=true LOG_FILE_PATH="${LOG_FILE}" \
uv run python -m uvicorn src.main:app --host 127.0.0.1 --port "${PORT}" \
>"${CONSOLE_LOG}" 2>&1 &
APP_PID=$!
echo "==> waiting for /readyz"
for _ in $(seq 1 60); do
if curl -fsS "http://127.0.0.1:${PORT}/readyz" >/dev/null 2>&1; then
break
fi
if ! kill -0 "${APP_PID}" 2>/dev/null; then
echo "the web process exited before becoming ready:" >&2
tail -20 "${CONSOLE_LOG}" >&2
exit 1
fi
sleep 1
done
if ! curl -fsS "http://127.0.0.1:${PORT}/readyz" >/dev/null 2>&1; then
# Most often an unbootstrapped Qdrant or an unreachable embedder host; the
# runbook's health/readiness section covers reading this.
echo "the web process never became ready:" >&2
tail -20 "${CONSOLE_LOG}" >&2
exit 1
fi
echo "==> running the smoke test"
SMOKE_BASE_URL="http://127.0.0.1:${PORT}" \
SMOKE_API_KEY="${API_KEY}" \
SMOKE_DOMAIN="${DOMAIN}" \
SMOKE_LOG_PATH="${LOG_FILE}" \
SMOKE_QDRANT_URL="${QDRANT_URL:-http://127.0.0.1:6343}" \
SMOKE_QDRANT_COLLECTION="${QDRANT_COLLECTION:-chunks}" \
uv run python -m pytest tests/e2e/test_compose_smoke.py -q
echo "==> smoke test passed"

View File

@@ -16,15 +16,23 @@ from src.application.auth.errors import (
MissingScopeError,
TenantInactiveError,
)
from src.application.files.errors import FileTooLargeError, InvalidUploadError
from src.application.domains.errors import DomainAlreadyExistsError, UnknownDomainError
from src.application.files.errors import (
FileTooLargeError,
InvalidUploadError,
SourceFileNotFoundError,
)
from src.application.ingestion.errors import (
ChunkLimitExceededError,
DocumentParseError,
EmbedderError,
IngestionAtCapacityError,
IngestionTimeoutError,
PointIndexingError,
UnsupportedSourceTypeError,
)
from src.application.points.errors import PointVersionConflictError
from src.application.points.point import PointNotFoundError
logger = structlog.get_logger(__name__)
@@ -39,11 +47,21 @@ _MAPPING: tuple[tuple[type[Exception], int, str], ...] = (
(TenantInactiveError, status.HTTP_401_UNAUTHORIZED, "tenant_not_found"),
(MissingScopeError, status.HTTP_403_FORBIDDEN, "missing_scope"),
(InvalidUploadError, status.HTTP_400_BAD_REQUEST, "validation_error"),
# 404, never 403: a cross-tenant point id must be indistinguishable from a
# nonexistent one, or the API becomes an existence oracle (ADR-0016).
(PointNotFoundError, status.HTTP_404_NOT_FOUND, "not_found"),
(SourceFileNotFoundError, status.HTTP_404_NOT_FOUND, "not_found"),
# Not "the version guard fired once" — that is retried. This is the service
# giving up after repeated re-plans, i.e. a genuinely contended point.
(PointVersionConflictError, status.HTTP_409_CONFLICT, "conflict"),
(UnknownDomainError, status.HTTP_400_BAD_REQUEST, "unknown_domain"),
(DomainAlreadyExistsError, status.HTTP_409_CONFLICT, "conflict"),
(DocumentParseError, status.HTTP_400_BAD_REQUEST, "validation_error"),
(UnsupportedSourceTypeError, status.HTTP_415_UNSUPPORTED_MEDIA_TYPE, "unsupported_media_type"),
(FileTooLargeError, status.HTTP_413_CONTENT_TOO_LARGE, "payload_too_large"),
(ChunkLimitExceededError, status.HTTP_413_CONTENT_TOO_LARGE, "payload_too_large"),
(EmbedderError, status.HTTP_502_BAD_GATEWAY, "embedder_error"),
(PointIndexingError, status.HTTP_502_BAD_GATEWAY, "index_error"),
(IngestionTimeoutError, status.HTTP_504_GATEWAY_TIMEOUT, "ingestion_timeout"),
)
@@ -92,7 +110,7 @@ def register_exception_handlers(app: FastAPI) -> None:
@app.exception_handler(RequestValidationError)
def _validation_handler(request: Request, exc: RequestValidationError) -> JSONResponse:
return JSONResponse(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
content=_envelope(
"validation_error",
"request validation failed",

View File

@@ -1,6 +1,10 @@
from fastapi import APIRouter
from src.api.routers.domains import router as domains_router
from src.api.routers.files import router as files_router
from src.api.routers.points import router as points_router
router = APIRouter()
router.include_router(domains_router)
router.include_router(files_router)
router.include_router(points_router)

110
src/api/routers/domains.py Normal file
View File

@@ -0,0 +1,110 @@
"""`/v1/domains` (ADR-0008, ADR-0009).
The management surface for a tenant's domain allowlist, used by the calling
backend rather than by an operator with a psql prompt.
Gated on `domains:read`/`domains:write`, deliberately **not** on `files:write`:
if an upload key could create domains, the allowlist would no longer prevent a
typo'd `domain` from creating a Qdrant partition, which is its only purpose.
"""
from typing import Annotated
from fastapi import APIRouter, Depends, status
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.api.dependencies.auth import require_scope
from src.api.schemas.domains import (
CreateDomainRequest,
DomainListResponse,
DomainResponse,
UpdateDomainRequest,
)
from src.application.auth.context import AuthContext
from src.application.domains import (
create_domain,
list_domains,
set_domain_status,
update_domain,
)
from src.bootstrap.dependencies import get_sessionmaker
router = APIRouter(prefix="/domains", tags=["domains"])
_RequireDomainsRead = Annotated[AuthContext, Depends(require_scope("domains:read"))]
_RequireDomainsWrite = Annotated[AuthContext, Depends(require_scope("domains:write"))]
_SessionmakerDep = Annotated[async_sessionmaker[AsyncSession], Depends(get_sessionmaker)]
@router.get("")
async def list_tenant_domains(
auth: _RequireDomainsRead,
sessionmaker: _SessionmakerDep,
include_disabled: bool = False,
) -> DomainListResponse:
results = await list_domains(
sessionmaker, tenant_id=auth.tenant_id, include_disabled=include_disabled
)
return DomainListResponse(domains=[DomainResponse.from_result(item) for item in results])
@router.post("", status_code=status.HTTP_201_CREATED)
async def create_tenant_domain(
request: CreateDomainRequest,
auth: _RequireDomainsWrite,
sessionmaker: _SessionmakerDep,
) -> DomainResponse:
result = await create_domain(
sessionmaker,
tenant_id=auth.tenant_id,
domain=request.domain,
display_name=request.display_name,
metadata=request.metadata,
)
return DomainResponse.from_result(result)
@router.patch("/{domain}")
async def update_tenant_domain(
domain: str,
request: UpdateDomainRequest,
auth: _RequireDomainsWrite,
sessionmaker: _SessionmakerDep,
) -> DomainResponse:
result = await update_domain(
sessionmaker,
tenant_id=auth.tenant_id,
domain=domain,
display_name=request.display_name,
)
return DomainResponse.from_result(result)
@router.delete("/{domain}")
async def disable_tenant_domain(
domain: str,
auth: _RequireDomainsWrite,
sessionmaker: _SessionmakerDep,
) -> DomainResponse:
"""Disable, not delete.
Blocks new uploads and drops the domain from pickers while leaving the
points already indexed under it intact and retrievable. Actually removing
them needs the tenant-erasure workflow plan 001 defers.
"""
result = await set_domain_status(
sessionmaker, tenant_id=auth.tenant_id, domain=domain, status="disabled"
)
return DomainResponse.from_result(result)
@router.post("/{domain}/enable")
async def enable_tenant_domain(
domain: str,
auth: _RequireDomainsWrite,
sessionmaker: _SessionmakerDep,
) -> DomainResponse:
result = await set_domain_status(
sessionmaker, tenant_id=auth.tenant_id, domain=domain, status="active"
)
return DomainResponse.from_result(result)

View File

@@ -1,4 +1,4 @@
"""`POST /v1/files`, `GET /v1/files/{file_id}` (ADR-0008).
"""`POST /v1/files`, `GET /v1/files/{file_id}`, `DELETE /v1/files/{file_id}` (ADR-0008).
Routes adapt HTTP to `application/files` calls; they do not parse, hash,
touch MinIO/Qdrant, or otherwise carry ingestion business logic (ADR-0015).
@@ -13,17 +13,24 @@ from fastapi import APIRouter, Depends, Form, HTTPException, Response, UploadFil
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.api.dependencies.auth import require_scope
from src.api.schemas.files import FileStatusResponse, FileUploadResponse
from src.api.schemas.files import FileDeleteResponse, FileStatusResponse, FileUploadResponse
from src.api.schemas.points import DEFAULT_PAGE_SIZE, LimitQuery, PointListResponse
from src.application.auth.context import AuthContext
from src.application.files.deletion import delete_source_file
from src.application.files.status import get_file_status
from src.application.files.upload import upload_source_file
from src.application.points.queries import list_file_points
from src.application.ports.embedding import DenseEmbedder, SparseEmbedder
from src.application.ports.object_storage import ObjectStorage
from src.application.ports.point_repository import PointRepository
from src.application.ports.point_storage import PointStorage
from src.bootstrap.dependencies import (
get_dense_embedders,
get_ingestion_concurrency_limiter,
get_ingestion_limiter,
get_object_storage,
get_point_repository,
get_point_storage,
get_sessionmaker,
get_settings,
get_sparse_embedder,
@@ -33,8 +40,12 @@ from src.config import Settings
router = APIRouter(prefix="/files", tags=["files"])
_RequireFilesWrite = Annotated[AuthContext, Depends(require_scope("files:write"))]
_RequirePointsRead = Annotated[AuthContext, Depends(require_scope("points:read"))]
_RequirePointsWrite = Annotated[AuthContext, Depends(require_scope("points:write"))]
_SessionmakerDep = Annotated[async_sessionmaker[AsyncSession], Depends(get_sessionmaker)]
_ObjectStorageDep = Annotated[ObjectStorage, Depends(get_object_storage)]
_PointStorageDep = Annotated[PointStorage, Depends(get_point_storage)]
_PointRepositoryDep = Annotated[PointRepository, Depends(get_point_repository)]
_SettingsDep = Annotated[Settings, Depends(get_settings)]
_IngestionLimiterDep = Annotated[CapacityLimiter, Depends(get_ingestion_limiter)]
_ConcurrencyLimiterDep = Annotated[Semaphore, Depends(get_ingestion_concurrency_limiter)]
@@ -50,6 +61,7 @@ async def upload_file(
auth: _RequireFilesWrite,
sessionmaker: _SessionmakerDep,
storage: _ObjectStorageDep,
point_storage: _PointStorageDep,
settings: _SettingsDep,
limiter: _IngestionLimiterDep,
concurrency_limiter: _ConcurrencyLimiterDep,
@@ -60,12 +72,14 @@ async def upload_file(
result = await upload_source_file(
sessionmaker=sessionmaker,
storage=storage,
point_storage=point_storage,
auth=auth,
domain=domain,
filename=file.filename or "",
data=data,
ingestion_settings=settings.ingestion,
chunking_settings=settings.chunking,
qdrant_settings=settings.qdrant,
thread_limiter=limiter,
concurrency_limiter=concurrency_limiter,
dense_embedders=dense_embedders,
@@ -86,3 +100,58 @@ async def get_file(
if result is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="file not found")
return FileStatusResponse.from_result(result)
@router.delete("/{file_id}")
async def delete_file(
file_id: uuid.UUID,
auth: _RequirePointsWrite,
sessionmaker: _SessionmakerDep,
repository: _PointRepositoryDep,
) -> FileDeleteResponse:
"""Soft-delete a file: every active point, then the `source_files` row.
Gated on `points:write` rather than `files:write` for the same reason as the
listing above — the data this destroys is points. Nothing is removed from
Qdrant (ADR-0002); the points are flagged inactive and the row is marked
`soft_deleted`, which is also what makes a later re-upload of the same bytes
ingest afresh instead of matching the duplicate path.
Deleting an already-deleted file is a success reporting `0` points.
"""
points_soft_deleted = await delete_source_file(
sessionmaker,
repository,
tenant_id=auth.tenant_id,
source_file_id=file_id,
actor=f"api_key:{auth.api_key_id}",
)
return FileDeleteResponse(
file_id=file_id, status="soft_deleted", points_soft_deleted=points_soft_deleted
)
@router.get("/{file_id}/points")
async def list_points_for_file(
file_id: uuid.UUID,
auth: _RequirePointsRead,
repository: _PointRepositoryDep,
limit: LimitQuery = DEFAULT_PAGE_SIZE,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointListResponse:
"""The same listing as `GET /v1/points?file_id=...`, addressed by file.
Gated on `points:read`, not `files:write`: the resource being read is the
file's chunks, so the scope follows the data rather than the URL prefix. An
upload-only key must not become a way to read every chunk of every file.
"""
page = await list_file_points(
repository,
tenant_id=auth.tenant_id,
file_id=file_id,
limit=limit,
cursor=cursor,
include_inactive=include_inactive,
)
return PointListResponse.from_page(page)

View File

@@ -23,7 +23,9 @@ async def readyz(request: Request, response: Response) -> dict[str, bool]:
postgres_ready, minio_ready, qdrant_ready = await asyncio.gather(
ping_postgres(resources.db_engine, timeout),
ping_minio(resources.minio_client, timeout),
ping_qdrant(resources.qdrant_client, timeout),
ping_qdrant(
resources.qdrant_client, timeout, collection=resources.settings.qdrant.collection
),
)
result = {

162
src/api/routers/points.py Normal file
View File

@@ -0,0 +1,162 @@
"""`/v1/points` read and soft-delete paths (ADR-0002, ADR-0008).
Routes adapt HTTP to `application/points` calls. They build no Qdrant filters
and hold no CRUD semantics (ADR-0015), and they never read a tenant from the
request — `auth.tenant_id` is the only source, which is what makes ADR-0002's
isolation rule structural rather than a habit.
**Route order is load-bearing.** `/count` and `/search` are declared before
`/{point_id}`. FastAPI matches in declaration order, so with `/{point_id}` first
a request for `/v1/points/count` would try to parse `"count"` as a UUID and
fail with `422` instead of counting anything. The failure is loud but confusing,
and it comes back the moment someone reorders these for tidiness.
Gated on `points:read`, separately from `files:write`: a key that can upload
documents should not thereby be able to read every chunk of every file, and
plan 002's mutating paths will want `points:write` distinct again.
"""
import uuid
from typing import Annotated
from fastapi import APIRouter, Depends, Query
from src.api.dependencies.auth import require_scope
from src.api.schemas.points import (
DEFAULT_PAGE_SIZE,
LimitQuery,
PointCountResponse,
PointListResponse,
PointResponse,
PointSearchResponse,
)
from src.application.auth.context import AuthContext
from src.application.points.deletion import soft_delete_point
from src.application.points.queries import (
count_points,
get_point,
list_file_points,
search_points,
)
from src.application.ports.point_repository import PointRepository
from src.bootstrap.dependencies import get_point_repository
router = APIRouter(prefix="/points", tags=["points"])
_RequirePointsRead = Annotated[AuthContext, Depends(require_scope("points:read"))]
_RequirePointsWrite = Annotated[AuthContext, Depends(require_scope("points:write"))]
_PointRepositoryDep = Annotated[PointRepository, Depends(get_point_repository)]
@router.get("/count")
async def count_tenant_points(
auth: _RequirePointsRead,
repository: _PointRepositoryDep,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointCountResponse:
count = await count_points(
repository,
tenant_id=auth.tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
return PointCountResponse(count=count)
@router.get("/search")
async def search_tenant_points(
auth: _RequirePointsRead,
repository: _PointRepositoryDep,
q: Annotated[str, Query(min_length=1)],
limit: LimitQuery = DEFAULT_PAGE_SIZE,
cursor: str | None = None,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointSearchResponse:
"""Keyword search over point content — **not** semantic retrieval.
Matches Qdrant's full-text payload index on `content`, combined with the
structured filters below. Results are unranked: the index filters rather
than scores, so there is no relevance order and no score to return. Callers
wanting ranked answers want the agent retrieval path (plan 003), not this.
"""
page = await search_points(
repository,
tenant_id=auth.tenant_id,
query=q,
limit=limit,
cursor=cursor,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
return PointSearchResponse.from_search(page, query=q)
@router.get("")
async def list_tenant_points(
auth: _RequirePointsRead,
repository: _PointRepositoryDep,
file_id: uuid.UUID,
limit: LimitQuery = DEFAULT_PAGE_SIZE,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointListResponse:
"""A file's points in `order_id` order.
`file_id` is required rather than optional: the pagination cursor is an
`order_id` value, and `order_id` is only unique within one file. Listing
across files would silently drop or repeat rows at every page boundary.
"""
page = await list_file_points(
repository,
tenant_id=auth.tenant_id,
file_id=file_id,
limit=limit,
cursor=cursor,
include_inactive=include_inactive,
)
return PointListResponse.from_page(page)
@router.get("/{point_id}")
async def get_tenant_point(
point_id: uuid.UUID,
auth: _RequirePointsRead,
repository: _PointRepositoryDep,
with_vectors: bool = False,
) -> PointResponse:
point = await get_point(
repository, tenant_id=auth.tenant_id, point_id=point_id, with_vectors=with_vectors
)
return PointResponse.from_point(point)
@router.delete("/{point_id}")
async def delete_tenant_point(
point_id: uuid.UUID,
auth: _RequirePointsWrite,
repository: _PointRepositoryDep,
) -> PointResponse:
"""Soft-delete one point and relink its neighbours around the gap.
The point is never removed from Qdrant (ADR-0002): it is flagged
`is_active=false` with `deleted_at` set, and its old neighbours are pointed
at each other in the same batch, so context-window expansion never walks
into it.
Deleting an already-inactive point is a no-op success rather than a `404` —
the response is the point as it stands, so the resulting `version` and
`deleted_at` are visible either way.
"""
point = await soft_delete_point(
repository,
tenant_id=auth.tenant_id,
point_id=point_id,
actor=f"api_key:{auth.api_key_id}",
)
return PointResponse.from_point(point)

View File

@@ -0,0 +1,63 @@
"""Public request/response models for `/v1/domains` (ADR-0008, ADR-0009).
`tenant_id` appears in none of these: it comes from the authenticated key, and
accepting it from a body would break the isolation boundary (ADR-0002).
"""
import uuid
from datetime import datetime
from pydantic import BaseModel, Field, field_validator
from src.application.domains.models import DomainResult
# Lowercase alphanumerics plus - and _; the key is embedded in every Qdrant
# payload and filtered on as a keyword, so it stays boring on purpose.
_DOMAIN_PATTERN = r"^[a-z0-9][a-z0-9_-]*$"
class DomainResponse(BaseModel):
id: uuid.UUID
domain: str
display_name: str
status: str
metadata: dict[str, object]
created_at: datetime
updated_at: datetime
@classmethod
def from_result(cls, result: DomainResult) -> "DomainResponse":
return cls(
id=result.id,
domain=result.domain,
display_name=result.display_name,
status=result.status,
metadata=result.metadata,
created_at=result.created_at,
updated_at=result.updated_at,
)
class DomainListResponse(BaseModel):
domains: list[DomainResponse]
class CreateDomainRequest(BaseModel):
domain: str = Field(min_length=1, max_length=80, pattern=_DOMAIN_PATTERN)
display_name: str = Field(min_length=1, max_length=200)
metadata: dict[str, object] = Field(default_factory=dict)
@field_validator("domain")
@classmethod
def _normalize(cls, value: str) -> str:
return value.strip()
class UpdateDomainRequest(BaseModel):
"""`domain` is absent by design — the key is immutable.
It is denormalized into every point payload and into `source_files`, so
renaming it is a migration rather than an edit (ADR-0009).
"""
display_name: str = Field(min_length=1, max_length=200)

View File

@@ -28,6 +28,20 @@ class FileUploadResponse(BaseModel):
)
class FileDeleteResponse(BaseModel):
"""What `DELETE /v1/files/{file_id}` did.
`points_soft_deleted` is reported rather than left implicit because the
delete is a soft one: nothing is removed from Qdrant, and the count is the
only way a caller can tell "deactivated 40 points" from "the file was
already deleted" — both of which are successes.
"""
file_id: uuid.UUID
status: str
points_soft_deleted: int
class FileStatusResponse(BaseModel):
file_id: uuid.UUID
source_filename: str

165
src/api/schemas/points.py Normal file
View File

@@ -0,0 +1,165 @@
"""Public request/response models for `/v1/points` (ADR-0002, ADR-0008).
The shape callers see, kept separate from `application/points`' domain models
(ADR-0015). Two rules are encoded here rather than left to route code:
- **Vectors are opt-in.** `PointResponse` omits them unless the caller asked,
so a listing does not ship megabytes of floats nobody reads (ADR-0008).
- **Server-owned fields are not accepted on input.** The request models simply
do not declare `tenant_id`, `version`, or `chunk_index`, and forbid extra
keys, so a client that sends one gets `422` from Pydantic instead of having
it silently ignored — ADR-0002's isolation rule enforced at the boundary.
"""
import uuid
from datetime import datetime
from typing import Annotated
from fastapi import Query
from pydantic import BaseModel, ConfigDict, Field
from src.application.points.point import Point
from src.application.ports.point_repository import PointPage
# A page ceiling the caller cannot raise. Scroll pages are materialized in
# memory both here and in Qdrant, so an unbounded `limit` is a cheap way for one
# request to hurt every other tenant sharing the process. Declared once because
# two routers paginate points -- `/v1/points` and `/v1/files/{file_id}/points` --
# and a ceiling that differs between them is a ceiling in only one of them.
DEFAULT_PAGE_SIZE = 50
MAX_PAGE_SIZE = 200
LimitQuery = Annotated[int, Query(ge=1, le=MAX_PAGE_SIZE)]
class PointResponse(BaseModel):
point_id: uuid.UUID
domain: str
file_id: uuid.UUID
chunk_id: uuid.UUID
content: str
content_type: str
source_filename: str
source_type: str
order_id: float
chunk_index: int
previous_chunk_id: uuid.UUID | None
next_chunk_id: uuid.UUID | None
is_active: bool
deleted_at: datetime | None
created_at: datetime
updated_at: datetime
created_by: str
updated_by: str
version: int
content_hash: str
embedding_model_version: str
vectors: dict[str, object] | None = None
@classmethod
def from_point(cls, point: Point) -> "PointResponse":
# `tenant_id` is present on `Point` and deliberately absent here: the
# caller already knows which tenant it authenticated as, and echoing it
# back invites clients to start sending it.
return cls.model_validate(point.model_dump(exclude={"tenant_id"}))
class PointListResponse(BaseModel):
"""A page of points plus the cursor for the next one.
Cursor-based rather than `limit`/`offset`: an offset cursor silently skips
or repeats rows when a concurrent insert shifts positions, which is exactly
the pagination defect plan 002 requires a test for.
"""
points: list[PointResponse]
next_cursor: str | None = None
@classmethod
def from_page(cls, page: PointPage) -> "PointListResponse":
return cls(
points=[PointResponse.from_point(point) for point in page.points],
next_cursor=page.next_cursor,
)
class PointCountResponse(BaseModel):
count: int
class PointSearchResponse(PointListResponse):
"""Results of a **keyword** match, not of semantic retrieval.
Named and documented so it cannot be mistaken for ADR-0003's hybrid
retrieval: these points matched a full-text filter on `content`, they are
not ranked by relevance, and there is no score to report. Anything that
wants ranked results wants the agent retrieval path in plan 003.
"""
query: str
@classmethod
def from_search(cls, page: PointPage, *, query: str) -> "PointSearchResponse":
return cls(
query=query,
points=[PointResponse.from_point(point) for point in page.points],
next_cursor=page.next_cursor,
)
class PointCreateRequest(BaseModel):
"""Create one point. The server assigns identity, ordering, and provenance.
`after_point_id` positions the new point rather than a raw `order_id`: the
caller says where in the sequence it goes and the server computes the
fractional key and relinks neighbours, which a client-supplied `order_id`
could not do correctly (ADR-0002). `None` means "at the start of the file".
"""
model_config = ConfigDict(extra="forbid")
file_id: uuid.UUID
content: str = Field(min_length=1)
content_type: str = "paragraph"
after_point_id: uuid.UUID | None = None
class PointReplaceRequest(BaseModel):
"""Replace a point's content under a version guard.
`version` here is the *expected* version, not a value being written — the
optimistic-concurrency precondition. A mismatch is `409`.
"""
model_config = ConfigDict(extra="forbid")
content: str = Field(min_length=1)
content_type: str | None = None
version: int
class PointPayloadPatchRequest(BaseModel):
"""Payload-only update of caller-writable fields, under a version guard."""
model_config = ConfigDict(extra="forbid")
payload: dict[str, object]
version: int
class PointReorderRequest(BaseModel):
"""Move a point to sit immediately after `after_point_id`.
`None` moves it to the front of the file. Expressed as a neighbour rather
than an `order_id` for the same reason as `PointCreateRequest`.
"""
model_config = ConfigDict(extra="forbid")
after_point_id: uuid.UUID | None = None
version: int

View File

@@ -8,6 +8,7 @@ must run with no Postgres session held open at all.
from datetime import UTC, datetime
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.context import AuthContext
@@ -16,28 +17,66 @@ from src.application.auth.keys import parse_api_key, verify_secret
from src.infrastructure.postgres.repositories import api_keys as api_keys_repo
from src.infrastructure.postgres.repositories import tenants as tenants_repo
logger = structlog.get_logger(__name__)
async def resolve_auth_context(
sessionmaker: async_sessionmaker[AsyncSession], bearer_token: str
) -> AuthContext:
"""Resolve a bearer token, logging the outcome either way (ADR-0011).
This runs on every authenticated request, so `auth.failed` is the one
event most likely to matter first when diagnosing a client integration
issue -- and the reason string alone (never logged; it can echo back
attacker-supplied key material) is not enough to tell a malformed token
apart from a revoked one without this.
"""
parsed = parse_api_key(bearer_token)
if parsed is None:
logger.warning("auth.failed", reason="malformed_key")
raise InvalidApiKeyError("malformed API key")
key_prefix, secret = parsed
async with sessionmaker() as session:
api_key = await api_keys_repo.get_by_prefix(session, key_prefix)
if api_key is None or not verify_secret(secret, api_key.key_hash):
logger.warning("auth.failed", reason="unknown_key", key_prefix=key_prefix)
raise InvalidApiKeyError("unknown API key")
if api_key.status != "active":
logger.warning(
"auth.failed",
reason="key_inactive",
key_prefix=key_prefix,
api_key_id=str(api_key.id),
key_status=api_key.status,
)
raise InvalidApiKeyError(f"API key is {api_key.status}")
if api_key.expires_at is not None and api_key.expires_at <= datetime.now(UTC):
logger.warning(
"auth.failed",
reason="key_expired",
key_prefix=key_prefix,
api_key_id=str(api_key.id),
)
raise InvalidApiKeyError("API key has expired")
tenant = await tenants_repo.get_by_id(session, api_key.tenant_id)
if tenant is None or tenant.status != "active":
logger.warning(
"auth.failed",
reason="tenant_inactive",
key_prefix=key_prefix,
api_key_id=str(api_key.id),
tenant_id=str(api_key.tenant_id),
)
raise TenantInactiveError("tenant is not active")
logger.info(
"auth.succeeded",
tenant_id=str(tenant.id),
api_key_id=str(api_key.id),
actor_type=api_key.actor_type,
)
return AuthContext(
tenant_id=tenant.id,
tenant_slug=tenant.slug,

View File

@@ -0,0 +1,27 @@
"""Tenant-domain management and the upload-time allowlist check (ADR-0009)."""
from src.application.domains.errors import (
DomainAlreadyExistsError,
DomainsError,
UnknownDomainError,
)
from src.application.domains.models import DomainResult
from src.application.domains.service import (
create_domain,
ensure_domain_allowed,
list_domains,
set_domain_status,
update_domain,
)
__all__ = [
"DomainAlreadyExistsError",
"DomainResult",
"DomainsError",
"UnknownDomainError",
"create_domain",
"ensure_domain_allowed",
"list_domains",
"set_domain_status",
"update_domain",
]

View File

@@ -0,0 +1,21 @@
"""Domain-management failures (ADR-0009). No HTTP knowledge here —
`src/api/errors.py` maps these to status codes.
"""
class DomainsError(Exception):
"""Base class for tenant-domain failures."""
class UnknownDomainError(DomainsError):
"""The upload named a domain the tenant has not registered, or one that is
disabled. Maps to `400`.
Rejecting is the whole point: an unrecognized `domain` would otherwise
create a new Qdrant partition silently, and a file in a partition nothing
queries is invisible rather than failed (ADR-0009).
"""
class DomainAlreadyExistsError(DomainsError):
"""The tenant already has a domain with this key. Maps to `409`."""

View File

@@ -0,0 +1,16 @@
"""Transport-agnostic results for the domain-management service."""
import uuid
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class DomainResult:
id: uuid.UUID
domain: str
display_name: str
status: str
metadata: dict[str, object]
created_at: datetime
updated_at: datetime

View File

@@ -0,0 +1,164 @@
"""Tenant-domain management (ADR-0009).
A tenant's domain set is per-tenant and varies in size — one may run 14
insurance lines, another 6 — so it is data, not an enum.
`ensure_domain_allowed` is the reason this package exists: it is the strict
allowlist check the upload path runs before anything is written. Everything
else here is the management surface the calling backend uses to populate that
allowlist, under its own `domains:write` scope so an upload key cannot create
partitions.
`tenant_id` is always a required parameter taken from `AuthContext`, never from
a request body (ADR-0002).
"""
import uuid
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.domains.errors import DomainAlreadyExistsError, UnknownDomainError
from src.application.domains.models import DomainResult
from src.infrastructure.postgres.models.tenant_domain import TenantDomain
from src.infrastructure.postgres.repositories import tenant_domains as repo
logger = structlog.get_logger(__name__)
async def _flush_and_refresh(session: AsyncSession, tenant_domain: TenantDomain) -> None:
"""Materialize server-generated columns before the row leaves the session.
`updated_at` is `onupdate=func.now()`, so after an UPDATE its value lives in
the database, not in the instance. Reading it later would trigger a lazy
load outside any greenlet context (`MissingGreenlet`), so it is fetched here
while the session is still open.
"""
await session.flush()
await session.refresh(tenant_domain)
def _to_result(tenant_domain: TenantDomain) -> DomainResult:
return DomainResult(
id=tenant_domain.id,
domain=tenant_domain.domain,
display_name=tenant_domain.display_name,
status=tenant_domain.status,
metadata=tenant_domain.metadata_,
created_at=tenant_domain.created_at,
updated_at=tenant_domain.updated_at,
)
async def ensure_domain_allowed(
session: AsyncSession, *, tenant_id: uuid.UUID, domain: str
) -> None:
"""Raise `UnknownDomainError` unless the tenant has this domain active.
Takes a session rather than a sessionmaker: the upload path calls this
inside its existing txn A, so the check costs no extra connection and
cannot pass and then go stale before the row is written.
Logs the rejection here rather than at the call site: this runs before any
`ingestion_jobs` row exists, so `upload_source_file`'s job-level
`ingestion.job.failed` event (ADR-0011) never fires for it -- without a log
here, a rejected upload would leave no operational trace at all.
"""
tenant_domain = await repo.get(session, tenant_id=tenant_id, domain=domain)
if tenant_domain is None:
logger.warning(
"domain.rejected", tenant_id=str(tenant_id), domain=domain, reason="unregistered"
)
raise UnknownDomainError(
f"domain '{domain}' is not registered for this tenant; "
f"create it via POST /v1/domains before uploading to it"
)
if tenant_domain.status != "active":
logger.warning(
"domain.rejected", tenant_id=str(tenant_id), domain=domain, reason="disabled"
)
raise UnknownDomainError(f"domain '{domain}' is disabled for this tenant")
async def list_domains(
sessionmaker: async_sessionmaker[AsyncSession],
*,
tenant_id: uuid.UUID,
include_disabled: bool = False,
) -> list[DomainResult]:
async with sessionmaker() as session:
found = await repo.list_for_tenant(
session, tenant_id=tenant_id, include_disabled=include_disabled
)
return [_to_result(item) for item in found]
async def create_domain(
sessionmaker: async_sessionmaker[AsyncSession],
*,
tenant_id: uuid.UUID,
domain: str,
display_name: str,
metadata: dict[str, object] | None = None,
) -> DomainResult:
async with sessionmaker() as session:
if await repo.get(session, tenant_id=tenant_id, domain=domain) is not None:
raise DomainAlreadyExistsError(f"domain '{domain}' already exists for this tenant")
created = repo.create(
session,
tenant_id=tenant_id,
domain=domain,
display_name=display_name,
metadata=metadata,
)
await session.commit()
logger.info("domain.created", tenant_id=str(tenant_id), domain=domain)
return _to_result(created)
async def update_domain(
sessionmaker: async_sessionmaker[AsyncSession],
*,
tenant_id: uuid.UUID,
domain: str,
display_name: str,
) -> DomainResult:
"""Only the label is mutable — see `repo.update_display_name`."""
async with sessionmaker() as session:
found = await repo.get(session, tenant_id=tenant_id, domain=domain)
if found is None:
raise UnknownDomainError(f"domain '{domain}' is not registered for this tenant")
repo.update_display_name(found, display_name=display_name)
await _flush_and_refresh(session, found)
await session.commit()
result = _to_result(found)
logger.info("domain.updated", tenant_id=str(tenant_id), domain=domain)
return result
async def set_domain_status(
sessionmaker: async_sessionmaker[AsyncSession],
*,
tenant_id: uuid.UUID,
domain: str,
status: str,
) -> DomainResult:
"""Disable or re-enable a domain.
Disabling blocks new uploads and hides the domain from pickers. It does not
touch the points already indexed under it — removing those needs the
tenant-erasure workflow plan 001 defers.
"""
async with sessionmaker() as session:
found = await repo.get(session, tenant_id=tenant_id, domain=domain)
if found is None:
raise UnknownDomainError(f"domain '{domain}' is not registered for this tenant")
repo.set_status(found, status=status)
await _flush_and_refresh(session, found)
await session.commit()
result = _to_result(found)
logger.info("domain.status_changed", tenant_id=str(tenant_id), domain=domain, status=status)
return result

View File

@@ -0,0 +1,81 @@
"""`DELETE /v1/files/{file_id}` — retire a file and deactivate its points.
Two stores have to agree here, and the phase boundaries are the same ones
ingestion uses (ADR-0017): a short Postgres transaction to authorize, then the
Qdrant work with **no session held**, then a short transaction to record the
outcome. Holding a session across the sweep would pin a pool connection for the
length of a multi-page delete.
The order — points first, Postgres second — is deliberate. If the sweep dies
half way, the row stays `active` and a retried `DELETE` finishes the job, since
the sweep only ever looks at points that are still active. The reverse order
would leave a row marked deleted while its points are still live and still
retrievable by the agent, which is the failure that actually matters.
"""
import uuid
from datetime import UTC, datetime
from time import perf_counter
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.files.errors import SourceFileNotFoundError
from src.application.points.deletion import soft_delete_file_points
from src.application.ports.point_repository import PointRepository
from src.infrastructure.postgres.repositories import source_files as source_files_repo
logger = structlog.get_logger(__name__)
async def delete_source_file(
sessionmaker: async_sessionmaker[AsyncSession],
repository: PointRepository,
*,
tenant_id: uuid.UUID,
source_file_id: uuid.UUID,
actor: str,
) -> int:
"""Soft-delete a file: every active point, then the `source_files` row.
Returns how many points the sweep deactivated. Raises
`SourceFileNotFoundError` (`404`) when the file is not this tenant's — the
check happens before anything is written, so a probe for another tenant's
file id cannot deactivate a single point.
Idempotent: a second call finds no active points and a row already marked
`soft_deleted`, and returns `0`.
"""
started = perf_counter()
async with sessionmaker() as session:
source_file = await source_files_repo.get_by_id(
session, tenant_id=tenant_id, source_file_id=source_file_id
)
if source_file is None:
raise SourceFileNotFoundError(f"file {source_file_id} not found")
points_soft_deleted = await soft_delete_file_points(
repository, tenant_id=tenant_id, file_id=source_file_id, actor=actor
)
async with sessionmaker() as session:
source_file = await source_files_repo.get_by_id(
session, tenant_id=tenant_id, source_file_id=source_file_id
)
if source_file is None:
raise SourceFileNotFoundError(f"file {source_file_id} not found")
source_files_repo.mark_soft_deleted(source_file, deleted_at=datetime.now(UTC))
await session.commit()
logger.info(
"files.soft_deleted",
tenant_id=str(tenant_id),
file_id=str(source_file_id),
points_soft_deleted=points_soft_deleted,
actor=actor,
# End to end, including both Postgres transactions. Comparing it with
# the sweep's own `duration_ms` on `points.file_soft_deleted` is what
# separates a slow Qdrant from a slow database.
duration_ms=round((perf_counter() - started) * 1000, 2),
)
return points_soft_deleted

View File

@@ -15,3 +15,11 @@ class InvalidUploadError(FilesError):
class FileTooLargeError(FilesError):
"""The upload exceeds `INGESTION_MAX_UPLOAD_SIZE_MB`. Maps to `413`."""
class SourceFileNotFoundError(FilesError):
"""No such source file *within the requesting tenant*. Maps to `404`.
Same non-disclosure rule as points (ADR-0016): a cross-tenant file id and a
nonexistent one are indistinguishable to the caller, so this is never `403`.
"""

View File

@@ -16,9 +16,10 @@ job stuck in `running`. The whole request additionally holds one of
phase 2 is bounded by `INGESTION_TIMEOUT_SECONDS` (`504`) (ADR-0017, plan 001
Phase 4).
Qdrant point upserts are Phase 5 work, not implemented here: this phase
parses, chunks, and embeds, so a successful job still reports
`chunks_indexed=0` — nothing is searchable yet.
Phase 2 ends by upserting the embedded chunks as tenant-scoped Qdrant points
(`src/application/points/`), so a successful upload is searchable by the time
the `201` returns. The collection those points land in is provisioned by a
deployment step, not by this path — see `src/cli/qdrant_bootstrap.py`.
"""
import uuid
@@ -29,6 +30,7 @@ from anyio import CapacityLimiter, Semaphore, fail_after, to_thread
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.context import AuthContext
from src.application.domains import ensure_domain_allowed
from src.application.files.errors import InvalidUploadError
from src.application.files.models import UploadResult
from src.application.files.storage_keys import source_file_object_key
@@ -45,10 +47,13 @@ from src.application.ingestion.errors import (
ChunkLimitExceededError,
EmbedderError,
IngestionTimeoutError,
PointIndexingError,
)
from src.application.points import index_chunks
from src.application.ports.embedding import DenseEmbedder, SparseEmbedder
from src.application.ports.object_storage import ObjectStorage
from src.config import ChunkingSettings, IngestionSettings
from src.application.ports.point_storage import PointStorage
from src.config import ChunkingSettings, IngestionSettings, QdrantSettings
from src.infrastructure.postgres.repositories import ingestion_jobs as jobs_repo
from src.infrastructure.postgres.repositories import source_files as source_files_repo
@@ -63,6 +68,16 @@ async def _mark_job_failed(
error_code: str,
error_message: str,
) -> None:
"""Write the terminal `failed` job row and emit its log event together.
Every failure branch below calls this, so logging here once closes every
branch at once rather than duplicating a `logger.warning` at each call
site (CLAUDE.md, "prefer deep modules") -- previously only
`storage_upload_failed` and `timeout` did that ad hoc, and
`parse_failed`/`chunk_limit_exceeded`/`embedding_failed`/`index_failed`
logged nothing at all: visible in `ingestion_job_events` but invisible to
log-based alerting (ADR-0011).
"""
async with sessionmaker() as session:
job = await jobs_repo.mark_terminal(
session,
@@ -83,17 +98,27 @@ async def _mark_job_failed(
)
await session.commit()
logger.warning(
"ingestion.job.failed",
tenant_id=str(tenant_id),
ingestion_job_id=str(ingestion_job_id),
error_code=error_code,
error_message=error_message,
)
async def upload_source_file(
*,
sessionmaker: async_sessionmaker[AsyncSession],
storage: ObjectStorage,
point_storage: PointStorage,
auth: AuthContext,
domain: str,
filename: str,
data: bytes,
ingestion_settings: IngestionSettings,
chunking_settings: ChunkingSettings,
qdrant_settings: QdrantSettings,
thread_limiter: CapacityLimiter,
concurrency_limiter: Semaphore,
dense_embedders: Sequence[DenseEmbedder],
@@ -112,6 +137,12 @@ async def upload_source_file(
async with acquire_ingestion_slot(concurrency_limiter):
async with sessionmaker() as session:
# Strict allowlist, checked inside txn A before anything is written
# (ADR-0009). An unregistered domain would otherwise create a new
# Qdrant partition silently, leaving the file invisible to
# retrieval rather than failing.
await ensure_domain_allowed(session, tenant_id=auth.tenant_id, domain=domain)
existing = await source_files_repo.find_active_by_content_hash(
session,
tenant_id=auth.tenant_id,
@@ -178,6 +209,15 @@ async def upload_source_file(
await session.commit()
ingestion_job_id = job.id
logger.info(
"ingestion.job.started",
tenant_id=str(auth.tenant_id),
ingestion_job_id=str(ingestion_job_id),
file_id=str(source_file_id),
domain=domain,
source_type=validated.source_type,
)
# Phase 2: no Postgres session open across this work (ADR-0017),
# bounded end-to-end by INGESTION_TIMEOUT_SECONDS.
try:
@@ -187,12 +227,6 @@ async def upload_source_file(
key=object_key, data=data, content_type=validated.content_type
)
except Exception as exc:
logger.warning(
"files.upload.storage_failed",
tenant_id=str(auth.tenant_id),
file_id=str(source_file_id),
ingestion_job_id=str(ingestion_job_id),
)
await _mark_job_failed(
sessionmaker,
tenant_id=auth.tenant_id,
@@ -249,13 +283,32 @@ async def upload_source_file(
error_message=str(exc),
)
raise
try:
indexed = await index_chunks(
embedded,
storage=point_storage,
tenant_id=auth.tenant_id,
domain=domain,
file_id=source_file_id,
source_filename=filename,
source_type=validated.source_type,
actor=f"api_key:{auth.api_key_id}",
dense_embedders=dense_embedders,
sparse_embedder=sparse_embedder,
settings=qdrant_settings,
thread_limiter=thread_limiter,
)
except PointIndexingError as exc:
await _mark_job_failed(
sessionmaker,
tenant_id=auth.tenant_id,
ingestion_job_id=ingestion_job_id,
error_code="index_failed",
error_message=str(exc),
)
raise
except TimeoutError:
logger.warning(
"files.upload.timeout",
tenant_id=str(auth.tenant_id),
file_id=str(source_file_id),
ingestion_job_id=str(ingestion_job_id),
)
await _mark_job_failed(
sessionmaker,
tenant_id=auth.tenant_id,
@@ -273,7 +326,11 @@ async def upload_source_file(
tenant_id=auth.tenant_id,
ingestion_job_id=ingestion_job_id,
status="succeeded",
points_created=0,
# An upsert with deterministic ids cannot tell an insert from
# an overwrite, so every written point is reported here and
# `points_updated` stays 0 rather than being guessed at.
points_created=indexed.points_upserted,
points_soft_deleted=indexed.points_soft_deleted,
)
jobs_repo.append_event(
session,
@@ -281,21 +338,29 @@ async def upload_source_file(
ingestion_job_id=ingestion_job_id,
level="info",
stage="completed",
message="chunks parsed and embedded; Qdrant indexing not yet implemented",
details={"chunks_parsed": len(chunks), "chunks_embedded": len(embedded)},
message="chunks parsed, embedded, and indexed",
details={
"chunks_parsed": len(chunks),
"chunks_embedded": len(embedded),
"points_upserted": indexed.points_upserted,
"points_soft_deleted": indexed.points_soft_deleted,
},
)
await session.commit()
logger.info(
"files.upload.succeeded",
"ingestion.job.completed",
tenant_id=str(auth.tenant_id),
file_id=str(source_file_id),
ingestion_job_id=str(ingestion_job_id),
file_id=str(source_file_id),
chunks_parsed=len(chunks),
points_upserted=indexed.points_upserted,
points_soft_deleted=indexed.points_soft_deleted,
)
return UploadResult(
file_id=source_file_id,
ingestion_job_id=ingestion_job_id,
status="succeeded",
chunks_indexed=0,
chunks_indexed=indexed.points_upserted,
is_new_attempt=True,
)

View File

@@ -49,6 +49,15 @@ class EmbedderError(IngestionError):
"""
class PointIndexingError(IngestionError):
"""Upserting or soft-deleting Qdrant points failed.
Maps to `502` — like `EmbedderError`, this is an upstream dependency
failing, not a malformed request. Kept distinct from `EmbedderError` so the
job's `error_code` says which dependency broke.
"""
class IngestionAtCapacityError(IngestionError):
"""`INGESTION_MAX_CONCURRENCY` in-process ingestions are already running.

View File

@@ -0,0 +1,18 @@
"""Ingestion-generated Qdrant point CRUD (ADR-0001, ADR-0002).
`index_chunks` is the entry point callers outside this package should use: it
dispatches payload construction, batching, bounded-concurrency upserts, and the
post-success soft-delete sweep. `build_chunk_payload` and the batching helpers
stay internal, exported mainly for their own unit tests.
The `/v1/points` surface lives here too, in its own modules with their own
entry points: `queries.py` for the read paths and `deletion.py` for soft delete
with neighbour relinking. They share this package because they share ADR-0001's
payload schema, not because they share a caller — `index_chunks` writes a whole
file at once, while those serve one admin edit at a time.
"""
from src.application.points.indexing import IndexingResult, index_chunks
from src.application.points.models import ChunkPoint
__all__ = ["ChunkPoint", "IndexingResult", "index_chunks"]

View File

@@ -0,0 +1,253 @@
"""Soft delete for `/v1/points` and for a whole file's points (ADR-0002).
The caller-facing entry points are `soft_delete_point` and
`soft_delete_file_points`. Routers call these; `patches_for_removal` and the
planning helpers stay internal, because getting a delete right is exactly the
composition a caller should not have to reassemble: read the point, load its
neighbours, compute the patches still missing, send them in **one** batch,
verify they landed, and retry against fresh versions if they did not.
Why the retry exists. Qdrant has no multi-point transaction, so a batch whose
second operation loses a version race applies its first operation anyway — and
a filtered `set_payload` that matched nothing still reports success. That
combination means "did my write land?" is only answerable by reading back, and
a single-shot delete would be able to leave the deactivation applied and a
neighbour's pointer stale. Since `patches_for_removal` plans from current state
towards a fixed end state, simply re-planning emits precisely the patches that
did not land, so the loop converges instead of re-doing work. Only exhausting
the attempts raises `PointVersionConflictError` (`409`).
Deleting an already-inactive point falls out of the same machinery rather than
needing a special case: its neighbours were relinked by the first delete, so the
plan is empty and the call is a no-op success — not a `404`, and not a second
relink.
"""
import uuid
from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from time import perf_counter
import structlog
from src.application.points.errors import PointVersionConflictError
from src.application.points.point import Point, PointNotFoundError
from src.application.points.relinking import (
neighbour_ids,
patch_for_deactivation,
patches_for_removal,
)
from src.application.ports.point_repository import PayloadPatch, PointRepository
logger = structlog.get_logger(__name__)
# Three plan-apply rounds, then a final verifying plan. Each round only re-emits
# what a concurrent writer displaced, so a caller that legitimately needs more
# than this is contending on the same points continuously and deserves the
# `409` rather than an unbounded loop inside a request.
_MAX_ATTEMPTS = 3
# One sweep page. Matches ADR-0002's 100-operation batch cap, so a page of
# points is always expressible as a single `points/batch` request.
_SWEEP_BATCH_SIZE = 100
# A hard ceiling on sweep rounds, so a file being concurrently re-ingested while
# it is deleted cannot spin here for the life of the request.
_MAX_SWEEP_ROUNDS = 1_000
def _elapsed_ms(started: float) -> float:
"""Wall-clock milliseconds since `started` (ADR-0011's `duration_ms`).
Worth carrying on these events even though the relinking itself is O(1):
what a delete actually spends is Qdrant round trips, and the whole-file
sweep spends a number of them proportional to the file's length. Timing the
operation is the only way to tell a slow store from a contended one, which
`rounds` on the same event then disambiguates.
"""
return round((perf_counter() - started) * 1000, 2)
async def soft_delete_point(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
point_id: uuid.UUID,
actor: str,
) -> Point:
"""Deactivate one point and relink its neighbours around the gap.
Returns the point as it now stands. Raises `PointNotFoundError` (`404`) if
it is not this tenant's — the same non-disclosure rule the read paths
follow — or `PointVersionConflictError` (`409`) if concurrent writers keep
displacing the plan.
"""
started = perf_counter()
rounds = 0
point, patches = await _plan_removal(
repository, tenant_id=tenant_id, point_id=point_id, actor=actor
)
for _ in range(_MAX_ATTEMPTS):
if not patches:
break
await repository.apply_patches(tenant_id=tenant_id, patches=patches)
rounds += 1
# The next plan doubles as verification: anything that did not land is
# still missing from the end state and comes back as a patch.
point, patches = await _plan_removal(
repository, tenant_id=tenant_id, point_id=point_id, actor=actor
)
if patches:
logger.warning(
"points.soft_delete.conflict",
tenant_id=str(tenant_id),
point_id=str(point_id),
file_id=str(point.file_id),
unsettled_points=[str(patch.point_id) for patch in patches],
rounds=rounds,
duration_ms=_elapsed_ms(started),
)
raise PointVersionConflictError(
f"point {point_id} could not be soft-deleted under concurrent modification"
)
if rounds:
logger.info(
"points.soft_deleted",
tenant_id=str(tenant_id),
point_id=str(point_id),
file_id=str(point.file_id),
version=point.version,
actor=actor,
# `rounds` is 1 unless a concurrent writer forced a re-plan, so a
# rising value here is contention, not slow relinking.
rounds=rounds,
duration_ms=_elapsed_ms(started),
)
return point
async def soft_delete_file_points(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
actor: str,
) -> int:
"""Deactivate every active point of one file, in batches.
No relinking: the whole file leaves the sequence at once, so no surviving
active point can be left pointing at a deactivated one, and the chain is
preserved intact for whoever reads the deleted file later.
Returns how many points were active when the sweep reached them. Each round
re-lists from the start rather than paging with a cursor — deactivated
points drop straight out of the default listing, so the listing itself is
the progress check, and a round that attempts the exact same ids as the one
before it made no progress and raises `PointVersionConflictError`.
"""
started = perf_counter()
swept: set[uuid.UUID] = set()
previous_attempt: frozenset[uuid.UUID] = frozenset()
rounds = 0
for _ in range(_MAX_SWEEP_ROUNDS):
page = await repository.list_by_file(
tenant_id=tenant_id, file_id=file_id, limit=_SWEEP_BATCH_SIZE
)
if not page.points:
if swept:
logger.info(
"points.file_soft_deleted",
tenant_id=str(tenant_id),
file_id=str(file_id),
points_soft_deleted=len(swept),
actor=actor,
# Two Qdrant round trips per round, so this is the delete
# path whose cost tracks the size of the file.
rounds=rounds,
duration_ms=_elapsed_ms(started),
)
return len(swept)
attempt = frozenset(point.point_id for point in page.points)
if attempt == previous_attempt:
logger.warning(
"points.file_soft_delete.conflict",
tenant_id=str(tenant_id),
file_id=str(file_id),
unsettled_points=[str(point_id) for point_id in sorted(attempt, key=str)],
rounds=rounds,
duration_ms=_elapsed_ms(started),
)
raise PointVersionConflictError(
f"file {file_id} could not be soft-deleted under concurrent modification"
)
previous_attempt = attempt
now = datetime.now(UTC)
await repository.apply_patches(
tenant_id=tenant_id,
patches=[patch_for_deactivation(point, actor=actor, now=now) for point in page.points],
)
swept |= attempt
rounds += 1
logger.warning(
"points.file_soft_delete.conflict",
tenant_id=str(tenant_id),
file_id=str(file_id),
reason="sweep_rounds_exhausted",
rounds=rounds,
duration_ms=_elapsed_ms(started),
)
raise PointVersionConflictError(f"file {file_id} still had active points after the sweep")
async def _plan_removal(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
point_id: uuid.UUID,
actor: str,
) -> tuple[Point, tuple[PayloadPatch, ...]]:
point = await repository.get(tenant_id=tenant_id, point_id=point_id)
if point is None:
raise PointNotFoundError(f"point {point_id} not found")
neighbours = await _load_neighbours(repository, tenant_id=tenant_id, point=point)
_warn_on_missing_neighbours(point, neighbours, tenant_id=tenant_id)
patches = patches_for_removal(point, neighbours, actor=actor, now=datetime.now(UTC))
return point, patches
async def _load_neighbours(
repository: PointRepository, *, tenant_id: uuid.UUID, point: Point
) -> dict[uuid.UUID, Point]:
wanted: Sequence[uuid.UUID] = neighbour_ids(point)
if not wanted:
return {}
found = await repository.get_many(tenant_id=tenant_id, point_ids=wanted)
return {neighbour.point_id: neighbour for neighbour in found}
def _warn_on_missing_neighbours(
point: Point, neighbours: Mapping[uuid.UUID, Point], *, tenant_id: uuid.UUID
) -> None:
"""A pointer naming a point that is not there means the chain is already broken.
Worth a log line rather than an exception: the delete can still complete the
part of the relink that does exist, and refusing would leave the caller with
a point it cannot remove through any endpoint.
"""
missing = [pointer for pointer in neighbour_ids(point) if pointer not in neighbours]
if missing:
logger.warning(
"points.relink.neighbour_missing",
tenant_id=str(tenant_id),
point_id=str(point.point_id),
file_id=str(point.file_id),
missing_neighbours=[str(pointer) for pointer in missing],
)

View File

@@ -0,0 +1,19 @@
"""Mutation failures for the `/v1/points` write paths (ADR-0002).
No HTTP knowledge here — `src/api/errors.py` owns the status mapping. Absence
lives on `PointNotFoundError` in `point.py`, next to the model whose read paths
raise it; this module is for the failures only a *write* can produce.
"""
class PointVersionConflictError(Exception):
"""A version-guarded write could not be landed against a moving target.
Raised when the service has re-read, recomputed, and re-applied its patches
the allowed number of times and the desired state still has not settled —
something else is writing the same points concurrently. Maps to `409`.
This is not "the guard fired once": a single stale guard is expected and is
retried, because Qdrant reports success for a filtered `set_payload` that
matched nothing. It means the retries were exhausted.
"""

View File

@@ -0,0 +1,203 @@
"""The one caller-facing entry point for indexing embedded chunks (ADR-0001, ADR-0017).
`index_chunks` is the only version of this step callers should reach for. It
owns the whole composition a correct upsert needs:
- building ADR-0001's payload for every chunk, with `tenant_id`/`domain` taken
from server-derived context;
- offloading that (and the per-chunk content hashing) to a thread, since it is
blocking CPU work (ADR-0017);
- batching at `QDRANT_UPSERT_BATCH_SIZE` inside ADR-0001's 64-256 band;
- bounding in-flight batches with an `asyncio.Semaphore` rather than an
unbounded `gather` (ADR-0017);
- running the soft-delete sweep for a shortened file **only after every batch
has succeeded**.
That last ordering is the point, not an implementation detail — see
`_deactivate_stale` below. `build_chunk_payload` and `_batches` stay internal;
pushing that composition onto every call site is exactly the obligation a deep
module absorbs once (CLAUDE.md, "prefer deep modules").
"""
import asyncio
import uuid
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import UTC, datetime
from anyio import CapacityLimiter, to_thread
from src.application.ingestion.errors import PointIndexingError
from src.application.ingestion.models import EmbeddedChunk
from src.application.points.models import ChunkPoint
from src.application.points.payload import build_chunk_payload
from src.application.ports.embedding import DenseEmbedder, SparseEmbedder
from src.application.ports.point_storage import PointStorage
from src.config import QdrantSettings
@dataclass(frozen=True)
class IndexingResult:
"""What one indexing pass wrote.
`points_upserted` counts points written, not points *created* — a
deterministic-id upsert cannot distinguish an insert from an overwrite, so
the ingestion job reports this as `points_created` and leaves
`points_updated` at zero rather than guessing.
"""
points_upserted: int
points_soft_deleted: int
def _embedding_model_version(
dense_embedders: Sequence[DenseEmbedder], sparse_embedder: SparseEmbedder
) -> str:
"""Compose the `embedding_model_version` payload value (ADR-0001).
Sorted so the string is stable regardless of the order the embedders were
wired in — an unstable value would make "which chunks need re-embedding?"
unanswerable, which is the field's only reason to exist.
"""
versions = sorted(
[embedder.model_version for embedder in dense_embedders] + [sparse_embedder.model_version]
)
return "+".join(versions)
def _batches(points: Sequence[ChunkPoint], size: int) -> list[Sequence[ChunkPoint]]:
return [points[i : i + size] for i in range(0, len(points), size)]
def _build_points(
embedded: Sequence[EmbeddedChunk],
*,
tenant_id: uuid.UUID,
domain: str,
file_id: uuid.UUID,
source_filename: str,
source_type: str,
actor: str,
embedding_model_version: str,
indexed_at: datetime,
) -> list[ChunkPoint]:
"""Blocking: hashes every chunk's content. Always called through a thread."""
return [
ChunkPoint(
point_id=item.chunk.chunk_id,
dense=item.dense,
sparse=item.sparse,
payload=build_chunk_payload(
item.chunk,
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
source_filename=source_filename,
source_type=source_type,
actor=actor,
embedding_model_version=embedding_model_version,
indexed_at=indexed_at,
),
)
for item in embedded
]
async def _upsert_bounded(
storage: PointStorage, batch: Sequence[ChunkPoint], *, semaphore: asyncio.Semaphore
) -> None:
async with semaphore:
try:
await storage.upsert_points(batch)
except Exception as exc:
raise PointIndexingError(f"upserting {len(batch)} points failed: {exc}") from exc
async def _deactivate_stale(
storage: PointStorage,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
from_chunk_index: int,
actor: str,
deleted_at: datetime,
) -> int:
"""Soft-delete points left over from a longer previous version of this file.
Chunk indices are contiguous from 0, so "index >= the new chunk count" is
exactly the set of points the new version no longer produces.
This runs **only after every upsert has succeeded**, and that ordering is
what keeps a failed attempt from damaging a working index. ADR-0001's
deterministic point ids mean a re-ingestion overwrites in place, so literal
atomic replacement is not available; what *is* guaranteed is that a failed
attempt never removes content (it can only leave a prefix updated), and that
a retry converges to the correct state. See ADR-0017.
"""
try:
return await storage.deactivate_points_from_index(
tenant_id=tenant_id,
file_id=file_id,
from_chunk_index=from_chunk_index,
deleted_at=deleted_at,
updated_by=actor,
)
except Exception as exc:
raise PointIndexingError(f"soft-deleting stale points failed: {exc}") from exc
async def index_chunks(
embedded: Sequence[EmbeddedChunk],
*,
storage: PointStorage,
tenant_id: uuid.UUID,
domain: str,
file_id: uuid.UUID,
source_filename: str,
source_type: str,
actor: str,
dense_embedders: Sequence[DenseEmbedder],
sparse_embedder: SparseEmbedder,
settings: QdrantSettings,
thread_limiter: CapacityLimiter,
) -> IndexingResult:
"""Upsert every embedded chunk as a tenant-scoped point, then sweep leftovers.
Raises `PointIndexingError` (502) if any batch or the sweep fails.
"""
if not embedded:
return IndexingResult(points_upserted=0, points_soft_deleted=0)
indexed_at = datetime.now(UTC)
points = await to_thread.run_sync(
lambda: _build_points(
embedded,
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
source_filename=source_filename,
source_type=source_type,
actor=actor,
embedding_model_version=_embedding_model_version(dense_embedders, sparse_embedder),
indexed_at=indexed_at,
),
limiter=thread_limiter,
)
semaphore = asyncio.Semaphore(settings.upsert_concurrency)
await asyncio.gather(
*(
_upsert_bounded(storage, batch, semaphore=semaphore)
for batch in _batches(points, settings.upsert_batch_size)
)
)
soft_deleted = await _deactivate_stale(
storage,
tenant_id=tenant_id,
file_id=file_id,
from_chunk_index=len(points),
actor=actor,
deleted_at=indexed_at,
)
return IndexingResult(points_upserted=len(points), points_soft_deleted=soft_deleted)

View File

@@ -0,0 +1,29 @@
"""Domain models for Qdrant points (ADR-0001).
Deliberately free of the `qdrant_client` SDK: `src/infrastructure/qdrant/`
converts these to `PointStruct`/`models.SparseVector` at upsert time
(ADR-0015 — application code and ports carry no infrastructure imports).
"""
import uuid
from pydantic import BaseModel, Field
from src.application.ingestion.models import SparseVector
class ChunkPoint(BaseModel):
"""One chunk, ready to upsert: its id, its named vectors, and its payload.
`point_id` is the chunk's deterministic UUIDv5 (`chunk_id_for`), so
re-ingesting a file overwrites its points rather than duplicating them
(ADR-0001).
`dense` is keyed by named-vector name (`dense_nomic`, `dense_openai`).
`late_interaction` is absent — ADR-0017 does not compute it at ingest.
"""
point_id: uuid.UUID
dense: dict[str, list[float]]
sparse: SparseVector
payload: dict[str, object] = Field(default_factory=dict)

View File

@@ -0,0 +1,70 @@
"""Builds ADR-0001's point payload from a chunk plus its ingestion context.
Internal to `src/application/points/` — callers use `index_chunks`, which owns
composing this with batching and the deactivation sweep. Exported for its own
unit tests, not as a surface to build payloads by hand.
"""
import uuid
from datetime import datetime
from hashlib import sha256
from src.application.ingestion.models import Chunk
def _optional_id(value: uuid.UUID | None) -> str | None:
return str(value) if value is not None else None
def build_chunk_payload(
chunk: Chunk,
*,
tenant_id: uuid.UUID,
domain: str,
file_id: uuid.UUID,
source_filename: str,
source_type: str,
actor: str,
embedding_model_version: str,
indexed_at: datetime,
) -> dict[str, object]:
"""Return ADR-0001's payload for one chunk.
`tenant_id` and `domain` are passed in from the server-derived `AuthContext`
and the validated request — never from anything the client could assert as
authority (ADR-0002's non-negotiable isolation rule).
UUIDs are serialized as strings because the `tenant_id`/`domain`/`file_id`/
`previous_chunk_id`/`next_chunk_id` payload indexes are *keyword* indexes;
a native UUID would not match a keyword filter.
**Known gap — `version` is always written as `1`.** ADR-0002 uses this field
for optimistic concurrency between ingestion and manual `/v1/points` edits,
which needs a read-check-write (one read per point). Ingestion is
authoritative for its own file today, so writing `1` is safe until
`/v1/points` exists; plan 002 owns closing this.
"""
timestamp = indexed_at.isoformat()
return {
"tenant_id": str(tenant_id),
"domain": domain,
"file_id": str(file_id),
"chunk_id": str(chunk.chunk_id),
"content": chunk.content,
"content_type": chunk.content_type.value,
"source_filename": source_filename,
"source_type": source_type,
"order_id": chunk.order_id,
"chunk_index": chunk.chunk_index,
"previous_chunk_id": _optional_id(chunk.previous_chunk_id),
"next_chunk_id": _optional_id(chunk.next_chunk_id),
"is_active": True,
"deleted_at": None,
"created_at": timestamp,
"updated_at": timestamp,
"created_by": actor,
"updated_by": actor,
"version": 1,
"content_hash": sha256(chunk.content.encode("utf-8")).hexdigest(),
"embedding_model_version": embedding_model_version,
}

View File

@@ -0,0 +1,126 @@
"""The caller-facing point model for `/v1/points` (ADR-0001, ADR-0002).
`ChunkPoint` in `models.py` is the *write* shape ingestion upserts: an id, its
named vectors, and an opaque payload dict. This module is the *read/edit* shape
the `/v1/points` surface works in, where the payload's individual fields matter
and the distinction between what a caller may write and what the server owns is
a security boundary rather than a convention.
That split is the reason this is a model and not a dict. ADR-0002's isolation
rule ("never accepted as client-supplied input") and its optimistic-concurrency
guard both fail open if a caller can smuggle `tenant_id` or `version` through a
payload update, so the writable field set is enumerated in one place here and
every write path validates against it.
"""
import uuid
from collections.abc import Mapping
from datetime import datetime
from typing import Self
from pydantic import BaseModel, ConfigDict, Field
# Fields the server derives and a caller may never set, patch, or override.
# `tenant_id` is authority, `version` is the concurrency guard, `chunk_index`
# derives the point id, and the rest are provenance the server timestamps.
SERVER_OWNED_FIELDS: frozenset[str] = frozenset(
{
"tenant_id",
"version",
"chunk_index",
"chunk_id",
"is_active",
"deleted_at",
"created_at",
"created_by",
"updated_at",
"updated_by",
"content_hash",
"embedding_model_version",
}
)
# Fields a caller may supply on create, replace, or payload patch. `order_id`
# is writable on create but moves only through `PATCH /v1/points/{id}/order`
# afterwards, because a bare `order_id` write would not relink neighbours.
CALLER_WRITABLE_FIELDS: frozenset[str] = frozenset(
{
"content",
"content_type",
"domain",
"source_filename",
"source_type",
"order_id",
"previous_chunk_id",
"next_chunk_id",
}
)
class Point(BaseModel):
"""One Qdrant point, read back with its ADR-0001 payload fields typed.
Vectors are deliberately absent: ADR-0008 returns them only when explicitly
requested, and every read path that does not ask for them should not pay to
deserialize them.
"""
model_config = ConfigDict(frozen=True)
point_id: uuid.UUID
tenant_id: uuid.UUID
domain: str
file_id: uuid.UUID
chunk_id: uuid.UUID
content: str
content_type: str
source_filename: str
source_type: str
order_id: float
chunk_index: int
previous_chunk_id: uuid.UUID | None = None
next_chunk_id: uuid.UUID | None = None
is_active: bool = True
deleted_at: datetime | None = None
created_at: datetime
updated_at: datetime
created_by: str
updated_by: str
version: int
content_hash: str
embedding_model_version: str
# Only populated when the caller explicitly asked for vectors.
vectors: dict[str, object] | None = Field(default=None)
@classmethod
def from_payload(
cls,
point_id: uuid.UUID,
payload: Mapping[str, object],
*,
vectors: Mapping[str, object] | None = None,
) -> Self:
"""Build a `Point` from a raw Qdrant payload dict.
Lives here rather than in the Qdrant adapter so the payload field names
are declared once, next to the model that mirrors them. The adapter
stays responsible for talking to the SDK, not for knowing ADR-0001's
schema twice.
"""
return cls.model_validate({**payload, "point_id": point_id, "vectors": vectors})
class PointNotFoundError(LookupError):
"""No such point *within the requesting tenant*.
Routes map this to `404`, never `403` — a caller must not be able to probe
for the existence of another tenant's point ids (ADR-0016). The error
deliberately carries no hint about which of the two cases occurred.
"""

View File

@@ -0,0 +1,115 @@
"""Read paths for `/v1/points` (ADR-0002, ADR-0008).
The caller-facing entry points for point reads. Routers call these; they never
touch the `PointRepository` directly, and never build a filter.
This module is thin on purpose but not empty, and the two things it does own are
exactly the ones a route would otherwise get wrong:
- **`tenant_id` always comes from the caller's `AuthContext`.** Every function
takes it as a required keyword and hands it to the repository. Nothing here
reads a tenant from a query string or body.
- **A keyword query is Persian-normalized before it reaches the index.**
Ingestion letter-folds chunk content (`normalize_persian_text`, ADR-0018), so
stored text contains Persian yeh/keheh. A query typed on an Arabic keyboard
carries U+064A/U+0643 and would match nothing at all — a silent empty result,
not an error. Folding the query the same way is what makes the two comparable.
Reads emit no log events. The request middleware already records every call, and
ADR-0011 reserves `INFO` for lifecycle events rather than per-read volume;
mutations get their own events when those paths land.
"""
import uuid
from src.application.ingestion.normalization import normalize_persian_text
from src.application.points.point import Point, PointNotFoundError
from src.application.ports.point_repository import PointPage, PointRepository
async def get_point(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
point_id: uuid.UUID,
with_vectors: bool = False,
) -> Point:
"""One point, or `PointNotFoundError` if it is not this tenant's.
Raises rather than returning `None` so a route cannot forget the check and
serve `200 null`. "Absent" and "another tenant's" are the same outcome by
design (ADR-0016: cross-tenant access is `404`, never `403`).
"""
point = await repository.get(tenant_id=tenant_id, point_id=point_id, with_vectors=with_vectors)
if point is None:
raise PointNotFoundError(f"point {point_id} not found")
return point
async def list_file_points(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
limit: int,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointPage:
"""One file's points in display (`order_id`) order.
An unknown or foreign `file_id` yields an empty page rather than an error:
the two are indistinguishable to the caller, which is the same
non-disclosure property `get_point` gets from raising.
"""
return await repository.list_by_file(
tenant_id=tenant_id,
file_id=file_id,
limit=limit,
cursor=cursor,
include_inactive=include_inactive,
)
async def count_points(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> int:
return await repository.count(
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
async def search_points(
repository: PointRepository,
*,
tenant_id: uuid.UUID,
query: str,
limit: int,
cursor: str | None = None,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointPage:
"""Keyword match on `content`, within this tenant.
**Not semantic retrieval.** Qdrant's full-text index filters rather than
scores, so results carry no relevance ranking and their order is
unspecified. Ranked retrieval is ADR-0003's hybrid path in plan 003; this
function must not grow a semantic mode (ADR-0002).
"""
return await repository.keyword_search(
tenant_id=tenant_id,
query=normalize_persian_text(query),
limit=limit,
cursor=cursor,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)

View File

@@ -0,0 +1,135 @@
"""Adjacency-pointer maintenance for a point leaving a file's sequence.
ADR-0001 keeps `previous_chunk_id`/`next_chunk_id` on every point so ADR-0003's
context-window expansion can walk a file in O(1) steps. ADR-0002 makes keeping
them correct an obligation of every operation that changes a point's position:
a partial relink is a defect, not a degraded-but-acceptable outcome.
The function below is the primitive that obligation reduces to. It is pure, and
it is written as **"what is still missing between the state I just read and the
state I want"** rather than "the patches a delete implies". That framing is what
makes the caller's retry loop correct: re-planning after a partial apply emits
exactly the patches that did not land, and re-planning after a completed delete
emits nothing at all. The three cases the plan calls out — a normal delete, a
second delete of an already-inactive point, and recovery from a half-applied
batch — are then one code path instead of three.
Note what is deliberately *not* patched: the departing point's own
`previous_chunk_id`/`next_chunk_id`. Nothing active points at it once its
neighbours are relinked, so those pointers are unreachable rather than stale,
and leaving them records where the point sat — which is what a later restore or
an audit reader would need. `src/application/points/deletion.py` relies on that
when it re-plans: the departing point's pointers are the only surviving record
of which two neighbours have to be joined.
"""
import uuid
from collections.abc import Mapping
from datetime import datetime
from src.application.points.point import Point
from src.application.ports.point_repository import PayloadPatch
def _optional_id(value: uuid.UUID | None) -> str | None:
return str(value) if value is not None else None
def _provenance(point: Point, *, actor: str, now: datetime) -> dict[str, object]:
"""The fields every mutation writes: who, when, and the next version.
Bumping `version` on a relinked *neighbour* is intentional. The neighbour's
payload really did change, so a concurrent editor holding the old version
must get a `409` rather than overwrite the pointer we just fixed.
"""
return {
"updated_at": now.isoformat(),
"updated_by": actor,
"version": point.version + 1,
}
def neighbour_ids(point: Point) -> tuple[uuid.UUID, ...]:
"""The ids `patches_for_removal` needs loaded, skipping the nulls."""
return tuple(
pointer for pointer in (point.previous_chunk_id, point.next_chunk_id) if pointer is not None
)
def patches_for_removal(
point: Point,
neighbours: Mapping[uuid.UUID, Point],
*,
actor: str,
now: datetime,
) -> tuple[PayloadPatch, ...]:
"""The patches still needed to remove `point` from its file's sequence.
Returns an empty tuple when the removal is already complete, which the
caller reads as both "converged" and "this was a no-op".
A neighbour absent from `neighbours` is skipped rather than patched blind:
its id came from the departing point's payload, so a missing one means the
chain was already broken, and inventing a patch for a point that is not
there would not fix it. The caller logs that case.
"""
patches: list[PayloadPatch] = []
if point.is_active:
patches.append(
PayloadPatch(
point_id=point.point_id,
payload={
"is_active": False,
"deleted_at": now.isoformat(),
**_provenance(point, actor=actor, now=now),
},
expected_version=point.version,
)
)
previous = neighbours.get(point.previous_chunk_id) if point.previous_chunk_id else None
if previous is not None and previous.next_chunk_id != point.next_chunk_id:
patches.append(
PayloadPatch(
point_id=previous.point_id,
payload={
"next_chunk_id": _optional_id(point.next_chunk_id),
**_provenance(previous, actor=actor, now=now),
},
expected_version=previous.version,
)
)
following = neighbours.get(point.next_chunk_id) if point.next_chunk_id else None
if following is not None and following.previous_chunk_id != point.previous_chunk_id:
patches.append(
PayloadPatch(
point_id=following.point_id,
payload={
"previous_chunk_id": _optional_id(point.previous_chunk_id),
**_provenance(following, actor=actor, now=now),
},
expected_version=following.version,
)
)
return tuple(patches)
def patch_for_deactivation(point: Point, *, actor: str, now: datetime) -> PayloadPatch:
"""Deactivate one point without touching any pointer.
Used by the whole-file sweep, where every point in the file leaves at once:
no active point survives to dangle, so there is no neighbour to relink and
the chain stays intact for a later reader of the deactivated file.
"""
return PayloadPatch(
point_id=point.point_id,
payload={
"is_active": False,
"deleted_at": now.isoformat(),
**_provenance(point, actor=actor, now=now),
},
expected_version=point.version,
)

View File

@@ -19,6 +19,14 @@ class DenseEmbedder(Protocol):
"""
name: str
model_version: str
"""Identifies the model that produced these vectors (ADR-0001).
Written into every point's `embedding_model_version` payload field, which
exists so a future model swap can tell which chunks need re-embedding. The
embedder is what knows this, so it is reported here rather than
reconstructed from configuration at the call site.
"""
async def embed_batch(self, texts: Sequence[str]) -> list[list[float]]:
"""Return one vector per input text, same order. Raises `EmbedderError`
@@ -37,6 +45,12 @@ class SparseEmbedder(Protocol):
"""
name: str
model_version: str
"""Identifies the analyzer/parameters that produced these vectors.
Same purpose as `DenseEmbedder.model_version`; for BM25 the "model" is the
analyzer choice (ADR-0005), which is equally a re-embedding trigger.
"""
def embed_batch(self, texts: Sequence[str], *, query: bool = False) -> list[SparseVector]:
"""Return one sparse vector per input text, same order.

View File

@@ -0,0 +1,131 @@
"""The point read/edit port for `/v1/points` (ADR-0002, ADR-0015).
Separate from `PointStorage`, which stays exactly the two bulk operations
ingestion performs. Reads, single-point edits, reordering, and keyword search
have a different caller, a different failure vocabulary, and a different
tenant-filter obligation, so they get their own port rather than accreting onto
the ingestion one.
`tenant_id` is a required keyword argument on **every** method. That is not
style: ADR-0002's isolation rule has to hold on every code path that touches the
collection, and an optional tenant filter is one forgotten argument away from a
cross-tenant read. Making it required moves that from a review question to a
type error.
`src/infrastructure/qdrant/point_repository.py` is the production adapter;
`tests.fakes.FakePointRepository` is the test double.
"""
import uuid
from collections.abc import Sequence
from typing import Protocol
from pydantic import BaseModel
from src.application.points.point import Point
class PointPage(BaseModel):
"""One page of points plus the cursor that continues it.
`next_cursor` is opaque to callers and encoded by the adapter: ordered
scrolls and keyword searches paginate by different Qdrant mechanisms, and
neither is a plain integer offset. `None` means the listing is exhausted.
"""
points: tuple[Point, ...]
next_cursor: str | None = None
class PayloadPatch(BaseModel):
"""Set these payload fields on one point, optionally guarded by `version`.
When `expected_version` is set, the adapter attaches it to the operation's
filter, so a concurrent write that has already moved the version on means
this patch matches nothing rather than clobbering it. The guard is what
makes a lost update impossible; detecting that it fired is the service's
job (see `apply_patches`).
"""
point_id: uuid.UUID
payload: dict[str, object]
expected_version: int | None = None
class PointRepository(Protocol):
async def get(
self, *, tenant_id: uuid.UUID, point_id: uuid.UUID, with_vectors: bool = False
) -> Point | None:
"""One point, or `None` if it does not exist *under this tenant*.
The two cases are deliberately indistinguishable — the route maps both
to `404` so a caller cannot probe for another tenant's point ids.
"""
...
async def get_many(
self, *, tenant_id: uuid.UUID, point_ids: Sequence[uuid.UUID]
) -> tuple[Point, ...]:
"""The subset of `point_ids` that exists under this tenant.
Order is not guaranteed and missing ids are silently absent: callers are
neighbour-relinking and batch precondition checks, both of which match
on id rather than position.
"""
...
async def list_by_file(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
limit: int,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointPage:
"""One file's points in `order_id` order (ADR-0008's `scroll`).
Scoped to a single file because the cursor is an `order_id` value, and
`order_id` is only unique within a file.
"""
...
async def count(
self,
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> int: ...
async def keyword_search(
self,
*,
tenant_id: uuid.UUID,
query: str,
limit: int,
cursor: str | None = None,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointPage:
"""Full-text payload match on `content`, plus structured filters.
Keyword matching, **not** semantic retrieval (ADR-0002). Qdrant's
full-text index is a filter, not a scorer, so results carry no relevance
ranking and their order is unspecified.
"""
...
async def apply_patches(self, *, tenant_id: uuid.UUID, patches: Sequence[PayloadPatch]) -> None:
"""Apply every patch in one Qdrant `points/batch` request.
Qdrant has no multi-point transaction, so this is not atomic and does
not pretend to be. ADR-0002's all-or-nothing rule is implemented one
layer up as validate-every-precondition-then-apply; the per-patch
`expected_version` guard here is what makes the residual window safe,
turning a lost update into a no-op the service can detect rather than a
silent clobber.
"""
...

View File

@@ -0,0 +1,45 @@
"""The point-storage port (ADR-0001, ADR-0015).
`src/infrastructure/qdrant/points.py` is the production adapter; tests use a
hand-written fake (ADR-0016). Application code depends on this Protocol, not on
the `qdrant_client` SDK.
Deliberately narrow: exactly the two operations ingestion performs. Reads,
single-point edits, reordering, and keyword search are plan 002's `/v1/points`
surface and belong on a port of their own rather than accreting here.
"""
import uuid
from collections.abc import Sequence
from datetime import datetime
from typing import Protocol
from src.application.points.models import ChunkPoint
class PointStorage(Protocol):
async def upsert_points(self, points: Sequence[ChunkPoint]) -> None:
"""Upsert one batch of points.
Callers own batching and concurrency bounding (ADR-0017's
`upsert_concurrency` semaphore), not this Protocol — the same division
`DenseEmbedder.embed_batch` uses.
"""
...
async def deactivate_points_from_index(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
from_chunk_index: int,
deleted_at: datetime,
updated_by: str,
) -> int:
"""Soft-delete this file's points at or past `from_chunk_index`.
Sets `is_active=false`/`deleted_at` rather than removing the points
(ADR-0002: delete is soft by default). Tenant-filtered — a `file_id`
alone is never sufficient authority. Returns how many points matched.
"""
...

View File

@@ -0,0 +1,9 @@
"""Operator-run tenant provisioning (ADR-0008, ADR-0009)."""
from src.application.tenants.provisioning import (
DEFAULT_SCOPES,
ProvisionResult,
provision_tenant,
)
__all__ = ["DEFAULT_SCOPES", "ProvisionResult", "provision_tenant"]

View File

@@ -0,0 +1,129 @@
"""Provision a tenant, its first API key, and its domains (ADR-0008, ADR-0009).
Nothing in the HTTP surface can bootstrap a tenant: every `/v1` route needs a
key, and a key can only exist once a tenant does. That chicken-and-egg is why
this is an operator-run deployment step (`src/cli/provision_tenant.py`) rather
than an endpoint — the same reasoning that keeps `alembic upgrade head` and
`qdrant_bootstrap` off the request path.
This is the package's only caller-facing entry point. It owns the whole
composition — tenant reuse-or-create, key generation and hashing, domain
registration, and the single transaction the three share — so a caller cannot
get the order wrong or commit a key whose tenant never landed (CLAUDE.md,
"prefer deep modules"). The plaintext key is returned exactly once and is never
logged (ADR-0011 forbids plaintext keys in logs); only its non-secret
`key_prefix` appears in the event.
"""
import uuid
from dataclasses import dataclass
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.keys import generate_api_key, hash_secret
from src.infrastructure.postgres.repositories import api_keys as api_keys_repo
from src.infrastructure.postgres.repositories import tenant_domains as domains_repo
from src.infrastructure.postgres.repositories import tenants as tenants_repo
logger = structlog.get_logger(__name__)
DEFAULT_SCOPES = (
"files:write",
"domains:read",
"domains:write",
"points:read",
"points:write",
)
@dataclass(frozen=True)
class ProvisionResult:
tenant_id: uuid.UUID
tenant_slug: str
tenant_created: bool
api_key_id: uuid.UUID
api_key_prefix: str
api_key: str
"""The plaintext bearer token. Only ever returned here — never stored, never logged."""
domains_created: tuple[str, ...]
domains_existing: tuple[str, ...]
async def provision_tenant(
sessionmaker: async_sessionmaker[AsyncSession],
*,
slug: str,
name: str | None = None,
key_name: str = "bootstrap",
scopes: tuple[str, ...] = DEFAULT_SCOPES,
domains: tuple[str, ...] = (),
actor_type: str = "backend",
) -> ProvisionResult:
"""Create (or reuse) the tenant, issue a key, and register `domains`.
Re-running with the same `slug` reuses the tenant and its existing domains
rather than failing, so an operator can add a key to a live tenant with the
same command they used to create it. A *new* key is issued on every run —
keys are write-once by construction (only the hash is stored), so there is
nothing to return for an existing one.
"""
key_prefix, secret, full_key = generate_api_key()
async with sessionmaker() as session:
tenant = await tenants_repo.get_by_slug(session, slug)
tenant_created = tenant is None
if tenant is None:
tenant = tenants_repo.create(session, slug=slug, name=name or slug)
# `api_keys.tenant_id` and `tenant_domains.tenant_id` FK to this row
# and the mapped classes carry no ORM relationship for the unit of
# work to order by itself, so the insert has to land first.
await session.flush()
api_key = api_keys_repo.create(
session,
tenant_id=tenant.id,
name=key_name,
key_prefix=key_prefix,
key_hash=hash_secret(secret),
scopes=list(scopes),
actor_type=actor_type,
created_by="cli:provision_tenant",
)
created: list[str] = []
existing: list[str] = []
for domain in domains:
if await domains_repo.get(session, tenant_id=tenant.id, domain=domain) is not None:
existing.append(domain)
continue
domains_repo.create(session, tenant_id=tenant.id, domain=domain, display_name=domain)
created.append(domain)
await session.flush()
tenant_id, api_key_id, tenant_slug = tenant.id, api_key.id, tenant.slug
await session.commit()
if tenant_created:
logger.info("tenant.provisioned", tenant_id=str(tenant_id), tenant_slug=tenant_slug)
for domain in created:
logger.info("domain.created", tenant_id=str(tenant_id), domain=domain)
logger.info(
"api_key.provisioned",
tenant_id=str(tenant_id),
api_key_id=str(api_key_id),
key_prefix=key_prefix,
scopes=list(scopes),
actor_type=actor_type,
)
return ProvisionResult(
tenant_id=tenant_id,
tenant_slug=tenant_slug,
tenant_created=tenant_created,
api_key_id=api_key_id,
api_key_prefix=key_prefix,
api_key=full_key,
domains_created=tuple(created),
domains_existing=tuple(existing),
)

View File

@@ -9,6 +9,8 @@ from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
from src.application.ports.embedding import DenseEmbedder, SparseEmbedder
from src.application.ports.object_storage import ObjectStorage
from src.application.ports.point_repository import PointRepository
from src.application.ports.point_storage import PointStorage
from src.config import Settings
@@ -20,6 +22,8 @@ class AppResources:
minio_client: Minio
qdrant_client: AsyncQdrantClient
object_storage: ObjectStorage
point_storage: PointStorage
point_repository: PointRepository
ingestion_limiter: CapacityLimiter
dense_embedders: Sequence[DenseEmbedder]
sparse_embedder: SparseEmbedder
@@ -46,6 +50,14 @@ def get_object_storage(request: Request) -> ObjectStorage:
return _resources(request).object_storage
def get_point_storage(request: Request) -> PointStorage:
return _resources(request).point_storage
def get_point_repository(request: Request) -> PointRepository:
return _resources(request).point_repository
def get_ingestion_limiter(request: Request) -> CapacityLimiter:
return _resources(request).ingestion_limiter

View File

@@ -20,6 +20,8 @@ from src.infrastructure.minio.storage import MinioObjectStorage
from src.infrastructure.observability.logging import configure_logging
from src.infrastructure.postgres.database import create_engine, create_sessionmaker
from src.infrastructure.qdrant.client import create_client as create_qdrant_client
from src.infrastructure.qdrant.point_repository import QdrantPointRepository
from src.infrastructure.qdrant.points import QdrantPointStorage
logger = structlog.get_logger(__name__)
@@ -58,7 +60,7 @@ def create_lifespan(
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
resolved_settings = settings or Settings()
configure_logging(resolved_settings.logging)
configure_logging(resolved_settings.logging, resolved_settings.app)
# tiktoken fetches its vocabulary over the network on first use, so warm
# it here: a missing vocabulary should fail the process at boot, not the
@@ -77,6 +79,16 @@ def create_lifespan(
logger.info("lifespan.minio.client.created")
qdrant_client = create_qdrant_client(resolved_settings.qdrant)
# No collection DDL here: `ensure_chunks_collection` is a deployment
# step (`python -m src.cli.qdrant_bootstrap`), for the same reason
# ADR-0009 keeps Alembic out of startup and ADR-0012 makes LangGraph's
# `.setup()` a deployment step.
point_storage = QdrantPointStorage(
qdrant_client, collection=resolved_settings.qdrant.collection
)
point_repository = QdrantPointRepository(
qdrant_client, collection=resolved_settings.qdrant.collection
)
logger.info("lifespan.qdrant.client.created")
nomic_settings = resolved_settings.embedding.nomic
@@ -136,6 +148,8 @@ def create_lifespan(
minio_client=minio_client,
qdrant_client=qdrant_client,
object_storage=object_storage,
point_storage=point_storage,
point_repository=point_repository,
ingestion_limiter=ingestion_limiter,
dense_embedders=dense_embedders,
sparse_embedder=sparse_embedder,

1
src/cli/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""Operator entry points that run as deployment steps, not at app startup."""

View File

@@ -0,0 +1,94 @@
"""Create a tenant, issue its first API key, and register its domains.
uv run python -m src.cli.provision_tenant --slug acme --domain fire
A deployment step, like `alembic upgrade head` and `src.cli.qdrant_bootstrap`.
It exists because nothing over HTTP can bootstrap a tenant: every `/v1` route
requires an API key, and a key cannot exist before its tenant does.
The plaintext key is printed to **stdout once** and never stored or logged —
Postgres holds only its SHA-256 hash (ADR-0009), so a lost key is reissued by
re-running this command, not recovered. Structured logs go to stderr/the log
sink and carry only the non-secret `key_prefix` (ADR-0011).
Re-running with the same `--slug` reuses the tenant and any domains it already
has, and issues an additional key.
"""
import argparse
import asyncio
import sys
import structlog
from src.application.tenants import DEFAULT_SCOPES, provision_tenant
from src.config import Settings
from src.infrastructure.observability.logging import configure_logging
from src.infrastructure.postgres.database import create_engine, create_sessionmaker
logger = structlog.get_logger(__name__)
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="python -m src.cli.provision_tenant",
description="Create a tenant, issue an API key, and register domains.",
)
parser.add_argument("--slug", required=True, help="URL-safe tenant identifier, e.g. 'acme'")
parser.add_argument("--name", default=None, help="Display name (defaults to --slug)")
parser.add_argument("--key-name", default="bootstrap", help="Label for the issued API key")
parser.add_argument(
"--scopes",
default=",".join(DEFAULT_SCOPES),
help=f"Comma-separated scopes for the key (default: {','.join(DEFAULT_SCOPES)})",
)
parser.add_argument(
"--domain",
action="append",
default=[],
dest="domains",
help="Domain to register; repeatable. Uploads reject an unregistered domain.",
)
return parser.parse_args(argv)
async def run(argv: list[str] | None = None, settings: Settings | None = None) -> int:
args = _parse_args(argv)
resolved = settings or Settings()
configure_logging(resolved.logging, resolved.app)
scopes = tuple(scope.strip() for scope in args.scopes.split(",") if scope.strip())
if not scopes:
logger.error("tenant.provision.failed", reason="no_scopes")
return 2
engine = create_engine(resolved.postgres)
try:
result = await provision_tenant(
create_sessionmaker(engine),
slug=args.slug,
name=args.name,
key_name=args.key_name,
scopes=scopes,
domains=tuple(args.domains),
)
finally:
await engine.dispose()
# stdout, not the logger: this is the one value the operator must copy, and
# it must never reach a log sink (ADR-0011).
print(f"tenant_id={result.tenant_id}")
print(f"tenant_slug={result.tenant_slug}")
print(f"api_key_id={result.api_key_id}")
print(f"domains={','.join(result.domains_created + result.domains_existing)}")
print(f"api_key={result.api_key}")
print("Store the api_key now -- only its hash is persisted and it cannot be shown again.")
return 0
def main() -> None:
sys.exit(asyncio.run(run()))
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,54 @@
"""Create the `chunks` collection — the Qdrant analogue of `alembic upgrade head`.
uv run python -m src.cli.qdrant_bootstrap
A deployment step, deliberately not part of the FastAPI lifespan: collection
creation is DDL, which ADR-0009 keeps out of application startup for Postgres
and ADR-0012 keeps out of it for LangGraph's `.setup()`. See
`src/infrastructure/qdrant/collection.py` for the full reasoning.
Idempotent and safe to re-run. Exits non-zero if an existing collection
diverges from the pinned schema, rather than leaving a silently degraded
sparse index behind.
"""
import asyncio
import sys
import structlog
from src.config import Settings
from src.infrastructure.observability.logging import configure_logging
from src.infrastructure.qdrant.client import create_client
from src.infrastructure.qdrant.collection import (
CollectionSchemaMismatchError,
ensure_chunks_collection,
)
logger = structlog.get_logger(__name__)
async def bootstrap(settings: Settings | None = None) -> int:
resolved = settings or Settings()
configure_logging(resolved.logging, resolved.app)
client = create_client(resolved.qdrant)
try:
created = await ensure_chunks_collection(client, collection=resolved.qdrant.collection)
except CollectionSchemaMismatchError as exc:
logger.error("qdrant.bootstrap.schema_mismatch", error=str(exc))
return 1
finally:
await client.close()
logger.info(
"qdrant.bootstrap.completed", collection=resolved.qdrant.collection, created=created
)
return 0
def main() -> None:
sys.exit(asyncio.run(bootstrap()))
if __name__ == "__main__":
main()

View File

@@ -109,12 +109,28 @@ class ChunkingSettings(BaseSettings):
class QdrantSettings(BaseSettings):
"""Qdrant connection and bulk-upsert bounds (ADR-0001, ADR-0017).
`collection` names the single shared collection all tenants live in
(ADR-0001); it is deliberately configurable so tests can point at a
disposable one. The vector *dimensions* are not settings -- they are model
facts pinned in `src/infrastructure/qdrant/collection.py`, and changing one
is a re-embedding migration.
`upsert_batch_size` sits inside ADR-0001's 64-256 bulk-upload band, and
`upsert_concurrency` bounds in-flight batches so ingestion issues parallel
streams rather than an unbounded `gather` (ADR-0017).
"""
model_config = SettingsConfigDict(
env_prefix="QDRANT_", extra="ignore", env_file=".env", env_ignore_empty=True
)
url: str = "http://127.0.0.1:6343"
api_key: str | None = None
collection: str = "chunks"
upsert_batch_size: int = 128
upsert_concurrency: int = 4
class NomicEmbeddingSettings(BaseSettings):
@@ -210,15 +226,34 @@ class AppLimitSettings(BaseSettings):
env: str = "local"
readiness_check_timeout_seconds: float = 2.0
# The deployed commit SHA or release tag (ADR-0011, "Bind process-level
# environment context"). Set by CI/CD at build/deploy time -- never
# computed at runtime by shelling out to git, which would fail in a
# container image with no .git directory.
service_version: str = "dev"
class LoggingSettings(BaseSettings):
"""Logging sinks (ADR-0011).
`json_format` controls stdout's renderer only. Production sets it `true`
so stdout is JSON for the container log collector; local development
leaves it `false` for a colored console renderer. `file_path`, when set,
is a second, independent handler that always renders JSON regardless of
`json_format` -- a developer can read a human console while still keeping
a machine-parseable file. Unset in production: stdout/stderr collection is
preferred there over a log file inside the container.
"""
model_config = SettingsConfigDict(
env_prefix="LOG_", extra="ignore", env_file=".env", env_ignore_empty=True
)
level: str = "INFO"
json_format: bool = False
file_path: str | None = None
file_max_bytes: int = 10 * 1024 * 1024
file_backup_count: int = 5
class Settings(BaseSettings):

View File

@@ -86,6 +86,7 @@ class Bm25SparseEmbedder:
name = "sparse"
def __init__(self, settings: SparseEmbeddingSettings) -> None:
self.model_version = f"bm25-{settings.analyzer}"
self._settings = settings
def embed_batch(self, texts: Sequence[str], *, query: bool = False) -> list[SparseVector]:

View File

@@ -52,6 +52,7 @@ class OpenAICompatibleEmbedder:
keep_alive: str | None = None,
) -> None:
self.name = name
self.model_version = model
self._client = client
self._model = model
self._dimensions = dimensions

View File

@@ -1,14 +1,50 @@
"""Logging configuration: structlog + stdlib, dual local sinks (ADR-0011).
Console and an optional file are independent, simultaneous handlers on the
same logger, not a single renderer chosen by a flag -- the same structlog
event fans out to both. The console handler is always human-readable
(`ConsoleRenderer`); the file handler, when enabled via `LOG_FILE_PATH`,
always renders JSON regardless of `LOG_JSON_FORMAT`, so a saved log stays
machine-parseable even when the terminal next to it is not.
`LOG_JSON_FORMAT` controls *stdout's* renderer only: production sets it `true`
so the container log collector gets JSON; local development leaves it `false`
for the colored console. `LOG_FILE_PATH` is expected to be unset in
production -- stdout/stderr collection is preferred there over a log file
inside the container.
"""
import logging
import logging.config
import sys
from collections.abc import Callable
import structlog
from src.config import LoggingSettings
from src.config import AppLimitSettings, LoggingSettings
def configure_logging(settings: LoggingSettings) -> None:
def _bind_environment(settings: AppLimitSettings) -> Callable[..., dict[str, object]]:
"""A static processor, not a contextvar: `env`/`service_version` don't
vary per request, and a contextvar bound before the first request would
be wiped by `RequestIdMiddleware`'s `clear_contextvars()` on that request.
Closing over `settings` at configure time makes every event carry them
instead, regardless of request context (ADR-0011).
"""
def processor(
logger: object, method_name: str, event_dict: dict[str, object]
) -> dict[str, object]:
event_dict["env"] = settings.env
event_dict["service_version"] = settings.service_version
return event_dict
return processor
def configure_logging(settings: LoggingSettings, app_settings: AppLimitSettings) -> None:
shared_processors = [
_bind_environment(app_settings),
structlog.contextvars.merge_contextvars,
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
@@ -27,55 +63,84 @@ def configure_logging(settings: LoggingSettings) -> None:
cache_logger_on_first_use=True,
)
renderer = (
console_renderer = (
structlog.processors.JSONRenderer()
if settings.json_format
else structlog.dev.ConsoleRenderer(colors=True)
)
formatters = {
"console": {
"()": structlog.stdlib.ProcessorFormatter,
"processors": [
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
console_renderer,
],
"foreign_pre_chain": [
structlog.stdlib.ExtraAdder(),
*shared_processors,
],
},
}
handlers: dict[str, dict[str, object]] = {
"console": {
"class": "logging.StreamHandler",
"level": settings.level,
"formatter": "console",
"stream": sys.stdout,
},
}
root_handlers = ["console"]
if settings.file_path is not None:
# File handler always renders JSON, independent of the console
# renderer chosen above -- a saved log stays machine-parseable even
# when stdout is the colored, human-readable renderer.
formatters["file"] = {
"()": structlog.stdlib.ProcessorFormatter,
"processors": [
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
structlog.processors.JSONRenderer(),
],
"foreign_pre_chain": [
structlog.stdlib.ExtraAdder(),
*shared_processors,
],
}
handlers["file"] = {
"class": "logging.handlers.RotatingFileHandler",
"level": settings.level,
"formatter": "file",
"filename": settings.file_path,
"maxBytes": settings.file_max_bytes,
"backupCount": settings.file_backup_count,
}
root_handlers.append("file")
logging.config.dictConfig(
{
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"default": {
"()": structlog.stdlib.ProcessorFormatter,
"processors": [
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
renderer,
],
"foreign_pre_chain": [
structlog.stdlib.ExtraAdder(),
*shared_processors,
],
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": settings.level,
"formatter": "default",
"stream": sys.stdout,
},
},
"formatters": formatters,
"handlers": handlers,
"loggers": {
"": {
"handlers": ["console"],
"handlers": root_handlers,
"level": settings.level,
"propagate": False,
},
"uvicorn": {
"handlers": ["console"],
"handlers": root_handlers,
"level": settings.level,
"propagate": False,
},
"uvicorn.access": {
"handlers": ["console"],
"handlers": root_handlers,
"level": settings.level,
"propagate": False,
},
"sqlalchemy.engine": {
"handlers": ["console"],
"handlers": root_handlers,
"level": "WARNING",
"propagate": False,
},

View File

@@ -4,6 +4,7 @@ from src.infrastructure.postgres.models.ingestion_job import IngestionJob
from src.infrastructure.postgres.models.ingestion_job_event import IngestionJobEvent
from src.infrastructure.postgres.models.source_file import SourceFile
from src.infrastructure.postgres.models.tenant import Tenant
from src.infrastructure.postgres.models.tenant_domain import TenantDomain
__all__ = [
"ApiKey",
@@ -12,4 +13,5 @@ __all__ = [
"IngestionJobEvent",
"SourceFile",
"Tenant",
"TenantDomain",
]

View File

@@ -0,0 +1,52 @@
import uuid
from datetime import datetime
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, String, UniqueConstraint, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from src.infrastructure.postgres.models.base import Base
TENANT_DOMAIN_STATUSES = ("active", "disabled")
class TenantDomain(Base):
"""A domain a tenant is allowed to ingest into (ADR-0009).
Tenants do not share a domain list — one may run 14 insurance lines and
another 6 — so this is a per-tenant table rather than an enum or a global
lookup.
Its purpose is to stop an arbitrary caller-supplied `domain` from silently
creating a new Qdrant partition. `domain` is denormalized into every point's
payload and into `source_files`, and a typo like `fier` for `fire` produces
no error anywhere: the file indexes into a partition retrieval never queries,
so it is invisible rather than failed.
`domain` is the immutable key. Renaming it would mean rewriting every point
payload that carries it, which is a migration, not an edit — `display_name`
is the mutable human-facing label instead.
"""
__tablename__ = "tenant_domains"
__table_args__ = (
UniqueConstraint("tenant_id", "domain", name="uq_tenant_domains_tenant_id_domain"),
CheckConstraint(f"status IN {TENANT_DOMAIN_STATUSES}", name="ck_tenant_domains_status"),
)
id: Mapped[uuid.UUID] = mapped_column(primary_key=True)
tenant_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("tenants.id", ondelete="CASCADE"), index=True
)
domain: Mapped[str] = mapped_column(String(80))
display_name: Mapped[str] = mapped_column(String(200))
status: Mapped[str] = mapped_column(String(20), default="active", server_default="active")
metadata_: Mapped[dict[str, object]] = mapped_column(
"metadata", JSONB, default=dict, server_default="{}"
)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
)
disabled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), default=None)

View File

@@ -1,4 +1,4 @@
"""API-key lookups (ADR-0008, ADR-0009).
"""API-key lookups and issuance (ADR-0008, ADR-0009).
Plain functions over an `AsyncSession` the caller owns. No function here
commits, rolls back, or closes the session (ADR-0012). Secret comparison
@@ -6,6 +6,8 @@ happens in `src/application/auth`, not here — this module only fetches rows
by their non-secret `key_prefix`.
"""
import uuid
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -15,3 +17,29 @@ from src.infrastructure.postgres.models.api_key import ApiKey
async def get_by_prefix(session: AsyncSession, key_prefix: str) -> ApiKey | None:
result = await session.execute(select(ApiKey).where(ApiKey.key_prefix == key_prefix))
return result.scalar_one_or_none()
def create(
session: AsyncSession,
*,
tenant_id: uuid.UUID,
name: str,
key_prefix: str,
key_hash: str,
scopes: list[str],
actor_type: str = "backend",
created_by: str | None = None,
) -> ApiKey:
"""Persist an issued key. The caller hashes the secret; this never sees it."""
api_key = ApiKey(
id=uuid.uuid4(),
tenant_id=tenant_id,
name=name,
key_prefix=key_prefix,
key_hash=key_hash,
scopes=scopes,
actor_type=actor_type,
created_by=created_by,
)
session.add(api_key)
return api_key

View File

@@ -7,6 +7,7 @@ signature error rather than a cross-tenant leak.
"""
import uuid
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -67,3 +68,19 @@ def create(
)
session.add(source_file)
return source_file
def mark_soft_deleted(source_file: SourceFile, *, deleted_at: datetime) -> None:
"""Retire a file: `status='soft_deleted'` plus `deleted_at` (ADR-0009).
Takes the already-loaded row rather than an id, because the caller fetched
it under its tenant filter and re-fetching here would be a second place
that could forget that filter.
Retiring the row matters beyond bookkeeping: `find_active_by_content_hash`
matches only `active` files, so a re-upload of the same bytes after a delete
creates a fresh file and re-ingests it, instead of taking the duplicate path
and returning a file whose points have all been deactivated.
"""
source_file.status = "soft_deleted"
source_file.deleted_at = deleted_at

View File

@@ -0,0 +1,73 @@
"""`tenant_domains` persistence (ADR-0009).
Plain functions over an `AsyncSession` the caller owns. No function here
commits, rolls back, or closes the session (ADR-0012). Every read and write is
tenant-scoped by a required `tenant_id` argument, so a missing filter is a
signature error rather than a cross-tenant leak.
"""
import uuid
from datetime import UTC, datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from src.infrastructure.postgres.models.tenant_domain import TenantDomain
async def get(session: AsyncSession, *, tenant_id: uuid.UUID, domain: str) -> TenantDomain | None:
result = await session.execute(
select(TenantDomain).where(
TenantDomain.tenant_id == tenant_id, TenantDomain.domain == domain
)
)
return result.scalar_one_or_none()
async def list_for_tenant(
session: AsyncSession, *, tenant_id: uuid.UUID, include_disabled: bool = False
) -> list[TenantDomain]:
statement = select(TenantDomain).where(TenantDomain.tenant_id == tenant_id)
if not include_disabled:
statement = statement.where(TenantDomain.status == "active")
result = await session.execute(statement.order_by(TenantDomain.domain))
return list(result.scalars().all())
def create(
session: AsyncSession,
*,
tenant_id: uuid.UUID,
domain: str,
display_name: str,
metadata: dict[str, object] | None = None,
) -> TenantDomain:
tenant_domain = TenantDomain(
id=uuid.uuid4(),
tenant_id=tenant_id,
domain=domain,
display_name=display_name,
metadata_=metadata or {},
)
session.add(tenant_domain)
return tenant_domain
def update_display_name(tenant_domain: TenantDomain, *, display_name: str) -> TenantDomain:
"""`domain` itself is deliberately not updatable.
It is denormalized into every Qdrant point payload and into `source_files`,
so changing the key would mean rewriting all of them — a migration, not an
edit. The label is what callers actually want to change.
"""
tenant_domain.display_name = display_name
return tenant_domain
def set_status(tenant_domain: TenantDomain, *, status: str) -> TenantDomain:
"""Disable/re-enable a domain. Existing points are untouched either way —
disabling blocks new uploads, it is not a delete (ADR-0002).
"""
tenant_domain.status = status
tenant_domain.disabled_at = datetime.now(UTC) if status == "disabled" else None
return tenant_domain

View File

@@ -1,4 +1,4 @@
"""Tenant lookups (ADR-0009).
"""Tenant lookups and creation (ADR-0009).
Plain functions over an `AsyncSession` the caller owns. No function here
commits, rolls back, or closes the session (ADR-0012).
@@ -6,6 +6,7 @@ commits, rolls back, or closes the session (ADR-0012).
import uuid
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from src.infrastructure.postgres.models.tenant import Tenant
@@ -13,3 +14,14 @@ from src.infrastructure.postgres.models.tenant import Tenant
async def get_by_id(session: AsyncSession, tenant_id: uuid.UUID) -> Tenant | None:
return await session.get(Tenant, tenant_id)
async def get_by_slug(session: AsyncSession, slug: str) -> Tenant | None:
result = await session.execute(select(Tenant).where(Tenant.slug == slug))
return result.scalar_one_or_none()
def create(session: AsyncSession, *, slug: str, name: str) -> Tenant:
tenant = Tenant(id=uuid.uuid4(), slug=slug, name=name)
session.add(tenant)
return tenant

View File

@@ -9,10 +9,22 @@ def create_client(settings: QdrantSettings) -> AsyncQdrantClient:
return AsyncQdrantClient(url=settings.url, api_key=settings.api_key)
async def ping(client: AsyncQdrantClient, timeout: float) -> bool:
async def ping(client: AsyncQdrantClient, timeout: float, *, collection: str) -> bool:
"""Whether Qdrant is reachable **and** the `chunks` collection exists.
Reachability alone is not readiness here. The collection is created by a
deployment step (`python -m src.cli.qdrant_bootstrap`, see ADR-0001
"Collection provisioning"), so a process can boot against a healthy Qdrant
that has no collection at all. Without this check that misconfiguration
stays invisible until the first upload fails with a `502` — after the
request has already paid for the MinIO write and the embedding round trips.
This is the Qdrant analogue of an unapplied Alembic migration, and it
belongs in `/readyz` for the same reason: it is a dependency-readiness
condition, not a process-health one.
"""
try:
async with asyncio.timeout(timeout):
await client.get_collections()
return await client.collection_exists(collection)
except Exception:
return False
return True

View File

@@ -0,0 +1,200 @@
"""The `chunks` collection schema and its provisioning (ADR-0001).
**This runs as a deployment step, never at FastAPI startup.** Creating a
collection is DDL, and the project already rules DDL out of the request/boot
path: ADR-0009 requires Alembic for Postgres schema, and ADR-0012 makes
LangGraph's `.setup()` a deployment step. Doing it in the lifespan would also
couple boot to Qdrant being reachable (that is `/readyz`'s job), race across
replicas, and hide a misconfigured collection until traffic arrives.
Entry point for operators: `uv run python -m src.cli.qdrant_bootstrap`.
Two properties of this schema are load-bearing and fail *silently* if wrong,
which is why `ensure_chunks_collection` verifies rather than skips:
- **`sparse` must carry `modifier=IDF`.** `src/infrastructure/embedding/bm25.py`
computes only BM25's term-frequency saturation; Qdrant supplies IDF from
collection-wide statistics. Without the modifier there is no error and no
warning — lexical retrieval just quietly loses its IDF term (ADR-0005).
- **The dense dimensions are pinned**: `dense_nomic` 768,
`dense_openai` 3072 (native, `dimensions` deliberately unset). They are
constants here rather than settings because they are model facts; changing
one is a re-embedding migration, not a config tweak (ADR-0001).
All four named vectors are defined at creation even though `late_interaction`
stays unpopulated until ADR-0003's rerank work (ADR-0017 does not compute it at
ingest). Sparse and multivector fields cannot be added to an existing
collection without recreating it, so deferring them is the one thing this
schema cannot afford.
"""
import structlog
from qdrant_client import AsyncQdrantClient, models
logger = structlog.get_logger(__name__)
DENSE_NOMIC_VECTOR = "dense_nomic"
DENSE_OPENAI_VECTOR = "dense_openai"
SPARSE_VECTOR = "sparse"
LATE_INTERACTION_VECTOR = "late_interaction"
DENSE_NOMIC_DIMENSIONS = 768
DENSE_OPENAI_DIMENSIONS = 3072
# jina-colbert-v2's per-token output dimension (ADR-0005).
LATE_INTERACTION_DIMENSIONS = 128
class CollectionSchemaMismatchError(RuntimeError):
"""An existing collection does not match the schema this code expects.
Raised loudly instead of returning: silently accepting a collection whose
dense size or sparse modifier differs is precisely the failure this
explicit bootstrap step exists to prevent.
"""
def _vectors_config() -> dict[str, models.VectorParams]:
return {
DENSE_NOMIC_VECTOR: models.VectorParams(
size=DENSE_NOMIC_DIMENSIONS, distance=models.Distance.COSINE
),
DENSE_OPENAI_VECTOR: models.VectorParams(
size=DENSE_OPENAI_DIMENSIONS, distance=models.Distance.COSINE
),
# Rerank-only: never independently ANN-searched, so its HNSW graph is
# disabled (m=0), and stored on disk so its larger footprint does not
# degrade dense/sparse query latency (ADR-0001).
LATE_INTERACTION_VECTOR: models.VectorParams(
size=LATE_INTERACTION_DIMENSIONS,
distance=models.Distance.COSINE,
multivector_config=models.MultiVectorConfig(
comparator=models.MultiVectorComparator.MAX_SIM
),
hnsw_config=models.HnswConfigDiff(m=0),
on_disk=True,
),
}
def _sparse_vectors_config() -> dict[str, models.SparseVectorParams]:
return {SPARSE_VECTOR: models.SparseVectorParams(modifier=models.Modifier.IDF)}
# `content`'s full-text index backs ADR-0002's keyword search. The
# `multilingual` tokenizer is the one that segments Persian correctly; `word`
# splits on non-alphanumerics, which mis-handles ZWNJ-joined compounds. No
# stemmer or stopword list is configured: `content` is already letter-folded by
# `normalize_persian_text` at ingest (ADR-0018), and the *ranked* Farsi lexical
# path is the benchmarked BM25 sparse vector, not this index. This one exists
# for exact keyword/filter matching, which ADR-0002 keeps deliberately distinct
# from retrieval.
_CONTENT_INDEX = models.TextIndexParams(
type=models.TextIndexType.TEXT,
tokenizer=models.TokenizerType.MULTILINGUAL,
lowercase=True,
)
# (field name, schema). `order_id` is float because Qdrant's `Range` conditions
# and `order_by` only support numeric/datetime payloads -- a keyword key could
# only be sorted client-side after fetching every chunk (ADR-0001).
_PayloadIndexSchema = models.PayloadSchemaType | models.KeywordIndexParams | models.TextIndexParams
_PAYLOAD_INDEXES: tuple[tuple[str, _PayloadIndexSchema], ...] = (
# `is_tenant` co-locates a tenant's vectors on disk for sequential reads,
# which is the whole point of payload-partitioned multitenancy.
(
"tenant_id",
models.KeywordIndexParams(type=models.KeywordIndexType.KEYWORD, is_tenant=True),
),
("domain", models.PayloadSchemaType.KEYWORD),
("file_id", models.PayloadSchemaType.KEYWORD),
("order_id", models.PayloadSchemaType.FLOAT),
("previous_chunk_id", models.PayloadSchemaType.KEYWORD),
("next_chunk_id", models.PayloadSchemaType.KEYWORD),
("content", _CONTENT_INDEX),
# Every read path filters on `is_active` (ADR-0002 implies `is_active: true`
# unless the caller opts in), and the re-ingestion sweep and `/v1/points`
# both range over `chunk_index`. Both were unindexed while ingestion was the
# only reader; plan 002 makes them hot.
("is_active", models.PayloadSchemaType.BOOL),
("chunk_index", models.PayloadSchemaType.INTEGER),
)
def _verify_existing(collection: str, info: models.CollectionInfo) -> None:
params = info.config.params
vectors = params.vectors
if not isinstance(vectors, dict):
raise CollectionSchemaMismatchError(
f"collection {collection!r} has an unnamed dense vector; ADR-0001 requires "
f"named vectors and this cannot be fixed without recreating the collection"
)
for name, expected_size in (
(DENSE_NOMIC_VECTOR, DENSE_NOMIC_DIMENSIONS),
(DENSE_OPENAI_VECTOR, DENSE_OPENAI_DIMENSIONS),
(LATE_INTERACTION_VECTOR, LATE_INTERACTION_DIMENSIONS),
):
existing = vectors.get(name)
if existing is None:
raise CollectionSchemaMismatchError(
f"collection {collection!r} is missing the {name!r} vector"
)
if existing.size != expected_size:
raise CollectionSchemaMismatchError(
f"collection {collection!r} has {name!r} at {existing.size} dimensions, "
f"expected {expected_size}; re-dimensioning is a re-embedding migration"
)
sparse = (params.sparse_vectors or {}).get(SPARSE_VECTOR)
if sparse is None:
raise CollectionSchemaMismatchError(
f"collection {collection!r} is missing the {SPARSE_VECTOR!r} vector; sparse "
f"vectors cannot be added without recreating the collection"
)
if sparse.modifier != models.Modifier.IDF:
raise CollectionSchemaMismatchError(
f"collection {collection!r} has {SPARSE_VECTOR!r} with modifier "
f"{sparse.modifier!r}, expected 'idf'; without it Qdrant applies no IDF "
f"and lexical retrieval silently degrades (ADR-0005)"
)
async def ensure_chunks_collection(client: AsyncQdrantClient, *, collection: str) -> bool:
"""Create the `chunks` collection and its payload indexes if absent.
Idempotent: an existing collection is verified against the pinned schema
and left alone. Returns whether it created the collection.
Raises `CollectionSchemaMismatchError` if an existing collection diverges.
"""
if await client.collection_exists(collection):
_verify_existing(collection, await client.get_collection(collection))
logger.info("qdrant.collection.verified", collection=collection)
# Payload indexes are additive and idempotent, so (re)creating them
# here is what lets an index be added to an already-live collection.
await _create_payload_indexes(client, collection=collection)
return False
await client.create_collection(
collection_name=collection,
vectors_config=_vectors_config(),
sparse_vectors_config=_sparse_vectors_config(),
# m=0 disables the global index; payload_m builds per-tenant graphs
# instead, per Qdrant's multitenant guidance (ADR-0001).
hnsw_config=models.HnswConfigDiff(m=0, payload_m=16),
)
logger.info("qdrant.collection.created", collection=collection)
await _create_payload_indexes(client, collection=collection)
return True
async def _create_payload_indexes(client: AsyncQdrantClient, *, collection: str) -> None:
for field_name, field_schema in _PAYLOAD_INDEXES:
await client.create_payload_index(
collection_name=collection, field_name=field_name, field_schema=field_schema
)
logger.info(
"qdrant.collection.payload_indexes.ensured",
collection=collection,
fields=[name for name, _ in _PAYLOAD_INDEXES],
)

View File

@@ -0,0 +1,247 @@
"""Qdrant adapter for the `PointRepository` port (ADR-0002, ADR-0015).
Every Qdrant filter for the `/v1/points` surface is built here. Routers and
application services never import `qdrant_client` — they pass a `tenant_id` and
get `Point` models back.
Two mechanics are worth reading before changing anything:
**Reads go through `scroll`, not `retrieve`.** `retrieve` fetches by id and
takes no filter, which would force the tenant check to happen *after* Qdrant
answered — exactly the "check it in Python afterwards" shape ADR-0002's
isolation rule exists to prevent. `scroll` with a `HasIdCondition` plus the
tenant condition pushes the check server-side, so a foreign id returns an empty
page rather than a row this code has to remember to reject.
**Ordered listing paginates by `order_id` value, not by offset.** Qdrant does
not return a `next_page_offset` when `order_by` is set, and an offset-based
cursor would skip or repeat rows when a concurrent insert shifts positions
underneath the reader. Ranging on `order_id > cursor` is stable instead: a point
inserted ahead of the cursor was already returned, and one inserted after it
shows up on a later page. This relies on `order_id` being unique within a file,
which ADR-0002 guarantees by rejecting a reorder whose gap would collapse onto a
neighbour value.
"""
import uuid
from collections.abc import Sequence
from qdrant_client import AsyncQdrantClient, models
from src.application.points.point import Point
from src.application.ports.point_repository import PayloadPatch, PointPage
# Qdrant's scroll returns `(records, next_page_offset)`; a record's id is a
# `str | int` union in the SDK, and every id this service writes is a UUID.
type _ScrollRecord = models.Record
def _uuid_of(record: _ScrollRecord) -> uuid.UUID:
return uuid.UUID(str(record.id))
def _to_point(record: _ScrollRecord, *, with_vectors: bool) -> Point:
vectors = record.vector if with_vectors and isinstance(record.vector, dict) else None
return Point.from_payload(_uuid_of(record), record.payload or {}, vectors=vectors)
def _conditions(
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> list[models.Condition]:
"""The base filter every point query carries.
`tenant_id` is unconditional and first. `is_active` is added unless the
caller explicitly opted into inactive points, which is ADR-0002's
"`is_active: true` implied" rule expressed once rather than per method.
"""
conditions: list[models.Condition] = [
models.FieldCondition(key="tenant_id", match=models.MatchValue(value=str(tenant_id)))
]
if domain is not None:
conditions.append(
models.FieldCondition(key="domain", match=models.MatchValue(value=domain))
)
if file_id is not None:
conditions.append(
models.FieldCondition(key="file_id", match=models.MatchValue(value=str(file_id)))
)
if not include_inactive:
conditions.append(
models.FieldCondition(key="is_active", match=models.MatchValue(value=True))
)
return conditions
class QdrantPointRepository:
"""A `PointRepository` (see `src/application/ports/point_repository.py`)."""
def __init__(self, client: AsyncQdrantClient, *, collection: str) -> None:
self._client = client
self._collection = collection
async def get(
self, *, tenant_id: uuid.UUID, point_id: uuid.UUID, with_vectors: bool = False
) -> Point | None:
records, _ = await self._client.scroll(
collection_name=self._collection,
scroll_filter=models.Filter(
must=[
*_conditions(tenant_id=tenant_id, include_inactive=True),
models.HasIdCondition(has_id=[str(point_id)]),
]
),
limit=1,
with_payload=True,
with_vectors=with_vectors,
)
if not records:
return None
return _to_point(records[0], with_vectors=with_vectors)
async def get_many(
self, *, tenant_id: uuid.UUID, point_ids: Sequence[uuid.UUID]
) -> tuple[Point, ...]:
if not point_ids:
return ()
records, _ = await self._client.scroll(
collection_name=self._collection,
scroll_filter=models.Filter(
must=[
*_conditions(tenant_id=tenant_id, include_inactive=True),
models.HasIdCondition(has_id=[str(point_id) for point_id in point_ids]),
]
),
limit=len(point_ids),
with_payload=True,
with_vectors=False,
)
return tuple(_to_point(record, with_vectors=False) for record in records)
async def list_by_file(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
limit: int,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointPage:
conditions = _conditions(
tenant_id=tenant_id, file_id=file_id, include_inactive=include_inactive
)
if cursor is not None:
conditions.append(
models.FieldCondition(key="order_id", range=models.Range(gt=float(cursor)))
)
records, _ = await self._client.scroll(
collection_name=self._collection,
scroll_filter=models.Filter(must=conditions),
order_by=models.OrderBy(key="order_id", direction=models.Direction.ASC),
limit=limit,
with_payload=True,
with_vectors=False,
)
points = tuple(_to_point(record, with_vectors=False) for record in records)
# A short page means the listing is exhausted. A full page might be, but
# claiming so would need an extra round trip; handing back a cursor that
# yields an empty final page is the cheaper honest answer.
next_cursor = repr(points[-1].order_id) if len(points) == limit else None
return PointPage(points=points, next_cursor=next_cursor)
async def count(
self,
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> int:
result = await self._client.count(
collection_name=self._collection,
count_filter=models.Filter(
must=_conditions(
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
),
exact=True,
)
return result.count
async def keyword_search(
self,
*,
tenant_id: uuid.UUID,
query: str,
limit: int,
cursor: str | None = None,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointPage:
conditions = _conditions(
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
conditions.append(models.FieldCondition(key="content", match=models.MatchText(text=query)))
# No `order_by` here, so Qdrant does return a page offset: the full-text
# index filters rather than scores, and imposing `order_id` ordering
# across files would be meaningless (`order_id` is per-file).
records, next_offset = await self._client.scroll(
collection_name=self._collection,
scroll_filter=models.Filter(must=conditions),
limit=limit,
offset=cursor,
with_payload=True,
with_vectors=False,
)
return PointPage(
points=tuple(_to_point(record, with_vectors=False) for record in records),
next_cursor=str(next_offset) if next_offset is not None else None,
)
async def apply_patches(self, *, tenant_id: uuid.UUID, patches: Sequence[PayloadPatch]) -> None:
if not patches:
return
await self._client.batch_update_points(
collection_name=self._collection,
update_operations=[
models.SetPayloadOperation(
set_payload=models.SetPayload(
payload=patch.payload,
filter=models.Filter(must=self._patch_conditions(tenant_id, patch)),
)
)
for patch in patches
],
)
@staticmethod
def _patch_conditions(tenant_id: uuid.UUID, patch: PayloadPatch) -> list[models.Condition]:
"""Address one point by id, under this tenant, at an expected version.
`include_inactive=True`: soft-deleting and relinking both have to reach
points the default read filter hides, and a patch is addressed by an
explicit id rather than discovered by a listing.
"""
conditions: list[models.Condition] = [
*_conditions(tenant_id=tenant_id, include_inactive=True),
models.HasIdCondition(has_id=[str(patch.point_id)]),
]
if patch.expected_version is not None:
conditions.append(
models.FieldCondition(
key="version", match=models.MatchValue(value=patch.expected_version)
)
)
return conditions

View File

@@ -0,0 +1,97 @@
"""Qdrant adapter for the `PointStorage` port (ADR-0001, ADR-0002, ADR-0015).
The `qdrant_client` SDK appears here and nowhere in `application/`. This module
translates the SDK-free `ChunkPoint` into `PointStruct`s and builds every
filter — routers and application services never construct Qdrant filters.
`AsyncQdrantClient` is genuinely async, so unlike the `minio` adapter nothing
here needs a thread offload.
"""
import uuid
from collections.abc import Sequence
from datetime import datetime
from qdrant_client import AsyncQdrantClient, models
from src.application.points.models import ChunkPoint
from src.infrastructure.qdrant.collection import SPARSE_VECTOR
def _tenant_file_filter(
tenant_id: uuid.UUID, file_id: uuid.UUID, *, from_chunk_index: int
) -> models.Filter:
"""Points of one file, at or past `from_chunk_index`, within one tenant.
`tenant_id` is always a condition, never optional: a `file_id` alone is not
authority to mutate anything (ADR-0002's isolation rule applies to every
code path, not just reads).
"""
return models.Filter(
must=[
models.FieldCondition(key="tenant_id", match=models.MatchValue(value=str(tenant_id))),
models.FieldCondition(key="file_id", match=models.MatchValue(value=str(file_id))),
models.FieldCondition(key="chunk_index", range=models.Range(gte=from_chunk_index)),
]
)
class QdrantPointStorage:
"""A `PointStorage` (see `src/application/ports/point_storage.py`)."""
def __init__(self, client: AsyncQdrantClient, *, collection: str) -> None:
self._client = client
self._collection = collection
async def upsert_points(self, points: Sequence[ChunkPoint]) -> None:
if not points:
return
await self._client.upsert(
collection_name=self._collection,
points=[
models.PointStruct(
id=str(point.point_id),
vector={
**point.dense,
SPARSE_VECTOR: models.SparseVector(
indices=point.sparse.indices, values=point.sparse.values
),
},
payload=point.payload,
)
for point in points
],
)
async def deactivate_points_from_index(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
from_chunk_index: int,
deleted_at: datetime,
updated_by: str,
) -> int:
"""Soft-delete via `set_payload` — the points stay for audit (ADR-0002).
Counts first so the caller can report how many points the sweep
touched; `set_payload` itself reports only an operation status.
"""
point_filter = _tenant_file_filter(tenant_id, file_id, from_chunk_index=from_chunk_index)
stale = await self._client.count(
collection_name=self._collection, count_filter=point_filter, exact=True
)
if stale.count == 0:
return 0
await self._client.set_payload(
collection_name=self._collection,
payload={
"is_active": False,
"deleted_at": deleted_at.isoformat(),
"updated_at": deleted_at.isoformat(),
"updated_by": updated_by,
},
points=models.FilterSelector(filter=point_filter),
)
return stale.count

View File

@@ -1,7 +1,8 @@
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Iterator
import pytest
import pytest_asyncio
import structlog
from asgi_lifespan import LifespanManager
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
@@ -9,6 +10,31 @@ from httpx import ASGITransport, AsyncClient
from src.config import Settings
from src.main import create_app
# Disposable Postgres/MinIO/Qdrant containers (ADR-0016). Registered here
# because `pytest_plugins` is only honoured in the root conftest, and both
# `tests/integration/*/` and `tests/e2e/` need the same containers. The
# fixtures are session-scoped but lazy, so unit runs still need no Docker.
pytest_plugins = ["tests.support.containers"]
@pytest.fixture(autouse=True)
def _reset_structlog_after_test() -> Iterator[None]:
"""Undo any real `configure_logging()` call before the next test runs.
Any test that exercises the app's lifespan (directly, or via the `client`/
`api_client` fixtures below and in `test_domains_api.py`) calls the real
`configure_logging()`, which mutates *global* structlog/stdlib state --
including `cache_logger_on_first_use=True`. Left in place, that setting
silently breaks `structlog.testing.capture_logs()` in unrelated tests
later in the same pytest process: a module-level
`logger = structlog.get_logger(__name__)` cached under the real config no
longer routes through `capture_logs()`'s temporary processor swap, so
assertions on captured events see nothing (ADR-0016: isolate per test --
this generalizes to global config mutations, not just data).
"""
yield
structlog.reset_defaults()
@pytest.fixture
def settings() -> Settings:
@@ -36,6 +62,24 @@ def app(settings: Settings) -> FastAPI:
return create_app(settings)
@pytest.fixture(autouse=True)
def _no_real_logging_configuration(monkeypatch: pytest.MonkeyPatch) -> None:
"""Prevent the app lifespan from calling the real `configure_logging()`.
It sets `cache_logger_on_first_use=True` (ADR-0011), which permanently
monkeypatches the `.bind` method on whichever module-level
`logger = structlog.get_logger(__name__)` instance is used first --
`structlog.reset_defaults()` only resets *global* config, not that
per-instance mutation, so real configuration leaking into one test would
silently break `structlog.testing.capture_logs()` in every test that runs
afterward in the same process (ADR-0016: isolate per test). Tests that
spin up the full app via `LifespanManager` (`client`, `api_client`) are
testing HTTP behavior, not logging output, so they don't need it for
real.
"""
monkeypatch.setattr("src.bootstrap.lifespan.configure_logging", lambda *a, **k: None)
@pytest_asyncio.fixture
async def client(app: FastAPI) -> AsyncIterator[AsyncClient]:
async with (

244
tests/e2e/conftest.py Normal file
View File

@@ -0,0 +1,244 @@
"""The vertical slice against real Postgres, MinIO, and Qdrant (ADR-0016).
These are the ADR-0016 end-to-end tests: "a small vertical-slice acceptance
contract through the real application composition". They run on
Testcontainers, in the default `uv run pytest` run, because ADR-0016 makes
Testcontainers "the standard automated integration-test resource mechanism"
and reserves Docker Compose for manual validation and the separate serialized
smoke test of the *running web process* (`tests/e2e/test_compose_smoke.py`).
What is real here: routing, auth, scopes, the domain allowlist, the ADR-0008
error envelope, the two-transaction upload, MinIO object writes, BM25 sparse
embedding, and Qdrant upserts into a per-test collection created by the
production `ensure_chunks_collection`.
What is faked, and only this: the two dense embedders. They are paid/remote
network calls, and ADR-0016 forbids routine runs from touching a live
provider. Their fakes return the pinned 768/3072 dimensions, because the real
collection rejects anything else.
"""
import uuid
from collections.abc import AsyncIterator, Callable, Coroutine
from contextlib import AsyncExitStack
from dataclasses import dataclass
from typing import Any
import pytest
import pytest_asyncio
from asgi_lifespan import LifespanManager
from httpx import ASGITransport, AsyncClient
from qdrant_client import AsyncQdrantClient, models
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
from src.bootstrap.dependencies import get_dense_embedders, get_sessionmaker
from src.config import MinioSettings, PostgresSettings, QdrantSettings, Settings
from src.infrastructure.postgres.database import create_sessionmaker
from src.infrastructure.qdrant.collection import (
DENSE_NOMIC_DIMENSIONS,
DENSE_OPENAI_DIMENSIONS,
ensure_chunks_collection,
)
from src.main import create_app
from tests.fakes import FakeDenseEmbedder
from tests.support.factories import create_api_key, create_tenant, create_tenant_domain
CSV_BYTES = b"question,answer\nhow do I file a claim,call the branch\nwhat is covered,see policy\n"
# Past validation's OOXML magic-byte check, so the failure lands in the DOCX
# parser rather than in upload validation -- the branch this exercises.
CORRUPT_DOCX_BYTES = b"PK\x03\x04" + b"not really a docx" * 8
@dataclass
class E2EStack:
"""One configured app instance plus the handles a test asserts against."""
client: AsyncClient
collection: str
dense_embedders: list[FakeDenseEmbedder]
StackFactory = Callable[..., Coroutine[Any, Any, E2EStack]]
@dataclass(frozen=True)
class Principal:
"""A provisioned tenant with a bearer token and a registered domain.
Plain values, not the ORM `Tenant`: tests re-read rows the app committed on
a shared connection, and any expiry/refresh of a held ORM instance would
lazy-load outside a greenlet context (`MissingGreenlet`).
"""
tenant_id: uuid.UUID
tenant_slug: str
token: str
domain: str
@property
def headers(self) -> dict[str, str]:
return {"Authorization": f"Bearer {self.token}"}
@pytest.fixture
def e2e_settings(
postgres_settings: PostgresSettings,
minio_settings: MinioSettings,
qdrant_settings: QdrantSettings,
) -> Settings:
"""Settings wired to this test's containers.
Every nested settings object is passed explicitly. `src/config.py` cascades
`.env` into each nested class, so omitting one would let a developer's real
`EMBEDDING_NOMIC_BASE_URL` (a colleague's Ollama box) into a routine test
run. The embedder URLs below point at a closed port for the same reason:
the lifespan warms the *real* embedders before the fakes are substituted,
and connection-refused is both instant and free.
"""
return Settings(
postgres=postgres_settings,
minio=minio_settings,
qdrant=qdrant_settings,
embedding={
"nomic": {"base_url": "http://127.0.0.1:1/v1", "timeout_seconds": 0.5},
"openai": {"base_url": "http://127.0.0.1:1/v1", "timeout_seconds": 0.5},
},
app={"readiness_check_timeout_seconds": 2.0},
)
@pytest_asyncio.fixture(loop_scope="session")
async def make_stack(
e2e_settings: Settings,
db_sessionmaker: async_sessionmaker[AsyncSession],
qdrant_client: AsyncQdrantClient,
) -> AsyncIterator[StackFactory]:
"""Build an app under test, optionally with tightened ingestion bounds.
A factory rather than a fixture because the capacity and timeout tests need
their own `INGESTION_*` values, and those are read at app construction.
"""
async with AsyncExitStack() as exit_stack:
async def _make(
*,
ingestion: dict[str, object] | None = None,
collection: str | None = None,
dense_delay_seconds: float = 0.0,
bootstrap_collection: bool = True,
sessionmaker: async_sessionmaker[AsyncSession] | None = None,
) -> E2EStack:
resolved_collection = collection or e2e_settings.qdrant.collection
if bootstrap_collection:
await ensure_chunks_collection(qdrant_client, collection=resolved_collection)
settings = e2e_settings.model_copy(
update={
"qdrant": e2e_settings.qdrant.model_copy(
update={"collection": resolved_collection}
),
"ingestion": e2e_settings.ingestion.model_copy(update=ingestion or {}),
}
)
dense = [
FakeDenseEmbedder(
name="dense_nomic",
dimensions=DENSE_NOMIC_DIMENSIONS,
value=0.1,
delay_seconds=dense_delay_seconds,
),
FakeDenseEmbedder(
name="dense_openai",
dimensions=DENSE_OPENAI_DIMENSIONS,
value=0.2,
delay_seconds=dense_delay_seconds,
),
]
app = create_app(settings)
# The session factory is the test's, so factory-created tenants are
# visible to the app and every write rolls back at teardown. Object
# storage, point storage, and the sparse embedder stay real.
resolved_sessionmaker = sessionmaker or db_sessionmaker
app.dependency_overrides[get_sessionmaker] = lambda: resolved_sessionmaker
app.dependency_overrides[get_dense_embedders] = lambda: dense
manager = await exit_stack.enter_async_context(LifespanManager(app))
client = await exit_stack.enter_async_context(
AsyncClient(
transport=ASGITransport(app=manager.app),
base_url="http://test",
timeout=30.0,
)
)
return E2EStack(client=client, collection=resolved_collection, dense_embedders=dense)
yield _make
@pytest_asyncio.fixture(loop_scope="session")
async def committed_sessionmaker(
postgres_engine: AsyncEngine,
) -> async_sessionmaker[AsyncSession]:
"""A session factory whose writes really commit, one connection each.
The default `db_sessionmaker` pins every session to a single connection
inside one rolled-back transaction, which is what isolates a test's writes
-- but savepoints on a shared connection cannot interleave, so any test
that issues genuinely *concurrent* requests deadlocks or fails with
`InvalidSavepointSpecificationError`. Those tests use this instead and rely
on their uuid-keyed tenant for isolation (ADR-0016 allows either).
"""
return create_sessionmaker(postgres_engine)
@pytest_asyncio.fixture(loop_scope="session")
async def principal(db_session: AsyncSession) -> Principal:
"""A tenant with an upload-scoped key and one registered domain.
Registering the domain is not optional: `POST /v1/files` rejects an
unregistered one with `400 unknown_domain` before anything else runs
(ADR-0009).
"""
return await provision_principal(db_session)
async def provision_principal(
session: AsyncSession, *, domain: str = "fire", scopes: list[str] | None = None
) -> Principal:
tenant = await create_tenant(session)
_, token = await create_api_key(session, tenant=tenant, scopes=scopes or ["files:write"])
await create_tenant_domain(session, tenant=tenant, domain=domain)
await session.commit()
return Principal(tenant_id=tenant.id, tenant_slug=tenant.slug, token=token, domain=domain)
def upload_payload(
*, domain: str, filename: str = "faq.csv", data: bytes = CSV_BYTES
) -> dict[str, Any]:
return {"files": {"file": (filename, data, "text/csv")}, "data": {"domain": domain}}
async def count_active_points(
client: AsyncQdrantClient, collection: str, *, tenant_id: uuid.UUID
) -> int:
"""Active points for one tenant, read through the raw client.
`PointStorage` is deliberately write-only (point reads are plan 002's
`/v1/points`), so the verification read here uses the SDK directly.
"""
result = await client.count(
collection_name=collection,
count_filter=models.Filter(
must=[
models.FieldCondition(
key="tenant_id", match=models.MatchValue(value=str(tenant_id))
),
models.FieldCondition(key="is_active", match=models.MatchValue(value=True)),
]
),
exact=True,
)
return result.count

View File

@@ -0,0 +1,143 @@
"""Operational smoke test against the *running web process* (ADR-0016).
Different in kind from every other test in this repo. Everything else drives
the app in-process through `ASGITransport`; this drives a real uvicorn process
over a real socket, backed by the Docker Compose stack, after the two
deployment steps (`alembic upgrade head`, `src.cli.qdrant_bootstrap`) have run
for real. ADR-0016 reserves Compose for exactly this: "manual local validation
and a later, serialized operational smoke test of the running web process",
run "as a serialized pre-release or scheduled gate".
It is therefore **skipped unless `SMOKE_BASE_URL` is set**, so a plain
`uv run pytest` never touches Docker Compose. Run it through `scripts/smoke.sh`,
which brings the stack up, provisions a tenant, starts the process, and exports
the environment below.
Because the app runs in its own process, this is also the only place in the
suite where the real `configure_logging()` is in effect --
`tests/conftest.py::_no_real_logging_configuration` no-ops it everywhere else
to keep global structlog state out of other tests. So the ADR-0011 log sink is
asserted here and nowhere else.
"""
import json
import os
import uuid
from collections.abc import AsyncIterator
from pathlib import Path
import pytest
import pytest_asyncio
from httpx import AsyncClient
from qdrant_client import AsyncQdrantClient, models
BASE_URL = os.environ.get("SMOKE_BASE_URL")
API_KEY = os.environ.get("SMOKE_API_KEY", "")
DOMAIN = os.environ.get("SMOKE_DOMAIN", "smoke")
LOG_PATH = os.environ.get("SMOKE_LOG_PATH", "")
QDRANT_URL = os.environ.get("SMOKE_QDRANT_URL", "http://127.0.0.1:6343")
QDRANT_COLLECTION = os.environ.get("SMOKE_QDRANT_COLLECTION", "chunks")
pytestmark = [
pytest.mark.e2e,
pytest.mark.slow,
pytest.mark.asyncio,
pytest.mark.skipif(
not BASE_URL,
reason="SMOKE_BASE_URL is unset; run this through scripts/smoke.sh",
),
# Well past the global 10s budget: this drives a real process over a
# socket, and the upload does real embedding work end to end.
pytest.mark.timeout(120),
]
# Unique per run so a re-run against the same persistent Compose volumes is a
# new file rather than an idempotent duplicate-hash hit (ADR-0017).
_CSV = f"question,answer\nsmoke run {uuid.uuid4().hex},indexed\n".encode()
@pytest_asyncio.fixture
async def client() -> AsyncIterator[AsyncClient]:
async with AsyncClient(base_url=BASE_URL or "", timeout=120.0) as async_client:
yield async_client
async def test_running_process_reports_healthy_and_ready() -> None:
"""Readiness must be green *after* the deployment steps, not before --
an unbootstrapped Qdrant is deliberately unready.
"""
async with AsyncClient(base_url=BASE_URL or "", timeout=30.0) as client:
healthz = await client.get("/healthz")
readyz = await client.get("/readyz")
assert healthz.status_code == 200
assert healthz.json() == {"status": "ok"}
assert readyz.status_code == 200, readyz.text
assert readyz.json() == {"postgres": True, "minio": True, "qdrant": True}
async def test_upload_through_the_running_process_indexes_retrievable_points(
client: AsyncClient,
) -> None:
"""The whole slice against the deployed shape: HTTP in, Qdrant points out."""
upload = await client.post(
"/v1/files",
files={"file": ("smoke.csv", _CSV, "text/csv")},
data={"domain": DOMAIN},
headers={"Authorization": f"Bearer {API_KEY}"},
)
assert upload.status_code == 201, upload.text
body = upload.json()
assert body["status"] == "succeeded"
assert body["chunks_indexed"] > 0
status = await client.get(
f"/v1/files/{body['file_id']}", headers={"Authorization": f"Bearer {API_KEY}"}
)
assert status.status_code == 200
assert status.json()["ingestion_status"] == "succeeded"
qdrant = AsyncQdrantClient(url=QDRANT_URL)
try:
count = await qdrant.count(
collection_name=QDRANT_COLLECTION,
count_filter=models.Filter(
must=[
models.FieldCondition(
key="file_id", match=models.MatchValue(value=body["file_id"])
),
models.FieldCondition(key="is_active", match=models.MatchValue(value=True)),
]
),
exact=True,
)
finally:
await qdrant.close()
assert count.count == body["chunks_indexed"]
async def test_running_process_writes_structured_ingestion_logs() -> None:
"""The real ADR-0011 sink, which only a separate process exercises.
An upload that succeeds while emitting nothing parseable is an upload no
operator can investigate -- and the runbook's failure procedure is written
against these exact event names.
"""
if not LOG_PATH:
pytest.skip("SMOKE_LOG_PATH is unset; scripts/smoke.sh normally provides it")
lines = Path(LOG_PATH).read_text("utf-8").splitlines()
events = [json.loads(line) for line in lines if line.startswith("{")]
by_name = {event.get("event") for event in events}
assert "ingestion.job.started" in by_name
assert "ingestion.job.completed" in by_name
completed = next(event for event in events if event.get("event") == "ingestion.job.completed")
assert completed["tenant_id"]
assert completed["ingestion_job_id"]
assert completed["points_upserted"] > 0
# Static process context bound once at startup (ADR-0011).
assert completed["env"]
assert completed["service_version"]

View File

@@ -0,0 +1,329 @@
"""The ingestion vertical slice end to end (plan 001, Phase 6).
Every test here asserts a reliability invariant ADR-0016 lists as a reusable
contract, not a happy path: the bound maps to its status code, the failure
still writes a terminal job row, the retry converges, and the tenant filter
holds. Failures additionally assert the ADR-0008 error envelope, because the
status code alone is not the contract a client integrates against.
"""
import asyncio
import uuid
import pytest
from httpx import Response
from qdrant_client import AsyncQdrantClient
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.infrastructure.postgres.models.ingestion_job import IngestionJob
from tests.e2e.conftest import (
CORRUPT_DOCX_BYTES,
Principal,
StackFactory,
count_active_points,
provision_principal,
upload_payload,
)
pytestmark = [
pytest.mark.e2e,
pytest.mark.postgres,
pytest.mark.minio,
pytest.mark.qdrant,
pytest.mark.asyncio(loop_scope="session"),
]
async def _latest_job(session: AsyncSession, *, tenant_id: uuid.UUID) -> IngestionJob:
"""The tenant's most recent job row, read fresh.
`populate_existing` matters: this session shares a connection with the
app's sessions, so without it the identity map can answer from a row
loaded before the request under test committed its update.
"""
result = await session.execute(
select(IngestionJob)
.where(IngestionJob.tenant_id == tenant_id)
.order_by(IngestionJob.created_at.desc(), IngestionJob.id)
.execution_options(populate_existing=True)
)
return result.scalars().first() or pytest.fail("no ingestion job was written")
def _envelope(response: Response) -> dict[str, object]:
"""The ADR-0008 error body. Asserting the code here, not just the HTTP
status, is the difference between testing the contract and testing the
number."""
error: dict[str, object] = response.json()["error"]
assert error["request_id"], "every error envelope carries its correlation id"
return error
async def test_upload_succeeds_and_indexes_points_under_the_tenant_filter(
make_stack: StackFactory, principal: Principal, qdrant_client: AsyncQdrantClient
) -> None:
stack = await make_stack()
response = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
assert response.status_code == 201
body = response.json()
assert body["status"] == "succeeded"
assert body["chunks_indexed"] > 0
indexed = await count_active_points(
qdrant_client, stack.collection, tenant_id=principal.tenant_id
)
assert indexed == body["chunks_indexed"]
async def test_get_file_reports_the_terminal_status_after_upload(
make_stack: StackFactory, principal: Principal
) -> None:
stack = await make_stack()
created = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
status = await stack.client.get(
f"/v1/files/{created.json()['file_id']}", headers=principal.headers
)
assert status.status_code == 200
assert status.json()["ingestion_status"] == "succeeded"
assert status.json()["domain"] == principal.domain
async def test_upload_duplicate_content_returns_the_existing_job_without_reingesting(
make_stack: StackFactory, principal: Principal, qdrant_client: AsyncQdrantClient
) -> None:
stack = await make_stack()
first = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
points_after_first = await count_active_points(
qdrant_client, stack.collection, tenant_id=principal.tenant_id
)
second = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
assert first.status_code == 201
# 200, not 201: nothing was created the second time (ADR-0017 idempotency).
assert second.status_code == 200
assert second.json()["file_id"] == first.json()["file_id"]
assert second.json()["ingestion_job_id"] == first.json()["ingestion_job_id"]
assert (
await count_active_points(qdrant_client, stack.collection, tenant_id=principal.tenant_id)
== points_after_first
)
async def test_upload_retried_after_a_failed_job_succeeds_without_duplicate_points(
make_stack: StackFactory,
principal: Principal,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
) -> None:
"""ADR-0017, "Re-running an ingestion stays safe": a failed attempt is
retried by re-uploading the same bytes, and deterministic point ids make
the retry converge instead of duplicating.
"""
stack = await make_stack()
stack.dense_embedders[0].fail_next = True
failed = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
failed_job = await _latest_job(db_session, tenant_id=principal.tenant_id)
retried = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
assert failed.status_code == 502
assert failed_job.status == "failed"
assert failed_job.error_code == "embedding_failed"
assert retried.status_code == 201
assert retried.json()["status"] == "succeeded"
# Same source file, new attempt -- a terminal job is never reused or
# transitioned back to running.
assert retried.json()["ingestion_job_id"] != str(failed_job.id)
indexed = await count_active_points(
qdrant_client, stack.collection, tenant_id=principal.tenant_id
)
assert indexed == retried.json()["chunks_indexed"]
async def test_get_file_for_another_tenants_file_returns_404_not_403(
make_stack: StackFactory, principal: Principal, db_session: AsyncSession
) -> None:
"""404, never 403: a 403 would confirm the file exists to a stranger."""
stack = await make_stack()
created = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
other = await provision_principal(db_session)
response = await stack.client.get(
f"/v1/files/{created.json()['file_id']}", headers=other.headers
)
assert response.status_code == 404
assert _envelope(response)["code"] == "not_found"
async def test_upload_to_an_unregistered_domain_is_rejected_before_any_job_is_written(
make_stack: StackFactory, principal: Principal, db_session: AsyncSession
) -> None:
stack = await make_stack()
response = await stack.client.post(
"/v1/files", **upload_payload(domain="never-registered"), headers=principal.headers
)
assert response.status_code == 400
assert _envelope(response)["code"] == "unknown_domain"
jobs = await db_session.execute(
select(IngestionJob).where(IngestionJob.tenant_id == principal.tenant_id)
)
assert jobs.scalars().all() == []
async def test_upload_at_capacity_is_rejected_with_503_and_retry_after(
make_stack: StackFactory, committed_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""ADR-0017 rejects rather than queues, so the second concurrent upload
must fail fast with a backoff hint instead of waiting for a slot.
This is the one test that needs two requests genuinely in flight at once,
so it runs on `committed_sessionmaker` -- see that fixture for why the
shared-connection default cannot serve concurrent sessions.
"""
async with committed_sessionmaker() as session:
principal = await provision_principal(session)
stack = await make_stack(
ingestion={"max_concurrency": 1},
dense_delay_seconds=0.4,
sessionmaker=committed_sessionmaker,
)
first, second = await asyncio.gather(
stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
),
stack.client.post(
"/v1/files",
**upload_payload(domain=principal.domain, filename="other.csv", data=b"a,b\n1,2\n"),
headers=principal.headers,
),
)
statuses = sorted([first.status_code, second.status_code])
assert statuses == [201, 503]
rejected = first if first.status_code == 503 else second
assert _envelope(rejected)["code"] == "ingestion_at_capacity"
assert rejected.headers["Retry-After"] == "1"
async def test_upload_exceeding_the_timeout_returns_504_and_writes_a_terminal_job(
make_stack: StackFactory, principal: Principal, db_session: AsyncSession
) -> None:
"""A timeout must never leave a job stuck in `running` (ADR-0016)."""
stack = await make_stack(ingestion={"timeout_seconds": 0.05}, dense_delay_seconds=0.5)
response = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
assert response.status_code == 504
assert _envelope(response)["code"] == "ingestion_timeout"
job = await _latest_job(db_session, tenant_id=principal.tenant_id)
assert job.status == "failed"
assert job.error_code == "timeout"
async def test_upload_of_an_unparseable_document_returns_400_and_a_failed_job(
make_stack: StackFactory, principal: Principal, db_session: AsyncSession
) -> None:
stack = await make_stack()
response = await stack.client.post(
"/v1/files",
**upload_payload(domain=principal.domain, filename="broken.docx", data=CORRUPT_DOCX_BYTES),
headers=principal.headers,
)
assert response.status_code == 400
assert _envelope(response)["code"] == "validation_error"
job = await _latest_job(db_session, tenant_id=principal.tenant_id)
assert job.status == "failed"
assert job.error_code == "parse_failed"
async def test_upload_returns_502_when_qdrant_indexing_fails(
make_stack: StackFactory, principal: Principal, db_session: AsyncSession
) -> None:
"""A real Qdrant error, not a fake: the collection the deployment step
should have created is missing, which is the failure an operator actually
hits when `qdrant_bootstrap` was skipped.
"""
stack = await make_stack(
collection=f"never_bootstrapped_{uuid.uuid4().hex}", bootstrap_collection=False
)
response = await stack.client.post(
"/v1/files", **upload_payload(domain=principal.domain), headers=principal.headers
)
assert response.status_code == 502
assert _envelope(response)["code"] == "index_error"
job = await _latest_job(db_session, tenant_id=principal.tenant_id)
assert job.status == "failed"
assert job.error_code == "index_failed"
async def test_upload_without_the_files_write_scope_is_forbidden(
make_stack: StackFactory, db_session: AsyncSession
) -> None:
stack = await make_stack()
reader = await provision_principal(db_session, scopes=["domains:read"])
response = await stack.client.post(
"/v1/files", **upload_payload(domain=reader.domain), headers=reader.headers
)
assert response.status_code == 403
assert _envelope(response)["code"] == "missing_scope"
async def test_readyz_reports_every_dependency_ready_against_real_services(
make_stack: StackFactory,
) -> None:
"""`/readyz` is dependency readiness, `/healthz` is process health."""
stack = await make_stack()
ready = await stack.client.get("/readyz")
healthy = await stack.client.get("/healthz")
assert healthy.status_code == 200
assert ready.status_code == 200
assert ready.json() == {"postgres": True, "minio": True, "qdrant": True}
async def test_readyz_is_unready_when_the_chunks_collection_is_missing(
make_stack: StackFactory,
) -> None:
"""A reachable but unbootstrapped Qdrant is deliberately not ready --
uploads to it would 502 (see the indexing test above).
"""
stack = await make_stack(
collection=f"never_bootstrapped_{uuid.uuid4().hex}", bootstrap_collection=False
)
response = await stack.client.get("/readyz")
assert response.status_code == 503
assert response.json()["qdrant"] is False

View File

@@ -1,10 +1,15 @@
"""Hand-written fakes for narrow application-owned ports (ADR-0016)."""
import asyncio
import uuid
from collections.abc import Sequence
from dataclasses import dataclass, field
from datetime import datetime
from src.application.ingestion.models import SparseVector
from src.application.points.models import ChunkPoint
from src.application.points.point import Point
from src.application.ports.point_repository import PayloadPatch, PointPage
@dataclass
@@ -29,6 +34,10 @@ class FakeDenseEmbedder:
name: str
dimensions: int = 4
value: float = 0.0
"""Component value of every returned vector. Non-zero where a real Qdrant
has to score the result, since a zero vector has no direction to compare."""
model_version: str = "fake-dense-v1"
calls: list[list[str]] = field(default_factory=list)
fail_next: bool = False
delay_seconds: float = 0.0
@@ -41,7 +50,7 @@ class FakeDenseEmbedder:
if self.fail_next:
self.fail_next = False
raise RuntimeError("simulated embedder failure")
return [[0.0] * self.dimensions for _ in texts]
return [[self.value] * self.dimensions for _ in texts]
@dataclass
@@ -49,6 +58,7 @@ class FakeSparseEmbedder:
"""A scripted `SparseEmbedder`. Returns an empty sparse vector per text."""
name: str = "sparse"
model_version: str = "fake-sparse-v1"
calls: list[list[str]] = field(default_factory=list)
fail_next: bool = False
@@ -58,3 +68,234 @@ class FakeSparseEmbedder:
self.fail_next = False
raise RuntimeError("simulated embedder failure")
return [SparseVector(indices=[], values=[]) for _ in texts]
@dataclass
class FakePointStorage:
"""An in-memory `PointStorage`.
`points` is keyed by point id, so a re-upsert of the same deterministic id
overwrites rather than accumulating — the property a test asserting "a
retry produces no duplicate points" needs the fake to actually model.
`fail_on_batch` fails the Nth (0-based) upsert batch, which is how a test
checks that the soft-delete sweep never runs after a partial failure.
"""
points: dict[str, ChunkPoint] = field(default_factory=dict)
upsert_batches: list[int] = field(default_factory=list)
deactivate_calls: list[dict[str, object]] = field(default_factory=list)
fail_on_batch: int | None = None
fail_deactivate: bool = False
max_in_flight: int = 0
_in_flight: int = 0
async def upsert_points(self, points: Sequence[ChunkPoint]) -> None:
self._in_flight += 1
self.max_in_flight = max(self.max_in_flight, self._in_flight)
try:
# Yield so concurrent batches actually overlap; without this the
# in-flight ceiling is trivially 1 and the bound goes untested.
await asyncio.sleep(0)
index = len(self.upsert_batches)
self.upsert_batches.append(len(points))
if self.fail_on_batch is not None and index == self.fail_on_batch:
raise RuntimeError("simulated point storage failure")
for point in points:
self.points[str(point.point_id)] = point
finally:
self._in_flight -= 1
async def deactivate_points_from_index(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
from_chunk_index: int,
deleted_at: datetime,
updated_by: str,
) -> int:
self.deactivate_calls.append(
{
"tenant_id": tenant_id,
"file_id": file_id,
"from_chunk_index": from_chunk_index,
"updated_by": updated_by,
}
)
if self.fail_deactivate:
raise RuntimeError("simulated deactivate failure")
def is_stale(point: ChunkPoint) -> bool:
chunk_index = point.payload.get("chunk_index")
return (
point.payload.get("file_id") == str(file_id)
and point.payload.get("tenant_id") == str(tenant_id)
and point.payload.get("is_active") is True
and isinstance(chunk_index, int)
and chunk_index >= from_chunk_index
)
stale = [point for point in self.points.values() if is_stale(point)]
for point in stale:
point.payload["is_active"] = False
point.payload["deleted_at"] = deleted_at.isoformat()
return len(stale)
@dataclass
class FakePointRepository:
"""An in-memory `PointRepository`.
Stores `Point` models keyed by point id. The filter, ordering, and version
semantics below are reimplemented in Python rather than stubbed, because
every service this fake stands behind is being tested *for* those semantics:
a stub that returned points in insertion order would pass a reorder test
that real Qdrant fails.
Three behaviours are modelled deliberately:
- `is_active` is filtered out unless `include_inactive`, matching ADR-0002's
implied read filter.
- `list_by_file` sorts by `order_id` and paginates by value, the same cursor
mechanism the Qdrant adapter uses, so a test can catch a cursor that skips
or repeats.
- `apply_patches` honours `expected_version` by silently matching nothing on
a mismatch — the no-op-not-error behaviour Qdrant's filtered `set_payload`
actually has, which is what makes the service's post-check necessary.
"""
points: dict[str, Point] = field(default_factory=dict)
patch_batches: list[list[PayloadPatch]] = field(default_factory=list)
fail_next: bool = False
def add(self, point: Point) -> None:
"""Seed a point. Test-only helper, not part of the port."""
self.points[str(point.point_id)] = point
def _visible(
self,
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> list[Point]:
return [
point
for point in self.points.values()
if point.tenant_id == tenant_id
and (domain is None or point.domain == domain)
and (file_id is None or point.file_id == file_id)
and (include_inactive or point.is_active)
]
async def get(
self, *, tenant_id: uuid.UUID, point_id: uuid.UUID, with_vectors: bool = False
) -> Point | None:
point = self.points.get(str(point_id))
if point is None or point.tenant_id != tenant_id:
return None
return point if with_vectors else point.model_copy(update={"vectors": None})
async def get_many(
self, *, tenant_id: uuid.UUID, point_ids: Sequence[uuid.UUID]
) -> tuple[Point, ...]:
wanted = {str(point_id) for point_id in point_ids}
return tuple(
point
for key, point in self.points.items()
if key in wanted and point.tenant_id == tenant_id
)
async def list_by_file(
self,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID,
limit: int,
cursor: str | None = None,
include_inactive: bool = False,
) -> PointPage:
ordered = sorted(
self._visible(tenant_id=tenant_id, file_id=file_id, include_inactive=include_inactive),
key=lambda point: point.order_id,
)
if cursor is not None:
ordered = [point for point in ordered if point.order_id > float(cursor)]
page = tuple(ordered[:limit])
next_cursor = repr(page[-1].order_id) if len(page) == limit else None
return PointPage(points=page, next_cursor=next_cursor)
async def count(
self,
*,
tenant_id: uuid.UUID,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> int:
return len(
self._visible(
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
)
async def keyword_search(
self,
*,
tenant_id: uuid.UUID,
query: str,
limit: int,
cursor: str | None = None,
domain: str | None = None,
file_id: uuid.UUID | None = None,
include_inactive: bool = False,
) -> PointPage:
# Whole-token containment, lowercased: an approximation of Qdrant's
# full-text index good enough to tell "matched" from "did not", which is
# all the service layer branches on. Ranking is not modelled because the
# real index does not rank either (ADR-0002).
terms = query.lower().split()
matches = [
point
for point in self._visible(
tenant_id=tenant_id,
domain=domain,
file_id=file_id,
include_inactive=include_inactive,
)
if all(term in point.content.lower().split() for term in terms)
]
start = int(cursor) if cursor is not None else 0
page = tuple(matches[start : start + limit])
next_start = start + limit
return PointPage(
points=page, next_cursor=str(next_start) if next_start < len(matches) else None
)
async def apply_patches(self, *, tenant_id: uuid.UUID, patches: Sequence[PayloadPatch]) -> None:
self.patch_batches.append(list(patches))
if self.fail_next:
self.fail_next = False
raise RuntimeError("simulated point repository failure")
for patch in patches:
point = self.points.get(str(patch.point_id))
if point is None or point.tenant_id != tenant_id:
continue
if patch.expected_version is not None and point.version != patch.expected_version:
continue
# Re-validated rather than `model_copy`d, because a patch payload
# carries wire values (an ISO `deleted_at`, a stringified
# `previous_chunk_id`) exactly as it would reach Qdrant. Copying
# without validation would leave the fake holding a `str` where a
# read from real Qdrant returns a `datetime`, and a service bug that
# depends on the difference would pass here and fail in production.
updated = point.model_dump()
updated.update(patch.payload)
self.points[str(patch.point_id)] = Point.model_validate(updated)

View File

@@ -1,39 +0,0 @@
from collections.abc import Iterator
import pytest
from testcontainers.community.minio import MinioContainer
from src.config import MinioSettings
from src.infrastructure.minio.client import create_client
_BUCKET = "test-source-files"
@pytest.fixture(scope="session")
def minio_container() -> Iterator[MinioContainer]:
with MinioContainer() as container:
yield container
@pytest.fixture(scope="session")
def minio_settings(minio_container: MinioContainer) -> MinioSettings:
config = minio_container.get_config()
# Pinned to IPv4 for the same reason as postgres_url (see
# tests/integration/postgres/conftest.py): `localhost` resolves to `::1`
# first, but Docker only publishes the mapped port on IPv4, so the
# connection hangs instead of failing.
endpoint = config["endpoint"].replace("localhost:", "127.0.0.1:")
return MinioSettings(
endpoint=endpoint,
access_key=config["access_key"],
secret_key=config["secret_key"],
secure=False,
bucket=_BUCKET,
)
@pytest.fixture(scope="session", autouse=True)
def _ensure_bucket(minio_settings: MinioSettings) -> None:
client = create_client(minio_settings)
if not client.bucket_exists(minio_settings.bucket):
client.make_bucket(minio_settings.bucket)

View File

@@ -1,98 +0,0 @@
from collections.abc import AsyncIterator, Iterator
import pytest
import pytest_asyncio
from alembic.command import upgrade
from alembic.config import Config
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
from testcontainers.community.postgres import PostgresContainer
from src.config import PostgresSettings
from src.infrastructure.postgres.database import create_engine
def _alembic_config(database_url: str) -> Config:
config = Config("alembic.ini")
config.set_main_option("sqlalchemy.url", database_url)
return config
def _settings_from_url(url: str) -> PostgresSettings:
# testcontainers returns postgresql+asyncpg://user:pass@host:port/db ;
# PostgresSettings builds its own dsn from parts, so parse the parts back out.
without_scheme = url.split("://", 1)[1]
creds, hostpart = without_scheme.split("@", 1)
user, password = creds.split(":", 1)
hostport, db = hostpart.split("/", 1)
host, port = hostport.split(":", 1)
return PostgresSettings(host=host, port=int(port), user=user, password=password, db=db)
@pytest.fixture(scope="session")
def postgres_container() -> Iterator[PostgresContainer]:
with PostgresContainer("postgres:17", driver="asyncpg") as container:
yield container
@pytest.fixture(scope="session")
def postgres_url(postgres_container: PostgresContainer) -> str:
"""The container's URL, pinned to IPv4.
Testcontainers reports the host as `localhost`, which resolves to `::1`
before `127.0.0.1`. Docker publishes the mapped port on IPv4 only, and the
IPv6 SYN is dropped rather than refused, so asyncpg blocks on the first
address until its connect timeout instead of falling back to the second --
the connection does not fail, it hangs.
"""
return postgres_container.get_connection_url().replace("@localhost:", "@127.0.0.1:")
@pytest.fixture(scope="session")
def migrated_postgres_url(postgres_url: str) -> str:
"""The container's URL, after Alembic has created the schema on it once."""
upgrade(_alembic_config(postgres_url), "head")
return postgres_url
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def postgres_engine(migrated_postgres_url: str) -> AsyncIterator[AsyncEngine]:
engine = create_engine(_settings_from_url(migrated_postgres_url))
try:
yield engine
finally:
await engine.dispose()
@pytest_asyncio.fixture(loop_scope="session")
async def db_sessionmaker(
postgres_engine: AsyncEngine,
) -> AsyncIterator[async_sessionmaker[AsyncSession]]:
"""A session *factory* per test, bound to a rolled-back outer transaction.
Every session it produces shares one connection/outer transaction, so
writes `commit()`ed by one session are visible to the next -- needed for
code under test that opens more than one session per operation (auth
resolution, the ADR-0017 two-phase upload) -- while the whole test's
writes still roll back together at teardown (ADR-0016: isolate data per
test).
"""
async with postgres_engine.connect() as connection:
outer_transaction = await connection.begin()
sessionmaker = async_sessionmaker(
bind=connection, expire_on_commit=False, join_transaction_mode="create_savepoint"
)
yield sessionmaker
await outer_transaction.rollback()
@pytest_asyncio.fixture(loop_scope="session")
async def db_session(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> AsyncIterator[AsyncSession]:
"""One session per test, bound to a rolled-back outer transaction.
Isolates each test's writes (ADR-0016: isolate data per test) without
needing a fresh container or unique keys per test.
"""
async with db_sessionmaker() as session:
yield session

View File

@@ -0,0 +1,134 @@
"""`resolve_auth_context` emits `auth.succeeded`/`auth.failed` (ADR-0011).
This runs on every authenticated request, so every rejection reason needs a
distinguishable log event -- previously none of them logged anything.
"""
from collections.abc import MutableMapping
from typing import Any
import pytest
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.errors import InvalidApiKeyError, TenantInactiveError
from src.application.auth.service import resolve_auth_context
from tests.support.factories import create_api_key, create_tenant
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
def _events_by_name(
logs: list[MutableMapping[str, Any]], name: str
) -> list[MutableMapping[str, Any]]:
return [entry for entry in logs if entry.get("event") == name]
async def test_valid_key_emits_auth_succeeded(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
_, full_key = await create_api_key(db_session, tenant=tenant, scopes=["files:write"])
await db_session.commit()
with structlog.testing.capture_logs() as logs:
auth = await resolve_auth_context(db_sessionmaker, full_key)
succeeded = _events_by_name(logs, "auth.succeeded")
assert len(succeeded) == 1
assert succeeded[0]["tenant_id"] == str(auth.tenant_id)
assert succeeded[0]["api_key_id"] == str(auth.api_key_id)
assert _events_by_name(logs, "auth.failed") == []
async def test_malformed_token_emits_auth_failed(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> None:
with structlog.testing.capture_logs() as logs, pytest.raises(InvalidApiKeyError):
await resolve_auth_context(db_sessionmaker, "not-a-bearer-token-at-all")
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert failed[0]["reason"] == "malformed_key"
async def test_wrong_secret_emits_auth_failed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
api_key, _ = await create_api_key(db_session, tenant=tenant)
await db_session.commit()
with structlog.testing.capture_logs() as logs, pytest.raises(InvalidApiKeyError):
await resolve_auth_context(db_sessionmaker, f"sk_{api_key.key_prefix}_wrong-secret")
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert failed[0]["reason"] == "unknown_key"
async def test_unknown_prefix_emits_auth_failed(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> None:
with structlog.testing.capture_logs() as logs, pytest.raises(InvalidApiKeyError):
await resolve_auth_context(db_sessionmaker, "sk_doesnotexist_secret")
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert failed[0]["reason"] == "unknown_key"
async def test_revoked_key_emits_auth_failed_with_key_status(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
_, full_key = await create_api_key(db_session, tenant=tenant, status="revoked")
await db_session.commit()
with structlog.testing.capture_logs() as logs, pytest.raises(InvalidApiKeyError):
await resolve_auth_context(db_sessionmaker, full_key)
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert failed[0]["reason"] == "key_inactive"
assert failed[0]["key_status"] == "revoked"
async def test_suspended_tenant_emits_auth_failed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session, status="suspended")
_, full_key = await create_api_key(db_session, tenant=tenant)
await db_session.commit()
with structlog.testing.capture_logs() as logs, pytest.raises(TenantInactiveError):
await resolve_auth_context(db_sessionmaker, full_key)
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert failed[0]["reason"] == "tenant_inactive"
assert failed[0]["tenant_id"] == str(tenant.id)
async def test_auth_failed_never_logs_the_secret(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""ADR-0011's redaction rule: never log plaintext API keys. `key_prefix`
is the non-secret lookup portion (same distinction `ApiKey.key_prefix`
makes); the secret itself must not appear in any field's value.
"""
tenant = await create_tenant(db_session)
api_key, _ = await create_api_key(db_session, tenant=tenant)
await db_session.commit()
wrong_secret = "definitely-not-the-real-secret"
with structlog.testing.capture_logs() as logs, pytest.raises(InvalidApiKeyError):
await resolve_auth_context(db_sessionmaker, f"sk_{api_key.key_prefix}_{wrong_secret}")
failed = _events_by_name(logs, "auth.failed")
assert len(failed) == 1
assert wrong_secret not in str(failed[0])

View File

@@ -0,0 +1,193 @@
"""`/v1/domains` over HTTP, against real Postgres (ADR-0008, ADR-0009).
The property worth testing at this layer is the scope boundary: an upload key
must not be able to create domains, or the allowlist stops preventing anything.
"""
from collections.abc import AsyncIterator
import pytest
import pytest_asyncio
from asgi_lifespan import LifespanManager
from httpx import ASGITransport, AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.bootstrap.dependencies import get_sessionmaker
from src.config import Settings
from src.main import create_app
from tests.support.factories import create_api_key, create_tenant, create_tenant_domain
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
@pytest_asyncio.fixture(loop_scope="session")
async def api_client(
settings: Settings, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> AsyncIterator[AsyncClient]:
"""The real app, with only the database swapped for the test's session
factory -- so routing, auth, scopes, and the error envelope are exercised.
"""
app = create_app(settings)
app.dependency_overrides[get_sessionmaker] = lambda: db_sessionmaker
async with (
LifespanManager(app) as manager,
AsyncClient(transport=ASGITransport(app=manager.app), base_url="http://test") as client,
):
yield client
def _auth(token: str) -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
async def test_create_domain_returns_201_and_lists_it(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
_, token = await create_api_key(
db_session, tenant=tenant, scopes=["domains:read", "domains:write"]
)
await db_session.commit()
created = await api_client.post(
"/v1/domains",
json={"domain": "fire", "display_name": "Fire insurance"},
headers=_auth(token),
)
listed = await api_client.get("/v1/domains", headers=_auth(token))
assert created.status_code == 201
assert created.json()["domain"] == "fire"
assert [item["domain"] for item in listed.json()["domains"]] == ["fire"]
async def test_create_domain_requires_the_domains_write_scope(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
"""An upload key creating domains would defeat the allowlist entirely."""
tenant = await create_tenant(db_session)
_, token = await create_api_key(db_session, tenant=tenant, scopes=["files:write"])
await db_session.commit()
response = await api_client.post(
"/v1/domains",
json={"domain": "fire", "display_name": "Fire"},
headers=_auth(token),
)
assert response.status_code == 403
assert response.json()["error"]["code"] == "missing_scope"
async def test_list_domains_never_shows_another_tenants_domains(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
owner = await create_tenant(db_session)
other = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=owner, domain="fire")
_, other_token = await create_api_key(db_session, tenant=other, scopes=["domains:read"])
await db_session.commit()
response = await api_client.get("/v1/domains", headers=_auth(other_token))
assert response.status_code == 200
assert response.json()["domains"] == []
async def test_create_domain_rejects_a_duplicate_with_409(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
_, token = await create_api_key(db_session, tenant=tenant, scopes=["domains:write"])
await db_session.commit()
response = await api_client.post(
"/v1/domains",
json={"domain": "fire", "display_name": "Fire"},
headers=_auth(token),
)
assert response.status_code == 409
assert response.json()["error"]["code"] == "conflict"
async def test_create_domain_rejects_a_malformed_key(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
_, token = await create_api_key(db_session, tenant=tenant, scopes=["domains:write"])
await db_session.commit()
response = await api_client.post(
"/v1/domains",
json={"domain": "Fire Insurance!", "display_name": "Fire"},
headers=_auth(token),
)
assert response.status_code == 422
async def test_patch_domain_cannot_rename_the_key(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
"""`domain` is not part of the update schema -- it is denormalized into
every point payload, so renaming it is a migration, not an edit.
"""
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
_, token = await create_api_key(
db_session, tenant=tenant, scopes=["domains:read", "domains:write"]
)
await db_session.commit()
response = await api_client.patch(
"/v1/domains/fire",
json={"display_name": "Fire & perils", "domain": "renamed"},
headers=_auth(token),
)
assert response.status_code == 200
assert response.json()["domain"] == "fire"
assert response.json()["display_name"] == "Fire & perils"
async def test_delete_domain_disables_it_without_removing_it(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
_, token = await create_api_key(
db_session, tenant=tenant, scopes=["domains:read", "domains:write"]
)
await db_session.commit()
deleted = await api_client.delete("/v1/domains/fire", headers=_auth(token))
default_list = await api_client.get("/v1/domains", headers=_auth(token))
full_list = await api_client.get(
"/v1/domains", params={"include_disabled": True}, headers=_auth(token)
)
assert deleted.status_code == 200
assert deleted.json()["status"] == "disabled"
assert default_list.json()["domains"] == []
assert [item["domain"] for item in full_list.json()["domains"]] == ["fire"]
async def test_patch_unknown_domain_returns_400(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
_, token = await create_api_key(db_session, tenant=tenant, scopes=["domains:write"])
await db_session.commit()
response = await api_client.patch(
"/v1/domains/absent", json={"display_name": "x"}, headers=_auth(token)
)
assert response.status_code == 400
assert response.json()["error"]["code"] == "unknown_domain"

View File

@@ -0,0 +1,184 @@
"""Tenant-domain management and the upload-time allowlist (ADR-0009)."""
import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.domains import (
DomainAlreadyExistsError,
UnknownDomainError,
create_domain,
ensure_domain_allowed,
list_domains,
set_domain_status,
update_domain,
)
from tests.support.factories import create_tenant, create_tenant_domain
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
async def test_ensure_domain_allowed_passes_for_a_registered_active_domain(
db_session: AsyncSession,
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fire")
async def test_ensure_domain_allowed_rejects_an_unregistered_domain(
db_session: AsyncSession,
) -> None:
"""The typo case: `fier` must not silently become a new Qdrant partition."""
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
with pytest.raises(UnknownDomainError, match="fier"):
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fier")
async def test_ensure_domain_allowed_rejects_a_disabled_domain(
db_session: AsyncSession,
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire", status="disabled")
with pytest.raises(UnknownDomainError, match="disabled"):
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fire")
async def test_ensure_domain_allowed_rejects_another_tenants_domain(
db_session: AsyncSession,
) -> None:
"""Domain lists are per-tenant; one tenant's `fire` is not another's."""
owner = await create_tenant(db_session)
other = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=owner, domain="fire")
with pytest.raises(UnknownDomainError):
await ensure_domain_allowed(db_session, tenant_id=other.id, domain="fire")
async def test_tenants_hold_independent_domain_sets_of_different_sizes(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
big = await create_tenant(db_session)
small = await create_tenant(db_session)
for index in range(14):
await create_tenant_domain(db_session, tenant=big, domain=f"line-{index:02d}")
for index in range(6):
await create_tenant_domain(db_session, tenant=small, domain=f"line-{index:02d}")
await db_session.commit()
assert len(await list_domains(db_sessionmaker, tenant_id=big.id)) == 14
assert len(await list_domains(db_sessionmaker, tenant_id=small.id)) == 6
async def test_create_domain_then_upload_is_allowed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await db_session.commit()
created = await create_domain(
db_sessionmaker, tenant_id=tenant.id, domain="car", display_name="Car insurance"
)
assert created.domain == "car"
assert created.status == "active"
async with db_sessionmaker() as session:
await ensure_domain_allowed(session, tenant_id=tenant.id, domain="car")
async def test_create_domain_rejects_a_duplicate_key_for_the_same_tenant(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
with pytest.raises(DomainAlreadyExistsError):
await create_domain(
db_sessionmaker, tenant_id=tenant.id, domain="fire", display_name="Fire again"
)
async def test_create_domain_allows_the_same_key_for_different_tenants(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
first = await create_tenant(db_session)
second = await create_tenant(db_session)
await db_session.commit()
await create_domain(db_sessionmaker, tenant_id=first.id, domain="fire", display_name="Fire")
await create_domain(db_sessionmaker, tenant_id=second.id, domain="fire", display_name="Fire")
assert len(await list_domains(db_sessionmaker, tenant_id=first.id)) == 1
assert len(await list_domains(db_sessionmaker, tenant_id=second.id)) == 1
async def test_update_domain_changes_only_the_display_name(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
updated = await update_domain(
db_sessionmaker, tenant_id=tenant.id, domain="fire", display_name="Fire & perils"
)
assert updated.display_name == "Fire & perils"
# The key is immutable: it is denormalized into every point payload.
assert updated.domain == "fire"
async def test_disabling_a_domain_blocks_new_uploads_without_deleting_it(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
disabled = await set_domain_status(
db_sessionmaker, tenant_id=tenant.id, domain="fire", status="disabled"
)
assert disabled.status == "disabled"
async with db_sessionmaker() as session:
with pytest.raises(UnknownDomainError):
await ensure_domain_allowed(session, tenant_id=tenant.id, domain="fire")
# Still there, just hidden from the default listing.
assert await list_domains(db_sessionmaker, tenant_id=tenant.id) == []
assert len(await list_domains(db_sessionmaker, tenant_id=tenant.id, include_disabled=True)) == 1
async def test_re_enabling_a_domain_restores_uploads(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire", status="disabled")
await db_session.commit()
await set_domain_status(db_sessionmaker, tenant_id=tenant.id, domain="fire", status="active")
async with db_sessionmaker() as session:
await ensure_domain_allowed(session, tenant_id=tenant.id, domain="fire")
async def test_update_domain_rejects_another_tenants_domain(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
owner = await create_tenant(db_session)
other = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=owner, domain="fire")
await db_session.commit()
with pytest.raises(UnknownDomainError):
await update_domain(
db_sessionmaker, tenant_id=other.id, domain="fire", display_name="hijacked"
)

View File

@@ -0,0 +1,142 @@
"""`domains/service.py` emits log events for the allowlist rejection and every
mutation (ADR-0011). `ensure_domain_allowed` is the one that matters most: it
runs before any `ingestion_jobs` row exists, so without its own log a rejected
upload leaves no operational trace at all.
"""
from collections.abc import MutableMapping
from typing import Any
import pytest
import structlog
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.domains import (
DomainAlreadyExistsError,
UnknownDomainError,
create_domain,
ensure_domain_allowed,
set_domain_status,
update_domain,
)
from tests.support.factories import create_tenant, create_tenant_domain
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
def _events_by_name(
logs: list[MutableMapping[str, Any]], name: str
) -> list[MutableMapping[str, Any]]:
return [entry for entry in logs if entry.get("event") == name]
async def test_ensure_domain_allowed_logs_nothing_when_the_domain_is_active(
db_session: AsyncSession,
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
with structlog.testing.capture_logs() as logs:
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fire")
assert _events_by_name(logs, "domain.rejected") == []
async def test_ensure_domain_allowed_logs_rejection_for_an_unregistered_domain(
db_session: AsyncSession,
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
with structlog.testing.capture_logs() as logs, pytest.raises(UnknownDomainError):
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fier")
rejected = _events_by_name(logs, "domain.rejected")
assert len(rejected) == 1
assert rejected[0]["reason"] == "unregistered"
assert rejected[0]["domain"] == "fier"
assert rejected[0]["tenant_id"] == str(tenant.id)
async def test_ensure_domain_allowed_logs_rejection_for_a_disabled_domain(
db_session: AsyncSession,
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire", status="disabled")
with structlog.testing.capture_logs() as logs, pytest.raises(UnknownDomainError):
await ensure_domain_allowed(db_session, tenant_id=tenant.id, domain="fire")
rejected = _events_by_name(logs, "domain.rejected")
assert len(rejected) == 1
assert rejected[0]["reason"] == "disabled"
async def test_create_domain_emits_domain_created(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await db_session.commit()
with structlog.testing.capture_logs() as logs:
await create_domain(
db_sessionmaker, tenant_id=tenant.id, domain="car", display_name="Car insurance"
)
created = _events_by_name(logs, "domain.created")
assert len(created) == 1
assert created[0]["domain"] == "car"
async def test_create_domain_duplicate_does_not_emit_domain_created(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
with structlog.testing.capture_logs() as logs, pytest.raises(DomainAlreadyExistsError):
await create_domain(
db_sessionmaker, tenant_id=tenant.id, domain="fire", display_name="Fire again"
)
assert _events_by_name(logs, "domain.created") == []
async def test_update_domain_emits_domain_updated(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
with structlog.testing.capture_logs() as logs:
await update_domain(
db_sessionmaker, tenant_id=tenant.id, domain="fire", display_name="Fire & perils"
)
updated = _events_by_name(logs, "domain.updated")
assert len(updated) == 1
assert updated[0]["domain"] == "fire"
async def test_set_domain_status_emits_domain_status_changed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
await create_tenant_domain(db_session, tenant=tenant, domain="fire")
await db_session.commit()
with structlog.testing.capture_logs() as logs:
await set_domain_status(
db_sessionmaker, tenant_id=tenant.id, domain="fire", status="disabled"
)
changed = _events_by_name(logs, "domain.status_changed")
assert len(changed) == 1
assert changed[0]["domain"] == "fire"
assert changed[0]["status"] == "disabled"

View File

@@ -10,6 +10,7 @@ pytestmark = [
EXPECTED_TABLES = {
"tenants",
"tenant_domains",
"api_keys",
"source_files",
"ingestion_jobs",
@@ -25,3 +26,17 @@ async def test_migrations_create_schema_from_empty_database(postgres_engine: Asy
)
assert EXPECTED_TABLES.issubset(set(table_names))
async def test_tenant_domains_enforces_one_row_per_tenant_and_key(
postgres_engine: AsyncEngine,
) -> None:
"""The unique constraint is what stops the same domain being registered
twice for a tenant while still letting two tenants share a key.
"""
async with postgres_engine.connect() as connection:
constraints = await connection.run_sync(
lambda sync_conn: inspect(sync_conn).get_unique_constraints("tenant_domains")
)
assert any(constraint["column_names"] == ["tenant_id", "domain"] for constraint in constraints)

View File

@@ -0,0 +1,95 @@
"""Tenant provisioning against real Postgres (ADR-0009).
The property worth a real database here is the one a fake cannot show: the
issued key authenticates through the *production* auth path, and the row it
authenticates against holds no plaintext.
"""
import pytest
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.service import resolve_auth_context
from src.application.domains import ensure_domain_allowed
from src.application.domains.errors import UnknownDomainError
from src.application.tenants import provision_tenant
from src.infrastructure.postgres.models.api_key import ApiKey
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
async def test_provision_tenant_issues_a_key_that_authenticates(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> None:
result = await provision_tenant(db_sessionmaker, slug="acme", domains=("fire",))
auth = await resolve_auth_context(db_sessionmaker, result.api_key)
assert auth.tenant_id == result.tenant_id
assert auth.tenant_slug == "acme"
assert auth.api_key_id == result.api_key_id
assert "files:write" in auth.scopes
async def test_provision_tenant_persists_only_the_key_hash(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""A plaintext key in Postgres would make every later hashing decision moot."""
result = await provision_tenant(db_sessionmaker, slug="acme")
stored = (
await db_session.execute(select(ApiKey).where(ApiKey.id == result.api_key_id))
).scalar_one()
assert result.api_key not in stored.key_hash
assert stored.key_hash != result.api_key
assert stored.key_prefix == result.api_key_prefix
assert result.api_key.startswith(f"sk_{result.api_key_prefix}_")
async def test_provision_tenant_registers_domains_so_uploads_are_allowed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
result = await provision_tenant(db_sessionmaker, slug="acme", domains=("fire",))
await ensure_domain_allowed(db_session, tenant_id=result.tenant_id, domain="fire")
with pytest.raises(UnknownDomainError):
await ensure_domain_allowed(db_session, tenant_id=result.tenant_id, domain="life")
assert result.domains_created == ("fire",)
async def test_provision_tenant_rerun_reuses_the_tenant_and_issues_a_new_key(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> None:
"""Adding a key to a live tenant must not need a different command."""
first = await provision_tenant(db_sessionmaker, slug="acme", domains=("fire",))
second = await provision_tenant(db_sessionmaker, slug="acme", domains=("fire", "life"))
assert second.tenant_id == first.tenant_id
assert second.tenant_created is False
assert second.api_key_id != first.api_key_id
assert second.domains_created == ("life",)
assert second.domains_existing == ("fire",)
# Both keys stay valid -- reprovisioning adds a key, it does not rotate one.
assert (await resolve_auth_context(db_sessionmaker, first.api_key)).tenant_id == first.tenant_id
assert (
await resolve_auth_context(db_sessionmaker, second.api_key)
).tenant_id == first.tenant_id
async def test_provision_tenant_honours_requested_scopes(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> None:
"""A key scoped to uploads must not be able to manage the allowlist."""
result = await provision_tenant(db_sessionmaker, slug="acme", scopes=("files:write",))
auth = await resolve_auth_context(db_sessionmaker, result.api_key)
assert auth.scopes == frozenset({"files:write"})
assert not auth.has_scope("domains:write")

View File

@@ -11,14 +11,25 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.context import AuthContext
from src.application.domains import UnknownDomainError
from src.application.files.models import UploadResult
from src.application.files.upload import upload_source_file
from src.application.ingestion.errors import IngestionAtCapacityError, IngestionTimeoutError
from src.application.ingestion.errors import (
IngestionAtCapacityError,
IngestionTimeoutError,
PointIndexingError,
)
from src.application.ports.object_storage import ObjectStorage
from src.config import ChunkingSettings, IngestionSettings
from src.application.ports.point_storage import PointStorage
from src.config import ChunkingSettings, IngestionSettings, QdrantSettings
from src.infrastructure.postgres.models.ingestion_job import IngestionJob
from tests.fakes import FakeDenseEmbedder, FakeObjectStorage, FakeSparseEmbedder
from tests.support.factories import create_api_key, create_tenant
from tests.fakes import (
FakeDenseEmbedder,
FakeObjectStorage,
FakePointStorage,
FakeSparseEmbedder,
)
from tests.support.factories import create_api_key, create_tenant, create_tenant_domain
pytestmark = [
pytest.mark.integration,
@@ -34,6 +45,7 @@ async def _upload(
sessionmaker: async_sessionmaker[AsyncSession],
storage: ObjectStorage,
auth: AuthContext,
point_storage: PointStorage | None = None,
domain: str = "general",
filename: str = "report.csv",
data: bytes = _CSV_BYTES,
@@ -41,12 +53,14 @@ async def _upload(
return await upload_source_file(
sessionmaker=sessionmaker,
storage=storage,
point_storage=point_storage if point_storage is not None else FakePointStorage(),
auth=auth,
domain=domain,
filename=filename,
data=data,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[
@@ -57,9 +71,12 @@ async def _upload(
)
async def _auth_for(db_session: AsyncSession) -> AuthContext:
async def _auth_for(db_session: AsyncSession, *, domain: str = "general") -> AuthContext:
tenant = await create_tenant(db_session)
api_key, _ = await create_api_key(db_session, tenant=tenant)
# Uploads reject an unregistered domain (ADR-0009), so register the one the
# helper below uploads to.
await create_tenant_domain(db_session, tenant=tenant, domain=domain)
await db_session.commit()
return AuthContext(
tenant_id=tenant.id,
@@ -83,7 +100,7 @@ async def test_upload_source_file_commits_running_job_before_storage_write(
result = await _upload(sessionmaker=db_sessionmaker, storage=storage, auth=auth)
assert result.status == "succeeded"
assert result.chunks_indexed == 0
assert result.chunks_indexed == 1
assert result.is_new_attempt
async with db_sessionmaker() as verify_session:
@@ -170,12 +187,14 @@ async def test_upload_source_file_timeout_writes_failed_job_and_raises(
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(timeout_seconds=0.05),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[slow_embedder, FakeDenseEmbedder(name="dense_openai")],
@@ -212,12 +231,14 @@ async def test_upload_source_file_at_capacity_rejects_before_any_job_row(
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=concurrency_limiter,
dense_embedders=[
@@ -238,3 +259,193 @@ async def test_upload_source_file_at_capacity_rejects_before_any_job_row(
.all()
)
assert jobs == []
async def test_upload_source_file_indexes_points_and_records_real_counters(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""The Phase 5 payoff: a successful upload is searchable, and the job row
says how many points it wrote rather than a hardcoded zero.
"""
auth = await _auth_for(db_session)
point_storage = FakePointStorage()
result = await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
data=b"name,value\nfirst,1\nsecond,2\nthird,3\n",
)
assert result.status == "succeeded"
assert result.chunks_indexed == len(point_storage.points)
assert result.chunks_indexed > 0
async with db_sessionmaker() as verify_session:
job = await verify_session.get(IngestionJob, result.ingestion_job_id)
assert job is not None
assert job.points_created == result.chunks_indexed
# An upsert cannot tell an insert from an overwrite, so this stays 0.
assert job.points_updated == 0
async def test_upload_source_file_indexes_points_under_the_authenticated_tenant(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
auth = await _auth_for(db_session, domain="fire")
point_storage = FakePointStorage()
result = await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
domain="fire",
)
payloads = [point.payload for point in point_storage.points.values()]
assert payloads
for payload in payloads:
assert payload["tenant_id"] == str(auth.tenant_id)
assert payload["domain"] == "fire"
assert payload["file_id"] == str(result.file_id)
assert payload["created_by"] == f"api_key:{auth.api_key_id}"
async def test_upload_source_file_index_failure_writes_terminal_failed_job(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
auth = await _auth_for(db_session)
point_storage = FakePointStorage(fail_on_batch=0)
with pytest.raises(PointIndexingError):
await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
)
async with db_sessionmaker() as verify_session:
jobs = (await verify_session.execute(select(IngestionJob))).scalars().all()
job = next(job for job in jobs if job.tenant_id == auth.tenant_id)
assert job.status == "failed"
assert job.error_code == "index_failed"
assert job.completed_at is not None
async def test_upload_source_file_failed_index_does_not_soft_delete_existing_points(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""A failed attempt must never remove content from a working index."""
auth = await _auth_for(db_session)
point_storage = FakePointStorage()
await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
data=b"name,value\nfirst,1\nsecond,2\n",
)
# fail_on_batch indexes into upsert_batches, which accumulates across
# uploads -- reset it so "batch 0" means the retry's first batch.
point_storage.deactivate_calls.clear()
point_storage.upsert_batches.clear()
point_storage.fail_on_batch = 0
with pytest.raises(PointIndexingError):
await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
data=b"name,value\nonly,1\n",
)
assert point_storage.deactivate_calls == []
assert all(point.payload["is_active"] is True for point in point_storage.points.values())
async def test_upload_source_file_retry_after_index_failure_produces_no_duplicate_points(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
auth = await _auth_for(db_session)
point_storage = FakePointStorage(fail_on_batch=0)
with pytest.raises(PointIndexingError):
await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
)
point_storage.fail_on_batch = None
retry = await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
point_storage=point_storage,
)
assert retry.status == "succeeded"
assert len(point_storage.points) == retry.chunks_indexed
async with db_sessionmaker() as verify_session:
jobs = (await verify_session.execute(select(IngestionJob))).scalars().all()
tenant_jobs = [job for job in jobs if job.tenant_id == auth.tenant_id]
# A terminal job never returns to `running` (ADR-0017); the retry is a new row.
assert len(tenant_jobs) == 2
assert {job.status for job in tenant_jobs} == {"failed", "succeeded"}
async def test_upload_source_file_rejects_an_unregistered_domain_before_any_write(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""A typo'd domain must fail loudly, not create a new Qdrant partition
whose contents retrieval never queries (ADR-0009).
"""
auth = await _auth_for(db_session, domain="fire")
storage = FakeObjectStorage()
point_storage = FakePointStorage()
with pytest.raises(UnknownDomainError, match="fier"):
await _upload(
sessionmaker=db_sessionmaker,
storage=storage,
auth=auth,
point_storage=point_storage,
domain="fier",
)
# Nothing was written anywhere: no object, no points, and no job row.
assert storage.objects == {}
assert point_storage.points == {}
async with db_sessionmaker() as verify_session:
jobs = (await verify_session.execute(select(IngestionJob))).scalars().all()
assert [job for job in jobs if job.tenant_id == auth.tenant_id] == []
async def test_upload_source_file_rejects_a_disabled_domain(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
tenant = await create_tenant(db_session)
api_key, _ = await create_api_key(db_session, tenant=tenant)
await create_tenant_domain(db_session, tenant=tenant, domain="fire", status="disabled")
await db_session.commit()
auth = AuthContext(
tenant_id=tenant.id,
tenant_slug=tenant.slug,
api_key_id=api_key.id,
scopes=frozenset({"files:write"}),
actor_type="backend",
)
with pytest.raises(UnknownDomainError, match="disabled"):
await _upload(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
auth=auth,
domain="fire",
)

View File

@@ -0,0 +1,223 @@
"""`upload_source_file` emits `ingestion.job.*` log events (ADR-0011).
Every failure branch funnels through `_mark_job_failed`, so this asserts the
log event once per branch rather than re-testing the Postgres job-row
behavior already covered in `test_upload_service.py`. Uses
`structlog.testing.capture_logs()`, which captures events independent of
whichever handlers/renderers happen to be configured in this process.
"""
from collections.abc import MutableMapping
from typing import Any
import pytest
import structlog
from anyio import CapacityLimiter, Semaphore
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.application.auth.context import AuthContext
from src.application.files.upload import upload_source_file
from src.application.ingestion.errors import (
EmbedderError,
IngestionTimeoutError,
PointIndexingError,
)
from src.config import ChunkingSettings, IngestionSettings, QdrantSettings
from tests.fakes import (
FakeDenseEmbedder,
FakeObjectStorage,
FakePointStorage,
FakeSparseEmbedder,
)
from tests.support.factories import create_api_key, create_tenant, create_tenant_domain
pytestmark = [
pytest.mark.integration,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
_CSV_BYTES = b"name,value\nfirst,1\n"
async def _auth_for(db_session: AsyncSession, *, domain: str = "general") -> AuthContext:
tenant = await create_tenant(db_session)
api_key, _ = await create_api_key(db_session, tenant=tenant)
await create_tenant_domain(db_session, tenant=tenant, domain=domain)
await db_session.commit()
return AuthContext(
tenant_id=tenant.id,
tenant_slug=tenant.slug,
api_key_id=api_key.id,
scopes=frozenset({"files:write"}),
actor_type="backend",
)
def _events_by_name(
logs: list[MutableMapping[str, Any]], name: str
) -> list[MutableMapping[str, Any]]:
return [entry for entry in logs if entry.get("event") == name]
async def test_successful_upload_emits_started_and_completed_events(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
auth = await _auth_for(db_session)
with structlog.testing.capture_logs() as logs:
result = await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[
FakeDenseEmbedder(name="dense_nomic"),
FakeDenseEmbedder(name="dense_openai"),
],
sparse_embedder=FakeSparseEmbedder(),
)
started = _events_by_name(logs, "ingestion.job.started")
completed = _events_by_name(logs, "ingestion.job.completed")
assert len(started) == 1
assert started[0]["tenant_id"] == str(auth.tenant_id)
assert started[0]["ingestion_job_id"] == str(result.ingestion_job_id)
assert len(completed) == 1
assert completed[0]["points_upserted"] == result.chunks_indexed
assert _events_by_name(logs, "ingestion.job.failed") == []
async def test_storage_failure_emits_ingestion_job_failed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
auth = await _auth_for(db_session)
with structlog.testing.capture_logs() as logs, pytest.raises(OSError):
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(fail_next=True),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[
FakeDenseEmbedder(name="dense_nomic"),
FakeDenseEmbedder(name="dense_openai"),
],
sparse_embedder=FakeSparseEmbedder(),
)
failed = _events_by_name(logs, "ingestion.job.failed")
assert len(failed) == 1
assert failed[0]["error_code"] == "storage_upload_failed"
assert failed[0]["tenant_id"] == str(auth.tenant_id)
async def test_embedding_failure_emits_ingestion_job_failed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""Previously silent: embedding_failed reached Postgres but never logged."""
auth = await _auth_for(db_session)
failing_embedder = FakeDenseEmbedder(name="dense_nomic", fail_next=True)
with structlog.testing.capture_logs() as logs, pytest.raises(EmbedderError):
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[failing_embedder, FakeDenseEmbedder(name="dense_openai")],
sparse_embedder=FakeSparseEmbedder(),
)
failed = _events_by_name(logs, "ingestion.job.failed")
assert len(failed) == 1
assert failed[0]["error_code"] == "embedding_failed"
async def test_index_failure_emits_ingestion_job_failed(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""Previously silent: index_failed reached Postgres but never logged."""
auth = await _auth_for(db_session)
with structlog.testing.capture_logs() as logs, pytest.raises(PointIndexingError):
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(fail_on_batch=0),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[
FakeDenseEmbedder(name="dense_nomic"),
FakeDenseEmbedder(name="dense_openai"),
],
sparse_embedder=FakeSparseEmbedder(),
)
failed = _events_by_name(logs, "ingestion.job.failed")
assert len(failed) == 1
assert failed[0]["error_code"] == "index_failed"
async def test_timeout_emits_ingestion_job_failed_not_a_duplicate_event(
db_session: AsyncSession, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> None:
"""The old ad-hoc `files.upload.timeout` log is gone -- `_mark_job_failed`
is now the single place a failure is logged, so there is exactly one
`ingestion.job.failed` event, not two events for one failure.
"""
auth = await _auth_for(db_session)
slow_embedder = FakeDenseEmbedder(name="dense_nomic", delay_seconds=10)
with structlog.testing.capture_logs() as logs, pytest.raises(IngestionTimeoutError):
await upload_source_file(
sessionmaker=db_sessionmaker,
storage=FakeObjectStorage(),
point_storage=FakePointStorage(),
auth=auth,
domain="general",
filename="report.csv",
data=_CSV_BYTES,
ingestion_settings=IngestionSettings(timeout_seconds=0.05),
chunking_settings=ChunkingSettings(),
qdrant_settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
concurrency_limiter=Semaphore(2),
dense_embedders=[slow_embedder, FakeDenseEmbedder(name="dense_openai")],
sparse_embedder=FakeSparseEmbedder(),
)
failed = _events_by_name(logs, "ingestion.job.failed")
assert len(failed) == 1
assert failed[0]["error_code"] == "timeout"

View File

@@ -0,0 +1,146 @@
"""`ensure_chunks_collection` against a real Qdrant (ADR-0001).
The assertions that matter most here are the ones for schema properties that
fail *silently* in production: the sparse `modifier=idf` and the pinned dense
dimensions.
"""
import pytest
from qdrant_client import AsyncQdrantClient, models
from src.config import QdrantSettings
from src.infrastructure.qdrant.collection import (
CollectionSchemaMismatchError,
ensure_chunks_collection,
)
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.asyncio(loop_scope="session"),
]
async def test_ensure_chunks_collection_creates_all_four_named_vectors(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
created = await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
assert created is True
info = await qdrant_client.get_collection(qdrant_settings.collection)
vectors = info.config.params.vectors
assert isinstance(vectors, dict)
assert vectors["dense_nomic"].size == 768
assert vectors["dense_openai"].size == 3072
assert vectors["late_interaction"].size == 128
assert vectors["late_interaction"].multivector_config is not None
assert vectors["late_interaction"].on_disk is True
async def test_ensure_chunks_collection_sets_the_sparse_idf_modifier(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""Without this, Qdrant applies no IDF and lexical retrieval silently
degrades -- no error, no warning (ADR-0005).
"""
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
info = await qdrant_client.get_collection(qdrant_settings.collection)
sparse = info.config.params.sparse_vectors
assert sparse is not None
assert sparse["sparse"].modifier == models.Modifier.IDF
async def test_ensure_chunks_collection_creates_the_payload_indexes(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
info = await qdrant_client.get_collection(qdrant_settings.collection)
schema = info.payload_schema
assert set(schema) >= {
"tenant_id",
"domain",
"file_id",
"order_id",
"previous_chunk_id",
"next_chunk_id",
"content",
"is_active",
"chunk_index",
}
# order_id must be numeric: Qdrant's Range/order_by reject keyword payloads.
assert schema["order_id"].data_type == models.PayloadSchemaType.FLOAT
assert schema["tenant_id"].data_type == models.PayloadSchemaType.KEYWORD
# `content` must be TEXT, not KEYWORD: ADR-0002's keyword search is a
# full-text match on it, and a keyword index would only match the entire
# chunk verbatim -- which never happens and would fail silently.
assert schema["content"].data_type == models.PayloadSchemaType.TEXT
assert schema["is_active"].data_type == models.PayloadSchemaType.BOOL
assert schema["chunk_index"].data_type == models.PayloadSchemaType.INTEGER
async def test_ensure_chunks_collection_adds_a_missing_index_to_a_live_collection(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""Payload indexes are additive, unlike vector config.
This is the operational claim the runbook makes when a new index ships:
re-running the bootstrap against an existing collection adds it in place, so
plan 002's `content`/`is_active`/`chunk_index` indexes do not require
recreating a collection that already holds a tenant's points.
"""
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
await qdrant_client.delete_payload_index(
collection_name=qdrant_settings.collection, field_name="content"
)
info = await qdrant_client.get_collection(qdrant_settings.collection)
assert "content" not in info.payload_schema
assert not await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
info = await qdrant_client.get_collection(qdrant_settings.collection)
assert info.payload_schema["content"].data_type == models.PayloadSchemaType.TEXT
async def test_ensure_chunks_collection_is_idempotent(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
assert await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
assert not await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
async def test_ensure_chunks_collection_rejects_a_mismatched_existing_collection(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""A wrong-dimension collection must fail loudly, not be silently accepted."""
await qdrant_client.create_collection(
collection_name=qdrant_settings.collection,
vectors_config={
"dense_nomic": models.VectorParams(size=384, distance=models.Distance.COSINE),
"dense_openai": models.VectorParams(size=3072, distance=models.Distance.COSINE),
"late_interaction": models.VectorParams(size=128, distance=models.Distance.COSINE),
},
sparse_vectors_config={"sparse": models.SparseVectorParams(modifier=models.Modifier.IDF)},
)
with pytest.raises(CollectionSchemaMismatchError, match="768"):
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
async def test_ensure_chunks_collection_rejects_a_collection_without_the_idf_modifier(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
await qdrant_client.create_collection(
collection_name=qdrant_settings.collection,
vectors_config={
"dense_nomic": models.VectorParams(size=768, distance=models.Distance.COSINE),
"dense_openai": models.VectorParams(size=3072, distance=models.Distance.COSINE),
"late_interaction": models.VectorParams(size=128, distance=models.Distance.COSINE),
},
sparse_vectors_config={"sparse": models.SparseVectorParams()},
)
with pytest.raises(CollectionSchemaMismatchError, match="idf"):
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)

View File

@@ -0,0 +1,86 @@
"""`QdrantPointRepository` against the shared `PointRepository` contract.
The identical scenarios run against the in-memory fake in
`tests/unit/application/points/test_point_repository_contract.py`. This file is
the half that decides whether the fake is telling the truth: filter
construction, `order_by: order_id` scroll, cursor pagination, full-text
matching, and the filtered `set_payload` version guard are all Qdrant
behaviours a fake can only approximate, and plan 002 Phase 1's exit criterion is
that the approximation holds.
Seeding differs from the fake's (real points need vectors), so it lives here;
everything asserted lives in `tests/support/point_contract.py`.
"""
import uuid
from collections.abc import Awaitable, Callable
import pytest
from qdrant_client import AsyncQdrantClient, models
from src.config import QdrantSettings
from src.infrastructure.qdrant.collection import ensure_chunks_collection
from src.infrastructure.qdrant.point_repository import QdrantPointRepository
from src.infrastructure.qdrant.points import QdrantPointStorage
from tests.support import point_contract
from tests.support.point_contract import chunk_point_for, seed_specs
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.asyncio(loop_scope="session"),
]
type Scenario = Callable[..., Awaitable[None]]
TENANT_A = uuid.UUID("11111111-1111-4111-8111-111111111111")
TENANT_B = uuid.UUID("22222222-2222-4222-8222-222222222222")
FILE_A = uuid.UUID("33333333-3333-4333-8333-333333333333")
FILE_B = uuid.UUID("44444444-4444-4444-8444-444444444444")
FILE_C = uuid.UUID("55555555-5555-4555-8555-555555555555")
IDS = {
"tenant_a": TENANT_A,
"tenant_b": TENANT_B,
"file_a": FILE_A,
"file_b": FILE_B,
"file_c": FILE_C,
}
async def _seeded(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> QdrantPointRepository:
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
storage = QdrantPointStorage(qdrant_client, collection=qdrant_settings.collection)
await storage.upsert_points(
[chunk_point_for(spec) for spec in seed_specs(TENANT_A, TENANT_B, FILE_A, FILE_B, FILE_C)]
)
# Payload indexes are built asynchronously; without waiting, a full-text or
# ordered query can run against a half-built index and return short.
await _await_indexing(qdrant_client, qdrant_settings.collection)
return QdrantPointRepository(qdrant_client, collection=qdrant_settings.collection)
async def _await_indexing(client: AsyncQdrantClient, collection: str) -> None:
for _ in range(100):
info = await client.get_collection(collection)
if info.status == models.CollectionStatus.GREEN and info.indexed_vectors_count is not None:
return
raise AssertionError(f"collection {collection!r} did not finish indexing")
def _arguments(scenario: Scenario) -> dict[str, uuid.UUID]:
return {name: value for name, value in IDS.items() if name in scenario.__annotations__}
@pytest.mark.parametrize(
"scenario",
[*point_contract.READ_SCENARIOS, *point_contract.WRITE_SCENARIOS],
ids=lambda scenario: scenario.__name__.removeprefix("scenario_"),
)
async def test_qdrant_point_repository_satisfies_the_contract(
scenario: Scenario, qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
repository = await _seeded(qdrant_client, qdrant_settings)
await scenario(repository, **_arguments(scenario))

View File

@@ -0,0 +1,182 @@
"""`QdrantPointStorage` against a real Qdrant (ADR-0001, ADR-0002).
Reads here go through the raw client rather than the port: `PointStorage` is
deliberately write-only, because point reads are plan 002's `/v1/points`
surface. The reads below are the test's own verification, not a preview of an
API this slice ships.
"""
import uuid
from datetime import UTC, datetime
import pytest
from qdrant_client import AsyncQdrantClient, models
from src.application.ingestion.chunking import chunk_id_for
from src.application.ingestion.models import SparseVector
from src.application.points.models import ChunkPoint
from src.config import QdrantSettings
from src.infrastructure.qdrant.collection import (
DENSE_NOMIC_DIMENSIONS,
DENSE_OPENAI_DIMENSIONS,
ensure_chunks_collection,
)
from src.infrastructure.qdrant.points import QdrantPointStorage
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.asyncio(loop_scope="session"),
]
def _point(tenant_id: uuid.UUID, file_id: uuid.UUID, chunk_index: int) -> ChunkPoint:
return ChunkPoint(
point_id=chunk_id_for(file_id, chunk_index),
dense={
"dense_nomic": [0.1] * DENSE_NOMIC_DIMENSIONS,
"dense_openai": [0.2] * DENSE_OPENAI_DIMENSIONS,
},
sparse=SparseVector(indices=[1, 2], values=[0.5, 0.25]),
payload={
"tenant_id": str(tenant_id),
"domain": "fire",
"file_id": str(file_id),
"chunk_id": str(chunk_id_for(file_id, chunk_index)),
"chunk_index": chunk_index,
"order_id": float(chunk_index + 1),
"content": f"chunk {chunk_index}",
"is_active": True,
},
)
async def _storage(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> QdrantPointStorage:
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
return QdrantPointStorage(qdrant_client, collection=qdrant_settings.collection)
async def _count_for_tenant(
client: AsyncQdrantClient, collection: str, tenant_id: uuid.UUID
) -> int:
result = await client.count(
collection_name=collection,
count_filter=models.Filter(
must=[
models.FieldCondition(
key="tenant_id", match=models.MatchValue(value=str(tenant_id))
)
]
),
exact=True,
)
return result.count
async def test_upsert_points_stores_points_readable_under_the_owning_tenant_filter(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
storage = await _storage(qdrant_client, qdrant_settings)
tenant_id, file_id = uuid.uuid4(), uuid.uuid4()
await storage.upsert_points([_point(tenant_id, file_id, i) for i in range(3)])
assert await _count_for_tenant(qdrant_client, qdrant_settings.collection, tenant_id) == 3
async def test_upsert_points_are_invisible_to_another_tenants_filter(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""The Qdrant-level form of "cross-tenant access finds nothing" (ADR-0002)."""
storage = await _storage(qdrant_client, qdrant_settings)
owner, other, file_id = uuid.uuid4(), uuid.uuid4(), uuid.uuid4()
await storage.upsert_points([_point(owner, file_id, i) for i in range(3)])
assert await _count_for_tenant(qdrant_client, qdrant_settings.collection, other) == 0
async def test_upsert_points_is_idempotent_for_deterministic_ids(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
storage = await _storage(qdrant_client, qdrant_settings)
tenant_id, file_id = uuid.uuid4(), uuid.uuid4()
points = [_point(tenant_id, file_id, i) for i in range(4)]
await storage.upsert_points(points)
await storage.upsert_points(points)
assert await _count_for_tenant(qdrant_client, qdrant_settings.collection, tenant_id) == 4
async def test_deactivate_points_from_index_soft_deletes_only_the_tail(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
storage = await _storage(qdrant_client, qdrant_settings)
tenant_id, file_id = uuid.uuid4(), uuid.uuid4()
await storage.upsert_points([_point(tenant_id, file_id, i) for i in range(5)])
deactivated = await storage.deactivate_points_from_index(
tenant_id=tenant_id,
file_id=file_id,
from_chunk_index=2,
deleted_at=datetime.now(UTC),
updated_by="api_key:test",
)
assert deactivated == 3
records, _ = await qdrant_client.scroll(
collection_name=qdrant_settings.collection,
scroll_filter=models.Filter(
must=[models.FieldCondition(key="file_id", match=models.MatchValue(value=str(file_id)))]
),
limit=10,
with_payload=True,
)
by_index = {
record.payload["chunk_index"]: record.payload["is_active"]
for record in records
if record.payload is not None
}
assert by_index == {0: True, 1: True, 2: False, 3: False, 4: False}
# Soft delete, not removal -- the points stay for audit (ADR-0002).
assert await _count_for_tenant(qdrant_client, qdrant_settings.collection, tenant_id) == 5
async def test_deactivate_points_from_index_does_not_touch_another_tenants_points(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""A file_id alone is never authority to mutate (ADR-0002)."""
storage = await _storage(qdrant_client, qdrant_settings)
owner, other, file_id = uuid.uuid4(), uuid.uuid4(), uuid.uuid4()
await storage.upsert_points([_point(owner, file_id, i) for i in range(3)])
deactivated = await storage.deactivate_points_from_index(
tenant_id=other,
file_id=file_id,
from_chunk_index=0,
deleted_at=datetime.now(UTC),
updated_by="api_key:intruder",
)
assert deactivated == 0
async def test_deactivate_points_from_index_returns_zero_when_nothing_is_stale(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
storage = await _storage(qdrant_client, qdrant_settings)
tenant_id, file_id = uuid.uuid4(), uuid.uuid4()
await storage.upsert_points([_point(tenant_id, file_id, i) for i in range(3)])
deactivated = await storage.deactivate_points_from_index(
tenant_id=tenant_id,
file_id=file_id,
from_chunk_index=3,
deleted_at=datetime.now(UTC),
updated_by="api_key:test",
)
assert deactivated == 0

View File

@@ -0,0 +1,395 @@
"""`/v1/points` read paths over HTTP (ADR-0002, ADR-0008).
Needs both containers: Postgres authenticates the key and derives the tenant,
Qdrant holds the points. Testing them together is the point — the invariant this
file exists for is that the tenant Postgres derived is the only one Qdrant is
ever queried with.
Everything below goes through the real app: real routing, real `require_scope`,
real error envelope. Only the session factory is overridden, so a route that
forgot its scope or read a tenant from the query string fails here.
"""
import uuid
from collections.abc import AsyncIterator
import pytest
import pytest_asyncio
from asgi_lifespan import LifespanManager
from httpx import ASGITransport, AsyncClient
from qdrant_client import AsyncQdrantClient
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.bootstrap.dependencies import get_sessionmaker
from src.config import QdrantSettings, Settings
from src.infrastructure.qdrant.collection import ensure_chunks_collection
from src.infrastructure.qdrant.points import QdrantPointStorage
from src.main import create_app
from tests.support.factories import create_api_key, create_tenant
from tests.support.point_contract import SeedSpec, chunk_point_for, point_id_for
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
FILE_ID = uuid.UUID("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa")
OTHER_FILE_ID = uuid.UUID("bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb")
@pytest.fixture
def api_settings(settings: Settings, qdrant_settings: QdrantSettings) -> Settings:
"""The unit-test settings with Qdrant pointed at the real container.
Everything else keeps its closed-port default, so this exercises the point
paths without the lifespan reaching a colleague's Ollama box or OpenAI.
"""
return settings.model_copy(update={"qdrant": qdrant_settings})
@pytest_asyncio.fixture(loop_scope="session")
async def api_client(
api_settings: Settings, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> AsyncIterator[AsyncClient]:
app = create_app(api_settings)
app.dependency_overrides[get_sessionmaker] = lambda: db_sessionmaker
async with (
LifespanManager(app) as manager,
AsyncClient(transport=ASGITransport(app=manager.app), base_url="http://test") as client,
):
yield client
def _auth(token: str) -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
async def _seed(
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
specs: tuple[SeedSpec, ...],
) -> None:
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
storage = QdrantPointStorage(qdrant_client, collection=qdrant_settings.collection)
await storage.upsert_points([chunk_point_for(spec) for spec in specs])
async def _tenant_with_key(
db_session: AsyncSession, *, scopes: list[str] | None = None
) -> tuple[uuid.UUID, str]:
tenant = await create_tenant(db_session)
_, token = await create_api_key(
db_session, tenant=tenant, scopes=scopes if scopes is not None else ["points:read"]
)
await db_session.commit()
return tenant.id, token
async def test_list_points_returns_a_files_points_in_order_id_order(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "first", 1.0),
SeedSpec(tenant_id, FILE_ID, 2, "third", 3.0),
SeedSpec(tenant_id, FILE_ID, 1, "second", 2.0),
),
)
response = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
assert response.status_code == 200
assert [point["content"] for point in response.json()["points"]] == [
"first",
"second",
"third",
]
async def test_list_points_omits_the_tenant_id_from_the_response(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""Echoing `tenant_id` back invites clients to start sending it."""
tenant_id, token = await _tenant_with_key(db_session)
await _seed(qdrant_client, qdrant_settings, (SeedSpec(tenant_id, FILE_ID, 0, "only", 1.0),))
response = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
assert "tenant_id" not in response.json()["points"][0]
async def test_get_point_returns_404_for_another_tenants_point(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""`404`, not `403`: the API must not confirm the id exists (ADR-0016)."""
owner_id, _ = await _tenant_with_key(db_session)
_, intruder_token = await _tenant_with_key(db_session)
await _seed(qdrant_client, qdrant_settings, (SeedSpec(owner_id, FILE_ID, 0, "secret", 1.0),))
response = await api_client.get(
f"/v1/points/{point_id_for(FILE_ID, 0)}", headers=_auth(intruder_token)
)
assert response.status_code == 404
assert response.json()["error"]["code"] == "not_found"
async def test_list_points_returns_nothing_for_another_tenants_file(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
owner_id, _ = await _tenant_with_key(db_session)
_, intruder_token = await _tenant_with_key(db_session)
await _seed(qdrant_client, qdrant_settings, (SeedSpec(owner_id, FILE_ID, 0, "secret", 1.0),))
response = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(intruder_token)
)
assert response.status_code == 200
assert response.json()["points"] == []
async def test_list_points_excludes_inactive_points_unless_asked(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "live", 1.0),
SeedSpec(tenant_id, FILE_ID, 1, "deleted", 2.0, is_active=False),
),
)
default = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
opted_in = await api_client.get(
"/v1/points",
params={"file_id": str(FILE_ID), "include_inactive": True},
headers=_auth(token),
)
assert [point["content"] for point in default.json()["points"]] == ["live"]
assert [point["content"] for point in opted_in.json()["points"]] == ["live", "deleted"]
async def test_list_points_pagination_does_not_skip_or_repeat_under_a_concurrent_insert(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""The defect an offset cursor would have.
Between page one and page two, a point is inserted *behind* the cursor
(`order_id` 0.5, ahead of everything already returned). An offset-based
cursor would shift every later row down by one and hand back `b` a second
time. A value-based cursor is unaffected: the reader has passed 2.0 and the
new row is behind it.
"""
tenant_id, token = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "a", 1.0),
SeedSpec(tenant_id, FILE_ID, 1, "b", 2.0),
SeedSpec(tenant_id, FILE_ID, 2, "c", 3.0),
SeedSpec(tenant_id, FILE_ID, 3, "d", 4.0),
),
)
first = await api_client.get(
"/v1/points",
params={"file_id": str(FILE_ID), "limit": 2},
headers=_auth(token),
)
await _seed(qdrant_client, qdrant_settings, (SeedSpec(tenant_id, FILE_ID, 9, "inserted", 0.5),))
second = await api_client.get(
"/v1/points",
params={
"file_id": str(FILE_ID),
"limit": 2,
"cursor": first.json()["next_cursor"],
},
headers=_auth(token),
)
assert [point["content"] for point in first.json()["points"]] == ["a", "b"]
assert [point["content"] for point in second.json()["points"]] == ["c", "d"]
async def test_count_points_respects_tenant_and_domain_filters(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
other_id, _ = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "one", 1.0),
SeedSpec(tenant_id, OTHER_FILE_ID, 0, "two", 1.0, domain="ops"),
SeedSpec(other_id, FILE_ID, 5, "theirs", 1.0),
),
)
everything = await api_client.get("/v1/points/count", headers=_auth(token))
scoped = await api_client.get(
"/v1/points/count", params={"domain": "ops"}, headers=_auth(token)
)
assert everything.json()["count"] == 2
assert scoped.json()["count"] == 1
async def test_search_points_matches_content_within_the_tenant_only(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
other_id, _ = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "annual leave policy", 1.0),
SeedSpec(tenant_id, FILE_ID, 1, "expense policy", 2.0),
SeedSpec(tenant_id, FILE_ID, 2, "office address", 3.0),
SeedSpec(other_id, OTHER_FILE_ID, 0, "their leave policy", 1.0),
),
)
response = await api_client.get(
"/v1/points/search", params={"q": "policy"}, headers=_auth(token)
)
body = response.json()
assert body["query"] == "policy"
assert {point["content"] for point in body["points"]} == {
"annual leave policy",
"expense policy",
}
async def test_search_points_folds_arabic_letterforms_in_the_query(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""A query typed on an Arabic keyboard must match Persian-folded content.
Ingestion folds U+064A/U+0643 to U+06CC/U+06A9 before storing, so an
unfolded query would match nothing — an empty result rather than an error,
which is exactly the kind of silent failure nobody reports as a bug.
Codepoints, not literals: the two forms are visually identical here.
"""
persian = "مدیریت" # stored, Persian yeh
arabic = "مديريت" # queried, Arabic yeh
assert persian != arabic
tenant_id, token = await _tenant_with_key(db_session)
await _seed(qdrant_client, qdrant_settings, (SeedSpec(tenant_id, FILE_ID, 0, persian, 1.0),))
response = await api_client.get("/v1/points/search", params={"q": arabic}, headers=_auth(token))
assert [point["content"] for point in response.json()["points"]] == [persian]
async def test_point_read_routes_require_the_points_read_scope(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
"""An upload key must not double as a way to read every chunk."""
_, token = await _tenant_with_key(db_session, scopes=["files:write"])
response = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
assert response.status_code == 403
assert response.json()["error"]["code"] == "missing_scope"
async def test_count_and_search_paths_are_not_parsed_as_point_ids(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""Route order regression guard.
`/{point_id}` declared before `/count` would make this `422` — Pydantic
failing to read "count" as a UUID — rather than a count. The bug is one
tidy-up reorder away, and nothing else in the suite would catch it.
Seeds nothing but still bootstraps the collection: querying a collection
that does not exist is a `500`, which `/readyz` is responsible for
preventing, and which would mask the `422` this test is watching for.
"""
_, token = await _tenant_with_key(db_session)
await _seed(qdrant_client, qdrant_settings, ())
count = await api_client.get("/v1/points/count", headers=_auth(token))
search = await api_client.get(
"/v1/points/search", params={"q": "anything"}, headers=_auth(token)
)
assert count.status_code == 200
assert search.status_code == 200
async def test_file_points_route_lists_the_same_points_as_the_query_form(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
await _seed(
qdrant_client,
qdrant_settings,
(
SeedSpec(tenant_id, FILE_ID, 0, "first", 1.0),
SeedSpec(tenant_id, FILE_ID, 1, "second", 2.0),
),
)
by_query = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
by_path = await api_client.get(f"/v1/files/{FILE_ID}/points", headers=_auth(token))
assert by_path.status_code == 200
assert by_path.json() == by_query.json()

View File

@@ -0,0 +1,276 @@
"""Soft delete over HTTP, against real Postgres and real Qdrant (ADR-0002).
The unit tests in `tests/unit/application/points/test_deletion.py` decide
whether the relink logic is right. This file decides whether it is right
*through the stack*: real routing and scope checks, a tenant derived from a real
API key, and a real filtered `set_payload` batch — the last of which is the part
a fake can only approximate, since Qdrant reports success for a patch that
matched nothing.
`DELETE /v1/files/{file_id}` is here too rather than with the upload tests: it
spans both stores, and the assertion that matters is the one about points.
"""
import uuid
from collections.abc import AsyncIterator
import pytest
import pytest_asyncio
from asgi_lifespan import LifespanManager
from httpx import ASGITransport, AsyncClient
from qdrant_client import AsyncQdrantClient
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from src.bootstrap.dependencies import get_sessionmaker
from src.config import QdrantSettings, Settings
from src.infrastructure.postgres.repositories import source_files as source_files_repo
from src.infrastructure.qdrant.collection import ensure_chunks_collection
from src.infrastructure.qdrant.points import QdrantPointStorage
from src.main import create_app
from tests.support.factories import create_api_key, create_source_file, create_tenant
from tests.support.point_contract import SeedSpec, chunk_point_for, point_id_for
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.postgres,
pytest.mark.asyncio(loop_scope="session"),
]
FILE_ID = uuid.UUID("cccccccc-cccc-4ccc-8ccc-cccccccccccc")
@pytest.fixture
def api_settings(settings: Settings, qdrant_settings: QdrantSettings) -> Settings:
return settings.model_copy(update={"qdrant": qdrant_settings})
@pytest_asyncio.fixture(loop_scope="session")
async def api_client(
api_settings: Settings, db_sessionmaker: async_sessionmaker[AsyncSession]
) -> AsyncIterator[AsyncClient]:
app = create_app(api_settings)
app.dependency_overrides[get_sessionmaker] = lambda: db_sessionmaker
async with (
LifespanManager(app) as manager,
AsyncClient(transport=ASGITransport(app=manager.app), base_url="http://test") as client,
):
yield client
def _auth(token: str) -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
async def _tenant_with_key(
db_session: AsyncSession, *, scopes: list[str] | None = None
) -> tuple[uuid.UUID, str]:
tenant = await create_tenant(db_session)
_, token = await create_api_key(
db_session,
tenant=tenant,
scopes=scopes if scopes is not None else ["points:read", "points:write"],
)
await db_session.commit()
return tenant.id, token
async def _seed_chain(
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
*,
tenant_id: uuid.UUID,
file_id: uuid.UUID = FILE_ID,
length: int = 3,
) -> list[uuid.UUID]:
"""A linked run of points, written the way ingestion writes them."""
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
ids = [point_id_for(file_id, index) for index in range(length)]
points = []
for index in range(length):
chunk_point = chunk_point_for(
SeedSpec(tenant_id, file_id, index, f"chunk {index}", float(index + 1))
)
chunk_point.payload["previous_chunk_id"] = str(ids[index - 1]) if index else None
chunk_point.payload["next_chunk_id"] = str(ids[index + 1]) if index + 1 < length else None
points.append(chunk_point)
storage = QdrantPointStorage(qdrant_client, collection=qdrant_settings.collection)
await storage.upsert_points(points)
return ids
async def _read_point(
api_client: AsyncClient, token: str, point_id: uuid.UUID
) -> dict[str, object]:
response = await api_client.get(f"/v1/points/{point_id}", headers=_auth(token))
assert response.status_code == 200
payload: dict[str, object] = response.json()
return payload
async def test_delete_point_deactivates_it_and_relinks_its_neighbours(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
first, middle, last = await _seed_chain(qdrant_client, qdrant_settings, tenant_id=tenant_id)
response = await api_client.delete(f"/v1/points/{middle}", headers=_auth(token))
assert response.status_code == 200
assert response.json()["is_active"] is False
assert response.json()["deleted_at"] is not None
assert (await _read_point(api_client, token, first))["next_chunk_id"] == str(last)
assert (await _read_point(api_client, token, last))["previous_chunk_id"] == str(first)
async def test_delete_point_keeps_the_point_in_qdrant(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""Soft delete means soft: the point is still there, just not listed."""
tenant_id, token = await _tenant_with_key(db_session)
_, middle, _ = await _seed_chain(qdrant_client, qdrant_settings, tenant_id=tenant_id)
await api_client.delete(f"/v1/points/{middle}", headers=_auth(token))
listed = await api_client.get(
"/v1/points", params={"file_id": str(FILE_ID)}, headers=_auth(token)
)
assert str(middle) not in [point["point_id"] for point in listed.json()["points"]]
assert (await _read_point(api_client, token, middle))["is_active"] is False
async def test_delete_point_is_a_noop_success_the_second_time(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant_id, token = await _tenant_with_key(db_session)
first, middle, _ = await _seed_chain(qdrant_client, qdrant_settings, tenant_id=tenant_id)
await api_client.delete(f"/v1/points/{middle}", headers=_auth(token))
version_after_first = (await _read_point(api_client, token, first))["version"]
again = await api_client.delete(f"/v1/points/{middle}", headers=_auth(token))
assert again.status_code == 200
assert again.json()["is_active"] is False
assert (await _read_point(api_client, token, first))["version"] == version_after_first
async def test_delete_point_returns_404_for_another_tenants_point(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""`404`, not `403`, and nothing is deactivated on the way to saying so."""
owner_id, owner_token = await _tenant_with_key(db_session)
_, intruder_token = await _tenant_with_key(db_session)
_, middle, _ = await _seed_chain(qdrant_client, qdrant_settings, tenant_id=owner_id)
response = await api_client.delete(f"/v1/points/{middle}", headers=_auth(intruder_token))
assert response.status_code == 404
assert response.json()["error"]["code"] == "not_found"
assert (await _read_point(api_client, owner_token, middle))["is_active"] is True
async def test_delete_point_requires_the_points_write_scope(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""A read-only key can see a point but must not be able to remove it."""
tenant_id, token = await _tenant_with_key(db_session, scopes=["points:read"])
_, middle, _ = await _seed_chain(qdrant_client, qdrant_settings, tenant_id=tenant_id)
response = await api_client.delete(f"/v1/points/{middle}", headers=_auth(token))
assert response.status_code == 403
assert (await _read_point(api_client, token, middle))["is_active"] is True
async def test_delete_file_deactivates_every_point_and_retires_the_row(
api_client: AsyncClient,
db_session: AsyncSession,
db_sessionmaker: async_sessionmaker[AsyncSession],
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
tenant = await create_tenant(db_session)
_, token = await create_api_key(
db_session, tenant=tenant, scopes=["points:read", "points:write"]
)
file_id = uuid.uuid4()
await create_source_file(db_session, tenant=tenant, source_file_id=file_id)
await db_session.commit()
await _seed_chain(qdrant_client, qdrant_settings, tenant_id=tenant.id, file_id=file_id)
response = await api_client.delete(f"/v1/files/{file_id}", headers=_auth(token))
assert response.status_code == 200
assert response.json()["points_soft_deleted"] == 3
listed = await api_client.get(
"/v1/points", params={"file_id": str(file_id)}, headers=_auth(token)
)
assert listed.json()["points"] == []
async with db_sessionmaker() as session:
row = await source_files_repo.get_by_id(
session, tenant_id=tenant.id, source_file_id=file_id
)
assert row is not None
assert row.status == "soft_deleted"
assert row.deleted_at is not None
async def test_delete_file_returns_404_for_another_tenants_file(
api_client: AsyncClient,
db_session: AsyncSession,
qdrant_client: AsyncQdrantClient,
qdrant_settings: QdrantSettings,
) -> None:
"""The authorization check runs before the sweep, so nothing is deactivated."""
owner = await create_tenant(db_session)
_, owner_token = await create_api_key(
db_session, tenant=owner, scopes=["points:read", "points:write"]
)
intruder = await create_tenant(db_session)
_, intruder_token = await create_api_key(
db_session, tenant=intruder, scopes=["points:read", "points:write"]
)
file_id = uuid.uuid4()
await create_source_file(db_session, tenant=owner, source_file_id=file_id)
await db_session.commit()
await _seed_chain(qdrant_client, qdrant_settings, tenant_id=owner.id, file_id=file_id)
response = await api_client.delete(f"/v1/files/{file_id}", headers=_auth(intruder_token))
assert response.status_code == 404
assert response.json()["error"]["code"] == "not_found"
listed = await api_client.get(
"/v1/points", params={"file_id": str(file_id)}, headers=_auth(owner_token)
)
assert len(listed.json()["points"]) == 3
async def test_delete_file_requires_the_points_write_scope(
api_client: AsyncClient, db_session: AsyncSession
) -> None:
tenant = await create_tenant(db_session)
_, token = await create_api_key(db_session, tenant=tenant, scopes=["files:write"])
file_id = uuid.uuid4()
await create_source_file(db_session, tenant=tenant, source_file_id=file_id)
await db_session.commit()
response = await api_client.delete(f"/v1/files/{file_id}", headers=_auth(token))
assert response.status_code == 403

View File

@@ -0,0 +1,45 @@
"""`/readyz`'s Qdrant probe against a real Qdrant.
The case worth an integration test is the one a fake cannot produce
convincingly: Qdrant is up and answering, but the collection the deployment
step was supposed to create is not there.
"""
import uuid
import pytest
from qdrant_client import AsyncQdrantClient
from src.config import QdrantSettings
from src.infrastructure.qdrant.client import ping
from src.infrastructure.qdrant.collection import ensure_chunks_collection
pytestmark = [
pytest.mark.integration,
pytest.mark.qdrant,
pytest.mark.asyncio(loop_scope="session"),
]
async def test_ping_is_not_ready_when_the_collection_was_never_bootstrapped(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""A healthy Qdrant with no collection is *not* ready: uploads would 502."""
assert await ping(qdrant_client, 5.0, collection=qdrant_settings.collection) is False
async def test_ping_is_ready_after_bootstrap(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
assert await ping(qdrant_client, 5.0, collection=qdrant_settings.collection) is True
async def test_ping_is_not_ready_for_a_different_collection_name(
qdrant_client: AsyncQdrantClient, qdrant_settings: QdrantSettings
) -> None:
"""A misconfigured QDRANT_COLLECTION is as unready as a missing one."""
await ensure_chunks_collection(qdrant_client, collection=qdrant_settings.collection)
assert await ping(qdrant_client, 5.0, collection=f"absent_{uuid.uuid4().hex}") is False

205
tests/support/containers.py Normal file
View File

@@ -0,0 +1,205 @@
"""Disposable real infrastructure for integration and e2e tests (ADR-0016).
Registered as a plugin from the root `tests/conftest.py` rather than living in
a per-boundary conftest, because `tests/e2e/` needs the same Postgres, MinIO,
and Qdrant containers that `tests/integration/*/` does, and `pytest_plugins`
is only honoured in the root conftest.
Nothing here is autouse: the container fixtures are session-scoped but lazy,
so `pytest -m unit` still runs without Docker.
"""
import uuid
from collections.abc import AsyncIterator, Iterator
import pytest
import pytest_asyncio
from alembic.command import upgrade
from alembic.config import Config
from qdrant_client import AsyncQdrantClient
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
from testcontainers.community.minio import MinioContainer
from testcontainers.community.postgres import PostgresContainer
from testcontainers.community.qdrant import QdrantContainer
from src.config import MinioSettings, PostgresSettings, QdrantSettings
from src.infrastructure.minio.client import create_client as create_minio_client
from src.infrastructure.postgres.database import create_engine
from src.infrastructure.qdrant.client import create_client as create_qdrant_client
_MINIO_BUCKET = "test-source-files"
# Pinned to match the `qdrant-client` major/minor in pyproject.toml. The
# testcontainers default image trails it far enough that the client emits an
# incompatibility warning, and testing against a version we do not deploy is
# the wrong signal anyway.
_QDRANT_IMAGE = "qdrant/qdrant:v1.19.0"
# --- Postgres ---------------------------------------------------------------
def _alembic_config(database_url: str) -> Config:
config = Config("alembic.ini")
config.set_main_option("sqlalchemy.url", database_url)
return config
def _postgres_settings_from_url(url: str) -> PostgresSettings:
# testcontainers returns postgresql+asyncpg://user:pass@host:port/db ;
# PostgresSettings builds its own dsn from parts, so parse the parts back out.
without_scheme = url.split("://", 1)[1]
creds, hostpart = without_scheme.split("@", 1)
user, password = creds.split(":", 1)
hostport, db = hostpart.split("/", 1)
host, port = hostport.split(":", 1)
return PostgresSettings(host=host, port=int(port), user=user, password=password, db=db)
@pytest.fixture(scope="session")
def postgres_container() -> Iterator[PostgresContainer]:
with PostgresContainer("postgres:17", driver="asyncpg") as container:
yield container
@pytest.fixture(scope="session")
def postgres_url(postgres_container: PostgresContainer) -> str:
"""The container's URL, pinned to IPv4.
Testcontainers reports the host as `localhost`, which resolves to `::1`
before `127.0.0.1`. Docker publishes the mapped port on IPv4 only, and the
IPv6 SYN is dropped rather than refused, so asyncpg blocks on the first
address until its connect timeout instead of falling back to the second --
the connection does not fail, it hangs.
"""
return postgres_container.get_connection_url().replace("@localhost:", "@127.0.0.1:")
@pytest.fixture(scope="session")
def migrated_postgres_url(postgres_url: str) -> str:
"""The container's URL, after Alembic has created the schema on it once."""
upgrade(_alembic_config(postgres_url), "head")
return postgres_url
@pytest.fixture(scope="session")
def postgres_settings(migrated_postgres_url: str) -> PostgresSettings:
"""Settings an application component can be constructed from directly."""
return _postgres_settings_from_url(migrated_postgres_url)
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def postgres_engine(postgres_settings: PostgresSettings) -> AsyncIterator[AsyncEngine]:
engine = create_engine(postgres_settings)
try:
yield engine
finally:
await engine.dispose()
@pytest_asyncio.fixture(loop_scope="session")
async def db_sessionmaker(
postgres_engine: AsyncEngine,
) -> AsyncIterator[async_sessionmaker[AsyncSession]]:
"""A session *factory* per test, bound to a rolled-back outer transaction.
Every session it produces shares one connection/outer transaction, so
writes `commit()`ed by one session are visible to the next -- needed for
code under test that opens more than one session per operation (auth
resolution, the ADR-0017 two-phase upload) -- while the whole test's
writes still roll back together at teardown (ADR-0016: isolate data per
test).
"""
async with postgres_engine.connect() as connection:
outer_transaction = await connection.begin()
sessionmaker = async_sessionmaker(
bind=connection, expire_on_commit=False, join_transaction_mode="create_savepoint"
)
yield sessionmaker
await outer_transaction.rollback()
@pytest_asyncio.fixture(loop_scope="session")
async def db_session(
db_sessionmaker: async_sessionmaker[AsyncSession],
) -> AsyncIterator[AsyncSession]:
"""One session per test, bound to a rolled-back outer transaction.
Isolates each test's writes (ADR-0016: isolate data per test) without
needing a fresh container or unique keys per test.
"""
async with db_sessionmaker() as session:
yield session
# --- MinIO ------------------------------------------------------------------
@pytest.fixture(scope="session")
def minio_container() -> Iterator[MinioContainer]:
with MinioContainer() as container:
yield container
@pytest.fixture(scope="session")
def minio_settings(minio_container: MinioContainer) -> MinioSettings:
"""Settings for the container, with the bucket already created.
Bucket creation belongs here rather than in a separate autouse fixture:
autouse in a *globally registered* plugin would pull a MinIO container
into every unit test run.
"""
config = minio_container.get_config()
# Pinned to IPv4 for the same reason as postgres_url above: `localhost`
# resolves to `::1` first, but Docker only publishes the mapped port on
# IPv4, so the connection hangs instead of failing.
endpoint = config["endpoint"].replace("localhost:", "127.0.0.1:")
settings = MinioSettings(
endpoint=endpoint,
access_key=config["access_key"],
secret_key=config["secret_key"],
secure=False,
bucket=_MINIO_BUCKET,
)
client = create_minio_client(settings)
if not client.bucket_exists(settings.bucket):
client.make_bucket(settings.bucket)
return settings
# --- Qdrant -----------------------------------------------------------------
@pytest.fixture(scope="session")
def qdrant_container() -> Iterator[QdrantContainer]:
with QdrantContainer(image=_QDRANT_IMAGE) as container:
yield container
@pytest.fixture(scope="session")
def qdrant_url(qdrant_container: QdrantContainer) -> str:
"""The container's REST URL, pinned to IPv4.
Same gotcha as postgres_url/minio_settings above: testcontainers reports
the host as `localhost`, which resolves to `::1` first, but Docker
publishes the mapped port on IPv4 only. The IPv6 SYN is dropped rather
than refused, so the client hangs until its timeout instead of falling
back to the second address -- the connection does not fail, it hangs.
"""
host = qdrant_container.get_container_host_ip().replace("localhost", "127.0.0.1")
return f"http://{host}:{qdrant_container.get_exposed_port(6333)}"
@pytest.fixture
def qdrant_settings(qdrant_url: str) -> QdrantSettings:
"""Settings naming a collection unique to this test (ADR-0016 isolation)."""
return QdrantSettings(url=qdrant_url, collection=f"chunks_{uuid.uuid4().hex}")
@pytest_asyncio.fixture(loop_scope="session")
async def qdrant_client(qdrant_settings: QdrantSettings) -> AsyncIterator[AsyncQdrantClient]:
client = create_qdrant_client(qdrant_settings)
try:
yield client
finally:
await client.close()

View File

@@ -11,7 +11,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
from src.application.auth.keys import generate_api_key, hash_secret
from src.infrastructure.postgres.models.api_key import ApiKey
from src.infrastructure.postgres.models.source_file import SourceFile
from src.infrastructure.postgres.models.tenant import Tenant
from src.infrastructure.postgres.models.tenant_domain import TenantDomain
async def create_tenant(
@@ -47,3 +49,57 @@ async def create_api_key(
session.add(api_key)
await session.flush()
return api_key, full_key
async def create_source_file(
session: AsyncSession,
*,
tenant: Tenant,
source_file_id: uuid.UUID | None = None,
domain: str = "general",
status: str = "active",
) -> SourceFile:
"""A `source_files` row for tests that address a file without uploading one.
`DELETE /v1/files/{file_id}` authorizes against this row before touching a
single point, so a delete test needs it even though the interesting state
lives in Qdrant.
"""
source_file = SourceFile(
id=source_file_id or uuid.uuid4(),
tenant_id=tenant.id,
domain=domain,
source_filename="handbook.docx",
source_type="docx",
content_sha256="0" * 64,
byte_size=1024,
storage_uri="s3://bucket/key",
status=status,
)
session.add(source_file)
await session.flush()
return source_file
async def create_tenant_domain(
session: AsyncSession,
*,
tenant: Tenant,
domain: str = "general",
status: str = "active",
) -> TenantDomain:
"""Register a domain so an upload to it passes the allowlist check.
Uploads reject an unregistered domain (ADR-0009), so any test that uploads
needs one of these.
"""
tenant_domain = TenantDomain(
id=uuid.uuid4(),
tenant_id=tenant.id,
domain=domain,
display_name=domain,
status=status,
)
session.add(tenant_domain)
await session.flush()
return tenant_domain

View File

@@ -0,0 +1,399 @@
"""One behavioural contract, run against both `PointRepository` implementations.
Plan 002 Phase 1's exit criterion is that the Qdrant adapter and the in-memory
fake agree on filtering, ordering, and pagination. Asserting that by writing two
parallel test files invites them to drift — the usual outcome is a fake that
quietly grows more forgiving than the real store, so unit tests keep passing
while production does something else.
So the scenarios live here once. `tests/unit/application/points/` runs them
against `FakePointRepository`; `tests/integration/qdrant/` runs the identical
functions against real Qdrant. A divergence fails one of the two runs rather
than hiding.
Each scenario takes an already-seeded repository plus the ids it was seeded
with, so seeding (which genuinely differs — one writes a dict, the other upserts
vectors) stays outside the shared code.
"""
import uuid
from dataclasses import dataclass
from datetime import UTC, datetime
from src.application.ingestion.chunking import chunk_id_for
from src.application.ingestion.models import SparseVector
from src.application.points.models import ChunkPoint
from src.application.points.point import Point
from src.application.ports.point_repository import PayloadPatch, PointRepository
from src.infrastructure.qdrant.collection import (
DENSE_NOMIC_DIMENSIONS,
DENSE_OPENAI_DIMENSIONS,
)
# Two tenants and two files, so every scenario can assert both that the right
# rows come back and that the wrong ones do not.
SEEDED_AT = datetime(2026, 1, 1, 12, 0, tzinfo=UTC)
@dataclass(frozen=True)
class SeedSpec:
"""One point to seed. Deliberately flat — the fixture builds the model."""
tenant_id: uuid.UUID
file_id: uuid.UUID
chunk_index: int
content: str
order_id: float
is_active: bool = True
version: int = 1
domain: str = "support"
def point_id_for(file_id: uuid.UUID, chunk_index: int) -> uuid.UUID:
"""The deterministic point id (ADR-0001), re-exported for readability."""
return chunk_id_for(file_id, chunk_index)
def build_point(spec: SeedSpec) -> Point:
point_id = point_id_for(spec.file_id, spec.chunk_index)
return Point(
point_id=point_id,
tenant_id=spec.tenant_id,
domain=spec.domain,
file_id=spec.file_id,
chunk_id=point_id,
content=spec.content,
content_type="paragraph",
source_filename="handbook.docx",
source_type="docx",
order_id=spec.order_id,
chunk_index=spec.chunk_index,
previous_chunk_id=None,
next_chunk_id=None,
is_active=spec.is_active,
deleted_at=None if spec.is_active else SEEDED_AT,
created_at=SEEDED_AT,
updated_at=SEEDED_AT,
created_by="ingestion",
updated_by="ingestion",
version=spec.version,
content_hash="0" * 64,
embedding_model_version="test-model",
)
def chunk_point_for(spec: SeedSpec) -> ChunkPoint:
"""A `SeedSpec` as something upsertable into real Qdrant.
The payload is taken straight off `build_point`'s read model rather than
hand-written, so the write shape and the read shape cannot drift apart. The
vectors are constant filler: nothing in plan 002's read paths scores by
similarity, so their values are irrelevant and their dimensions are not.
"""
point = build_point(spec)
return ChunkPoint(
point_id=point.point_id,
dense={
"dense_nomic": [0.1] * DENSE_NOMIC_DIMENSIONS,
"dense_openai": [0.2] * DENSE_OPENAI_DIMENSIONS,
},
sparse=SparseVector(indices=[1, 2], values=[0.5, 0.25]),
payload=point.model_dump(mode="json", exclude={"point_id", "vectors"}),
)
def seed_specs(
tenant_a: uuid.UUID,
tenant_b: uuid.UUID,
file_a: uuid.UUID,
file_b: uuid.UUID,
file_c: uuid.UUID,
) -> tuple[SeedSpec, ...]:
"""The fixed corpus every scenario below assumes.
Ordering is deliberately not insertion order: `order_id` 3.0 is seeded
before 2.0 so a repository that returns rows in write order fails
`scenario_list_orders_by_order_id` instead of passing by accident.
Tenant B owns its own file. Sharing a `file_id` across tenants would be a
fixture that cannot occur: point ids are derived from `file_id` plus
`chunk_index` alone, so two tenants in one file would collide on a single
id and the corpus would be testing an impossible state.
"""
return (
SeedSpec(tenant_a, file_a, 0, "onboarding checklist for new staff", 1.0),
SeedSpec(tenant_a, file_a, 2, "payroll schedule and bank details", 3.0),
SeedSpec(tenant_a, file_a, 1, "expense policy and receipts", 2.0),
SeedSpec(tenant_a, file_a, 3, "retired parking policy", 4.0, is_active=False),
SeedSpec(tenant_a, file_b, 0, "supplier onboarding contacts", 1.0, domain="ops"),
SeedSpec(tenant_b, file_c, 0, "other tenant onboarding secrets", 1.0),
)
async def scenario_get_returns_point_within_tenant(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
point = await repository.get(tenant_id=tenant_a, point_id=point_id_for(file_a, 0))
assert point is not None
assert point.content == "onboarding checklist for new staff"
assert point.chunk_index == 0
assert point.order_id == 1.0
async def scenario_get_hides_other_tenants_point(
repository: PointRepository, *, tenant_b: uuid.UUID, file_a: uuid.UUID
) -> None:
"""Tenant B probing a real point id that belongs to tenant A gets nothing.
The point exists, so this distinguishes a tenant filter from a plain
existence check — the difference between `404` and a leak.
"""
assert await repository.get(tenant_id=tenant_b, point_id=point_id_for(file_a, 0)) is None
async def scenario_get_reaches_inactive_points(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
"""A direct `get` is not subject to the implied `is_active` read filter.
Soft delete and relinking both address points by id and must be able to see
a deactivated one; only listings hide them.
"""
point = await repository.get(tenant_id=tenant_a, point_id=point_id_for(file_a, 3))
assert point is not None
assert point.is_active is False
async def scenario_get_many_returns_only_this_tenants_points(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
found = await repository.get_many(
tenant_id=tenant_a,
point_ids=[point_id_for(file_a, 0), point_id_for(file_a, 1), uuid.uuid4()],
)
assert {point.chunk_index for point in found} == {0, 1}
async def scenario_list_orders_by_order_id(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
page = await repository.list_by_file(tenant_id=tenant_a, file_id=file_a, limit=10)
assert [point.order_id for point in page.points] == [1.0, 2.0, 3.0]
assert page.next_cursor is None
async def scenario_list_excludes_inactive_by_default(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
default = await repository.list_by_file(tenant_id=tenant_a, file_id=file_a, limit=10)
assert 4.0 not in [point.order_id for point in default.points]
opted_in = await repository.list_by_file(
tenant_id=tenant_a, file_id=file_a, limit=10, include_inactive=True
)
assert [point.order_id for point in opted_in.points] == [1.0, 2.0, 3.0, 4.0]
async def scenario_list_paginates_without_skipping_or_repeating(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
first = await repository.list_by_file(tenant_id=tenant_a, file_id=file_a, limit=2)
assert [point.order_id for point in first.points] == [1.0, 2.0]
assert first.next_cursor is not None
second = await repository.list_by_file(
tenant_id=tenant_a, file_id=file_a, limit=2, cursor=first.next_cursor
)
assert [point.order_id for point in second.points] == [3.0]
assert second.next_cursor is None
async def scenario_list_is_scoped_to_one_file(
repository: PointRepository, *, tenant_a: uuid.UUID, file_b: uuid.UUID
) -> None:
page = await repository.list_by_file(tenant_id=tenant_a, file_id=file_b, limit=10)
assert [point.content for point in page.points] == ["supplier onboarding contacts"]
async def scenario_count_respects_tenant_and_filters(
repository: PointRepository,
*,
tenant_a: uuid.UUID,
tenant_b: uuid.UUID,
file_a: uuid.UUID,
) -> None:
assert await repository.count(tenant_id=tenant_a) == 4
assert await repository.count(tenant_id=tenant_a, include_inactive=True) == 5
assert await repository.count(tenant_id=tenant_a, file_id=file_a) == 3
assert await repository.count(tenant_id=tenant_a, domain="ops") == 1
assert await repository.count(tenant_id=tenant_b) == 1
async def scenario_keyword_search_matches_content_within_tenant(
repository: PointRepository, *, tenant_a: uuid.UUID, tenant_b: uuid.UUID
) -> None:
found = await repository.keyword_search(tenant_id=tenant_a, query="onboarding", limit=10)
assert {point.content for point in found.points} == {
"onboarding checklist for new staff",
"supplier onboarding contacts",
}
# The same term matches a different tenant's point, which must not leak.
other = await repository.keyword_search(tenant_id=tenant_b, query="onboarding", limit=10)
assert {point.content for point in other.points} == {"other tenant onboarding secrets"}
async def scenario_keyword_search_excludes_inactive_by_default(
repository: PointRepository, *, tenant_a: uuid.UUID
) -> None:
""" "retired parking policy" is seeded inactive, so only the opt-in sees it."""
default = await repository.keyword_search(tenant_id=tenant_a, query="parking", limit=10)
assert default.points == ()
opted_in = await repository.keyword_search(
tenant_id=tenant_a, query="parking", limit=10, include_inactive=True
)
assert {point.content for point in opted_in.points} == {"retired parking policy"}
async def scenario_keyword_search_filters_by_domain(
repository: PointRepository, *, tenant_a: uuid.UUID
) -> None:
found = await repository.keyword_search(
tenant_id=tenant_a, query="onboarding", limit=10, domain="ops"
)
assert {point.content for point in found.points} == {"supplier onboarding contacts"}
async def scenario_patch_applies_when_version_matches(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
target = point_id_for(file_a, 0)
await repository.apply_patches(
tenant_id=tenant_a,
patches=[
PayloadPatch(
point_id=target,
payload={"content": "amended onboarding checklist", "version": 2},
expected_version=1,
)
],
)
point = await repository.get(tenant_id=tenant_a, point_id=target)
assert point is not None
assert point.content == "amended onboarding checklist"
assert point.version == 2
async def scenario_patch_is_a_noop_when_version_is_stale(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
"""A stale guard must match nothing rather than raise or clobber.
This is the behaviour the service's post-apply check depends on: Qdrant
reports success for a filtered `set_payload` that matched zero points, so
"did my write land?" is only answerable by reading back.
"""
target = point_id_for(file_a, 1)
await repository.apply_patches(
tenant_id=tenant_a,
patches=[
PayloadPatch(
point_id=target,
payload={"content": "should not land"},
expected_version=99,
)
],
)
point = await repository.get(tenant_id=tenant_a, point_id=target)
assert point is not None
assert point.content == "expense policy and receipts"
async def scenario_patch_cannot_cross_tenants(
repository: PointRepository, *, tenant_a: uuid.UUID, tenant_b: uuid.UUID, file_a: uuid.UUID
) -> None:
"""Tenant B patching tenant A's point id must change nothing.
An unguarded patch (no `expected_version`) addressed by a known-good id is
the sharpest form of the probe: only the tenant condition can stop it, so a
repository that filtered on id alone would rewrite another tenant's row.
"""
target = point_id_for(file_a, 0)
await repository.apply_patches(
tenant_id=tenant_b,
patches=[PayloadPatch(point_id=target, payload={"content": "hijacked"})],
)
victim = await repository.get(tenant_id=tenant_a, point_id=target)
assert victim is not None
assert victim.content == "onboarding checklist for new staff"
async def scenario_patch_batch_applies_every_patch_in_one_call(
repository: PointRepository, *, tenant_a: uuid.UUID, file_a: uuid.UUID
) -> None:
"""One `apply_patches` call carrying several points applies all of them.
This is the primitive soft delete is built on: a deactivation plus its two
neighbour relinks go out together, and a batch that silently applied only
its first operation would leave the pointer chain broken — the exact defect
ADR-0002 rules out. Nulling a pointer is included because `None` and
"absent" are different payload values, and only one of them clears a link.
"""
deleted, previous, following = (point_id_for(file_a, index) for index in (1, 0, 2))
await repository.apply_patches(
tenant_id=tenant_a,
patches=[
PayloadPatch(
point_id=deleted,
payload={"is_active": False, "version": 2},
expected_version=1,
),
PayloadPatch(
point_id=previous,
payload={"next_chunk_id": str(following), "version": 2},
expected_version=1,
),
PayloadPatch(
point_id=following,
payload={"previous_chunk_id": None, "version": 2},
expected_version=1,
),
],
)
applied = {
point.point_id: point
for point in await repository.get_many(
tenant_id=tenant_a, point_ids=[deleted, previous, following]
)
}
assert applied[deleted].is_active is False
assert applied[previous].next_chunk_id == following
assert applied[following].previous_chunk_id is None
assert [applied[point_id].version for point_id in (deleted, previous, following)] == [2, 2, 2]
# Scenarios that only read, safe to run in any order against one seeded corpus.
READ_SCENARIOS = (
scenario_get_returns_point_within_tenant,
scenario_get_hides_other_tenants_point,
scenario_get_reaches_inactive_points,
scenario_get_many_returns_only_this_tenants_points,
scenario_list_orders_by_order_id,
scenario_list_excludes_inactive_by_default,
scenario_list_paginates_without_skipping_or_repeating,
scenario_list_is_scoped_to_one_file,
scenario_count_respects_tenant_and_filters,
scenario_keyword_search_matches_content_within_tenant,
scenario_keyword_search_excludes_inactive_by_default,
scenario_keyword_search_filters_by_domain,
)
# Scenarios that mutate, and therefore need a freshly seeded corpus each.
WRITE_SCENARIOS = (
scenario_patch_applies_when_version_matches,
scenario_patch_is_a_noop_when_version_is_stale,
scenario_patch_cannot_cross_tenants,
scenario_patch_batch_applies_every_patch_in_one_call,
)

View File

@@ -0,0 +1,379 @@
"""Soft delete and neighbour relinking against the fake repository (ADR-0016).
The point of these tests is the pointer chain, not the HTTP surface. ADR-0002
treats a partial relink as a defect, and the ways to produce one are all here:
deleting at either boundary, deleting the same point twice, losing a version
race half way through a batch, and pointing at a neighbour that is gone.
`tests/integration/qdrant/test_points_deletion.py` runs the two central cases
against real Qdrant. What only this file can do cheaply is force the races.
"""
import uuid
from collections.abc import Sequence
from dataclasses import dataclass
from typing import override
import pytest
import structlog.testing
from src.application.points.deletion import soft_delete_file_points, soft_delete_point
from src.application.points.errors import PointVersionConflictError
from src.application.points.point import Point, PointNotFoundError
from src.application.ports.point_repository import PayloadPatch
from tests.fakes import FakePointRepository
from tests.support.point_contract import SeedSpec, build_point
pytestmark = [pytest.mark.unit, pytest.mark.asyncio]
TENANT = uuid.UUID("11111111-1111-4111-8111-111111111111")
OTHER_TENANT = uuid.UUID("22222222-2222-4222-8222-222222222222")
FILE = uuid.UUID("33333333-3333-4333-8333-333333333333")
OTHER_FILE = uuid.UUID("44444444-4444-4444-8444-444444444444")
ACTOR = "api_key:test"
def _chain(count: int, *, tenant: uuid.UUID = TENANT, file_id: uuid.UUID = FILE) -> list[Point]:
"""`count` points of one file, linked head to tail in `order_id` order."""
points = [
build_point(SeedSpec(tenant, file_id, index, f"chunk {index}", float(index + 1)))
for index in range(count)
]
return [
point.model_copy(
update={
"previous_chunk_id": points[index - 1].point_id if index else None,
"next_chunk_id": (points[index + 1].point_id if index + 1 < len(points) else None),
}
)
for index, point in enumerate(points)
]
def _seeded(*points: Point) -> FakePointRepository:
repository = FakePointRepository()
for point in points:
repository.add(point)
return repository
def _stored(repository: FakePointRepository, point_id: uuid.UUID) -> Point:
return repository.points[str(point_id)]
def _walk(repository: FakePointRepository, head: uuid.UUID) -> list[uuid.UUID]:
"""Follow `next_chunk_id` from `head`, guarding against a cycle."""
visited: list[uuid.UUID] = []
current: uuid.UUID | None = head
while current is not None and len(visited) <= len(repository.points):
visited.append(current)
current = _stored(repository, current).next_chunk_id
return visited
async def test_soft_delete_point_relinks_the_neighbours_of_a_middle_point() -> None:
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
deleted = await soft_delete_point(
repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR
)
assert deleted.is_active is False
assert deleted.deleted_at is not None
assert _stored(repository, first.point_id).next_chunk_id == last.point_id
assert _stored(repository, last.point_id).previous_chunk_id == first.point_id
async def test_soft_delete_point_sends_the_deactivation_and_both_relinks_in_one_batch() -> None:
"""A partial relink is a defect, so the three patches must not be split up."""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
assert len(repository.patch_batches) == 1
assert {patch.point_id for patch in repository.patch_batches[0]} == {
middle.point_id,
first.point_id,
last.point_id,
}
async def test_soft_delete_point_bumps_the_version_of_every_point_it_touches() -> None:
"""A relinked neighbour really changed, so a stale editor of it must `409`."""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
assert [_stored(repository, point.point_id).version for point in (first, middle, last)] == [
2,
2,
2,
]
assert _stored(repository, first.point_id).updated_by == ACTOR
async def test_soft_delete_point_leaves_the_new_head_without_a_previous_pointer() -> None:
first, second, third = _chain(3)
repository = _seeded(first, second, third)
await soft_delete_point(repository, tenant_id=TENANT, point_id=first.point_id, actor=ACTOR)
assert _stored(repository, second.point_id).previous_chunk_id is None
async def test_soft_delete_point_leaves_the_new_tail_without_a_next_pointer() -> None:
first, second, third = _chain(3)
repository = _seeded(first, second, third)
await soft_delete_point(repository, tenant_id=TENANT, point_id=third.point_id, actor=ACTOR)
assert _stored(repository, second.point_id).next_chunk_id is None
async def test_soft_delete_point_keeps_the_deleted_points_own_pointers() -> None:
"""Nothing active points at it any more, so its pointers record where it sat.
That record is what the retry re-plans from, and what a later restore or an
audit reader would need to place the point back in the sequence.
"""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
stored = _stored(repository, middle.point_id)
assert stored.previous_chunk_id == first.point_id
assert stored.next_chunk_id == last.point_id
async def test_soft_delete_point_is_a_noop_for_an_already_inactive_point() -> None:
"""Not a `404`, and not a second relink — no patch is issued at all."""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
versions_after_first = {
point.point_id: _stored(repository, point.point_id).version
for point in (first, middle, last)
}
repository.patch_batches.clear()
again = await soft_delete_point(
repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR
)
assert again.is_active is False
assert repository.patch_batches == []
assert {
point.point_id: _stored(repository, point.point_id).version
for point in (first, middle, last)
} == versions_after_first
async def test_soft_delete_point_raises_not_found_for_another_tenants_point() -> None:
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
with pytest.raises(PointNotFoundError):
await soft_delete_point(
repository, tenant_id=OTHER_TENANT, point_id=middle.point_id, actor=ACTOR
)
assert repository.patch_batches == []
assert _stored(repository, middle.point_id).is_active is True
async def test_soft_delete_point_raises_not_found_for_an_unknown_point() -> None:
repository = _seeded(*_chain(2))
with pytest.raises(PointNotFoundError):
await soft_delete_point(repository, tenant_id=TENANT, point_id=uuid.uuid4(), actor=ACTOR)
async def test_soft_delete_point_skips_a_neighbour_that_is_not_there() -> None:
"""A pointer naming an absent point means the chain was already broken.
The delete completes the half of the relink that exists rather than
refusing, which would leave the point unremovable through any endpoint.
"""
first, middle, last = _chain(3)
repository = _seeded(first, middle) # `last` is never stored
deleted = await soft_delete_point(
repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR
)
assert deleted.is_active is False
assert _stored(repository, first.point_id).next_chunk_id == last.point_id
async def test_soft_delete_point_visits_every_active_point_exactly_once_after_several_deletes() -> (
None
):
"""The plan's traversal property, over a sequence of deletes.
Walking `next_chunk_id` from the head must reach every surviving point once
and never step into a deactivated one.
"""
points = _chain(5)
repository = _seeded(*points)
for index in (1, 3):
await soft_delete_point(
repository, tenant_id=TENANT, point_id=points[index].point_id, actor=ACTOR
)
walked = _walk(repository, points[0].point_id)
assert walked == [points[0].point_id, points[2].point_id, points[4].point_id]
assert all(_stored(repository, point_id).is_active for point_id in walked)
@dataclass
class _ContendedRepository(FakePointRepository):
"""Bumps one point's version just before an apply, as a rival writer would.
That makes the patch guarding on the old version match nothing while the
rest of the batch lands — Qdrant's real behaviour, and the partial apply the
service's retry exists to repair. `rounds` bounds how long the rival keeps
interfering.
"""
rival: uuid.UUID | None = None
rounds: int = 0
@override
async def apply_patches(self, *, tenant_id: uuid.UUID, patches: Sequence[PayloadPatch]) -> None:
if self.rounds > 0 and self.rival is not None:
self.rounds -= 1
victim = self.points[str(self.rival)]
self.points[str(self.rival)] = victim.model_copy(update={"version": victim.version + 1})
await super().apply_patches(tenant_id=tenant_id, patches=patches)
async def test_soft_delete_point_repairs_a_partially_applied_batch_on_retry() -> None:
first, middle, last = _chain(3)
repository = _ContendedRepository(rival=first.point_id, rounds=1)
for point in (first, middle, last):
repository.add(point)
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
# The first batch left `first` stale; the second re-planned only what was
# still missing, rather than re-deactivating the already-inactive point.
assert len(repository.patch_batches) == 2
assert [patch.point_id for patch in repository.patch_batches[1]] == [first.point_id]
assert _stored(repository, first.point_id).next_chunk_id == last.point_id
assert _stored(repository, middle.point_id).is_active is False
async def test_soft_delete_point_conflicts_when_the_relink_never_settles() -> None:
first, middle, last = _chain(3)
repository = _ContendedRepository(rival=first.point_id, rounds=99)
for point in (first, middle, last):
repository.add(point)
with pytest.raises(PointVersionConflictError):
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
async def test_soft_delete_file_points_deactivates_every_active_point_of_the_file() -> None:
points = _chain(3)
other_file = _chain(2, file_id=OTHER_FILE)
other_tenant = _chain(1, tenant=OTHER_TENANT, file_id=uuid.uuid4())
repository = _seeded(*points, *other_file, *other_tenant)
swept = await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
assert swept == 3
assert all(not _stored(repository, point.point_id).is_active for point in points)
assert all(_stored(repository, point.point_id).is_active for point in other_file)
assert all(_stored(repository, point.point_id).is_active for point in other_tenant)
async def test_soft_delete_file_points_leaves_the_chain_intact() -> None:
"""No survivor can dangle, so the sweep rewrites no pointer at all."""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
assert _stored(repository, first.point_id).next_chunk_id == middle.point_id
assert _stored(repository, middle.point_id).previous_chunk_id == first.point_id
assert _stored(repository, last.point_id).previous_chunk_id == middle.point_id
async def test_soft_delete_file_points_pages_past_one_batch() -> None:
"""More points than one sweep page, so the re-listing loop has to run."""
points = _chain(230)
repository = _seeded(*points)
swept = await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
assert swept == 230
assert all(not _stored(repository, point.point_id).is_active for point in points)
async def test_soft_delete_file_points_is_a_noop_the_second_time() -> None:
repository = _seeded(*_chain(3))
await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
repository.patch_batches.clear()
swept = await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
assert swept == 0
assert repository.patch_batches == []
async def test_soft_delete_file_points_returns_zero_for_an_unknown_file() -> None:
repository = _seeded(*_chain(2))
assert (
await soft_delete_file_points(
repository, tenant_id=TENANT, file_id=uuid.uuid4(), actor=ACTOR
)
== 0
)
async def test_soft_delete_point_logs_its_duration_and_round_count() -> None:
"""ADR-0011's `duration_ms`, plus the field that explains a slow one.
Relinking is O(1), so a delete's cost is Qdrant round trips; `rounds` above
1 means a concurrent writer forced a re-plan rather than the store being
slow, and the two fields are only useful together.
"""
first, middle, last = _chain(3)
repository = _seeded(first, middle, last)
with structlog.testing.capture_logs() as logs:
await soft_delete_point(repository, tenant_id=TENANT, point_id=middle.point_id, actor=ACTOR)
event = next(entry for entry in logs if entry["event"] == "points.soft_deleted")
assert event["rounds"] == 1
assert isinstance(event["duration_ms"], float)
async def test_soft_delete_file_points_logs_a_round_per_sweep_page() -> None:
"""The sweep is the delete path whose cost tracks the size of the file."""
repository = _seeded(*_chain(230))
with structlog.testing.capture_logs() as logs:
await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)
event = next(entry for entry in logs if entry["event"] == "points.file_soft_deleted")
assert event["points_soft_deleted"] == 230
assert event["rounds"] == 3
assert isinstance(event["duration_ms"], float)
async def test_soft_delete_file_points_conflicts_when_a_round_makes_no_progress() -> None:
"""A sweep round that attempts the same ids as the one before is stuck."""
points = _chain(3)
repository = _ContendedRepository(rival=points[0].point_id, rounds=99)
for point in points:
repository.add(point)
with pytest.raises(PointVersionConflictError):
await soft_delete_file_points(repository, tenant_id=TENANT, file_id=FILE, actor=ACTOR)

View File

@@ -0,0 +1,227 @@
"""`index_chunks`: payload correctness, bounded batching, and the ordering
that keeps a failed attempt from damaging a working index (ADR-0001, ADR-0017).
"""
import uuid
import pytest
from anyio import CapacityLimiter
from src.application.ingestion.chunking import chunk_id_for
from src.application.ingestion.errors import PointIndexingError
from src.application.ingestion.models import Chunk, ContentType, EmbeddedChunk, SparseVector
from src.application.points import index_chunks
from src.config import QdrantSettings
from tests.fakes import FakeDenseEmbedder, FakePointStorage, FakeSparseEmbedder
pytestmark = [pytest.mark.unit, pytest.mark.asyncio]
_TENANT_ID = uuid.UUID("11111111-1111-1111-1111-111111111111")
_FILE_ID = uuid.UUID("22222222-2222-2222-2222-222222222222")
_API_KEY_ID = uuid.UUID("33333333-3333-3333-3333-333333333333")
_ACTOR = f"api_key:{_API_KEY_ID}"
def _embedded(count: int) -> list[EmbeddedChunk]:
chunks = [
Chunk(
chunk_id=chunk_id_for(_FILE_ID, index),
chunk_index=index,
order_id=float(index + 1),
content=f"chunk {index}",
content_type=ContentType.PARAGRAPH,
token_count=2,
character_count=7,
)
for index in range(count)
]
for position, chunk in enumerate(chunks):
if position > 0:
chunk.previous_chunk_id = chunks[position - 1].chunk_id
if position < len(chunks) - 1:
chunk.next_chunk_id = chunks[position + 1].chunk_id
return [
EmbeddedChunk(
chunk=chunk,
dense={"dense_nomic": [0.0] * 4, "dense_openai": [1.0] * 4},
sparse=SparseVector(indices=[7], values=[0.5]),
)
for chunk in chunks
]
async def _index(
storage: FakePointStorage,
*,
count: int = 3,
settings: QdrantSettings | None = None,
domain: str = "fire",
):
return await index_chunks(
_embedded(count),
storage=storage,
tenant_id=_TENANT_ID,
domain=domain,
file_id=_FILE_ID,
source_filename="policy.docx",
source_type="docx",
actor=_ACTOR,
dense_embedders=[
FakeDenseEmbedder(name="dense_nomic", model_version="nomic-embed-text-v2-moe"),
FakeDenseEmbedder(name="dense_openai", model_version="text-embedding-3-large"),
],
sparse_embedder=FakeSparseEmbedder(model_version="bm25-fa_norm_stop"),
settings=settings or QdrantSettings(),
thread_limiter=CapacityLimiter(2),
)
async def test_index_chunks_writes_every_adr_0001_payload_field() -> None:
storage = FakePointStorage()
await _index(storage, count=3)
payload = storage.points[str(chunk_id_for(_FILE_ID, 1))].payload
assert payload["tenant_id"] == str(_TENANT_ID)
assert payload["domain"] == "fire"
assert payload["file_id"] == str(_FILE_ID)
assert payload["chunk_id"] == str(chunk_id_for(_FILE_ID, 1))
assert payload["content"] == "chunk 1"
assert payload["content_type"] == "paragraph"
assert payload["source_filename"] == "policy.docx"
assert payload["source_type"] == "docx"
assert payload["order_id"] == 2.0
assert payload["chunk_index"] == 1
assert payload["previous_chunk_id"] == str(chunk_id_for(_FILE_ID, 0))
assert payload["next_chunk_id"] == str(chunk_id_for(_FILE_ID, 2))
assert payload["is_active"] is True
assert payload["deleted_at"] is None
assert payload["created_by"] == _ACTOR
assert payload["updated_by"] == _ACTOR
assert payload["version"] == 1
assert payload["created_at"] == payload["updated_at"]
assert isinstance(payload["content_hash"], str)
# Sorted, so wiring order cannot change the value (ADR-0001).
assert payload["embedding_model_version"] == (
"bm25-fa_norm_stop+nomic-embed-text-v2-moe+text-embedding-3-large"
)
async def test_index_chunks_uses_null_neighbours_at_the_file_ends() -> None:
storage = FakePointStorage()
await _index(storage, count=3)
assert storage.points[str(chunk_id_for(_FILE_ID, 0))].payload["previous_chunk_id"] is None
assert storage.points[str(chunk_id_for(_FILE_ID, 2))].payload["next_chunk_id"] is None
async def test_index_chunks_derives_tenant_and_domain_from_the_caller_not_the_chunk() -> None:
"""Tenant identity is server-derived; nothing in the chunk can assert it."""
storage = FakePointStorage()
await _index(storage, count=1, domain="car")
payload = storage.points[str(chunk_id_for(_FILE_ID, 0))].payload
assert payload["tenant_id"] == str(_TENANT_ID)
assert payload["domain"] == "car"
async def test_index_chunks_uses_deterministic_point_ids() -> None:
storage = FakePointStorage()
result = await _index(storage, count=4)
assert result.points_upserted == 4
assert set(storage.points) == {str(chunk_id_for(_FILE_ID, i)) for i in range(4)}
async def test_index_chunks_repeated_run_produces_no_duplicate_points() -> None:
storage = FakePointStorage()
await _index(storage, count=4)
await _index(storage, count=4)
assert len(storage.points) == 4
async def test_index_chunks_batches_at_the_configured_size() -> None:
storage = FakePointStorage()
settings = QdrantSettings(upsert_batch_size=2, upsert_concurrency=4)
await _index(storage, count=5, settings=settings)
assert storage.upsert_batches == [2, 2, 1]
async def test_index_chunks_bounds_in_flight_batches() -> None:
storage = FakePointStorage()
settings = QdrantSettings(upsert_batch_size=1, upsert_concurrency=2)
await _index(storage, count=8, settings=settings)
assert len(storage.upsert_batches) == 8
assert storage.max_in_flight <= 2
async def test_index_chunks_soft_deletes_only_points_past_the_new_chunk_count() -> None:
storage = FakePointStorage()
await _index(storage, count=5)
result = await _index(storage, count=2)
assert result.points_soft_deleted == 3
assert storage.points[str(chunk_id_for(_FILE_ID, 1))].payload["is_active"] is True
assert storage.points[str(chunk_id_for(_FILE_ID, 2))].payload["is_active"] is False
assert storage.points[str(chunk_id_for(_FILE_ID, 4))].payload["is_active"] is False
async def test_index_chunks_does_not_soft_delete_when_an_upsert_batch_fails() -> None:
"""A failed attempt must never remove content from a working index."""
storage = FakePointStorage()
await _index(storage, count=5)
storage.deactivate_calls.clear()
storage.fail_on_batch = 1
with pytest.raises(PointIndexingError):
await _index(storage, count=2, settings=QdrantSettings(upsert_batch_size=1))
assert storage.deactivate_calls == []
assert all(point.payload["is_active"] is True for point in storage.points.values())
async def test_index_chunks_raises_point_indexing_error_when_a_batch_fails() -> None:
storage = FakePointStorage(fail_on_batch=0)
with pytest.raises(PointIndexingError, match="upserting"):
await _index(storage, count=2)
async def test_index_chunks_raises_point_indexing_error_when_the_sweep_fails() -> None:
storage = FakePointStorage(fail_deactivate=True)
with pytest.raises(PointIndexingError, match="soft-deleting"):
await _index(storage, count=2)
async def test_index_chunks_on_empty_input_touches_no_storage() -> None:
storage = FakePointStorage()
result = await index_chunks(
[],
storage=storage,
tenant_id=_TENANT_ID,
domain="fire",
file_id=_FILE_ID,
source_filename="empty.csv",
source_type="csv",
actor=_ACTOR,
dense_embedders=[FakeDenseEmbedder(name="dense_nomic")],
sparse_embedder=FakeSparseEmbedder(),
settings=QdrantSettings(),
thread_limiter=CapacityLimiter(2),
)
assert result.points_upserted == 0
assert storage.upsert_batches == []
assert storage.deactivate_calls == []

View File

@@ -0,0 +1,57 @@
"""`FakePointRepository` against the shared `PointRepository` contract.
The same scenarios run against real Qdrant in
`tests/integration/qdrant/test_point_repository.py`. Keeping both runs green is
what stops the fake from drifting into something more permissive than the store
it stands in for — see `tests/support/point_contract.py`.
"""
import uuid
from collections.abc import Awaitable, Callable
import pytest
from src.application.ports.point_repository import PointRepository
from tests.fakes import FakePointRepository
from tests.support import point_contract
from tests.support.point_contract import build_point, seed_specs
pytestmark = [pytest.mark.unit, pytest.mark.asyncio]
type Scenario = Callable[..., Awaitable[None]]
TENANT_A = uuid.UUID("11111111-1111-4111-8111-111111111111")
TENANT_B = uuid.UUID("22222222-2222-4222-8222-222222222222")
FILE_A = uuid.UUID("33333333-3333-4333-8333-333333333333")
FILE_B = uuid.UUID("44444444-4444-4444-8444-444444444444")
FILE_C = uuid.UUID("55555555-5555-4555-8555-555555555555")
IDS = {
"tenant_a": TENANT_A,
"tenant_b": TENANT_B,
"file_a": FILE_A,
"file_b": FILE_B,
"file_c": FILE_C,
}
def _seeded() -> FakePointRepository:
repository = FakePointRepository()
for spec in seed_specs(TENANT_A, TENANT_B, FILE_A, FILE_B, FILE_C):
repository.add(build_point(spec))
return repository
def _arguments(scenario: Scenario) -> dict[str, uuid.UUID]:
"""Pass only the ids a scenario declares, so each one names its own inputs."""
return {name: value for name, value in IDS.items() if name in scenario.__annotations__}
@pytest.mark.parametrize(
"scenario",
[*point_contract.READ_SCENARIOS, *point_contract.WRITE_SCENARIOS],
ids=lambda scenario: scenario.__name__.removeprefix("scenario_"),
)
async def test_fake_point_repository_satisfies_the_contract(scenario: Scenario) -> None:
repository: PointRepository = _seeded()
await scenario(repository, **_arguments(scenario))

View File

@@ -0,0 +1,103 @@
"""`application/points/queries.py` against the fake repository (ADR-0016).
The HTTP-level behaviour is covered in `tests/integration/qdrant/`. What is
worth isolating here is the small amount of logic the service actually owns —
raising instead of returning `None`, and folding a query's Arabic letterforms —
because both are cheap to test directly and expensive to notice when broken.
"""
import uuid
import pytest
from src.application.points.point import PointNotFoundError
from src.application.points.queries import (
count_points,
get_point,
list_file_points,
search_points,
)
from tests.fakes import FakePointRepository
from tests.support.point_contract import SeedSpec, build_point, point_id_for
pytestmark = [pytest.mark.unit, pytest.mark.asyncio]
TENANT = uuid.UUID("11111111-1111-4111-8111-111111111111")
OTHER_TENANT = uuid.UUID("22222222-2222-4222-8222-222222222222")
FILE = uuid.UUID("33333333-3333-4333-8333-333333333333")
def _repository(*specs: SeedSpec) -> FakePointRepository:
repository = FakePointRepository()
for spec in specs:
repository.add(build_point(spec))
return repository
async def test_get_point_returns_the_point_for_its_owner() -> None:
repository = _repository(SeedSpec(TENANT, FILE, 0, "hello", 1.0))
point = await get_point(repository, tenant_id=TENANT, point_id=point_id_for(FILE, 0))
assert point.content == "hello"
async def test_get_point_raises_for_another_tenants_point() -> None:
"""Raising, not returning `None`, so a route cannot serve `200 null`."""
repository = _repository(SeedSpec(TENANT, FILE, 0, "hello", 1.0))
with pytest.raises(PointNotFoundError):
await get_point(repository, tenant_id=OTHER_TENANT, point_id=point_id_for(FILE, 0))
async def test_get_point_raises_for_an_unknown_point() -> None:
repository = _repository()
with pytest.raises(PointNotFoundError):
await get_point(repository, tenant_id=TENANT, point_id=uuid.uuid4())
async def test_search_points_folds_arabic_letterforms_before_matching() -> None:
"""The query is normalized the same way ingestion normalized the content.
Stored content carries Persian yeh (U+06CC) because `normalize_persian_text`
folded it at ingest. A query typed with Arabic yeh (U+064A) is a different
codepoint and would match nothing — silently, with no error to notice.
"""
persian = "مدیریت"
arabic = "مديريت"
assert persian != arabic
repository = _repository(SeedSpec(TENANT, FILE, 0, persian, 1.0))
found = await search_points(repository, tenant_id=TENANT, query=arabic, limit=10)
assert [point.content for point in found.points] == [persian]
async def test_search_points_does_not_reach_another_tenant() -> None:
repository = _repository(SeedSpec(OTHER_TENANT, FILE, 0, "secret", 1.0))
found = await search_points(repository, tenant_id=TENANT, query="secret", limit=10)
assert found.points == ()
async def test_list_file_points_returns_an_empty_page_for_a_foreign_file() -> None:
"""Indistinguishable from an unknown file — the same non-disclosure rule."""
repository = _repository(SeedSpec(OTHER_TENANT, FILE, 0, "secret", 1.0))
page = await list_file_points(repository, tenant_id=TENANT, file_id=FILE, limit=10)
assert page.points == ()
assert page.next_cursor is None
async def test_count_points_excludes_inactive_unless_asked() -> None:
repository = _repository(
SeedSpec(TENANT, FILE, 0, "live", 1.0),
SeedSpec(TENANT, FILE, 1, "gone", 2.0, is_active=False),
)
assert await count_points(repository, tenant_id=TENANT) == 1
assert await count_points(repository, tenant_id=TENANT, include_inactive=True) == 2

View File

@@ -37,6 +37,7 @@ class _TrackingDenseEmbedder:
"""
name: str
model_version: str = "stub-v1"
dimensions: int = 3
batches: list[list[str]] = field(default_factory=list)
in_flight: int = 0
@@ -54,6 +55,7 @@ class _TrackingDenseEmbedder:
@dataclass
class _FailingDenseEmbedder:
name: str
model_version: str = "stub-v1"
async def embed_batch(self, texts: Sequence[str]) -> list[list[float]]:
raise RuntimeError("boom")
@@ -62,6 +64,7 @@ class _FailingDenseEmbedder:
@dataclass
class _StubSparseEmbedder:
name: str = "sparse"
model_version: str = "stub-sparse-v1"
calls: list[list[str]] = field(default_factory=list)
def embed_batch(self, texts: Sequence[str], *, query: bool = False) -> list[SparseVector]:
@@ -72,6 +75,7 @@ class _StubSparseEmbedder:
@dataclass
class _FailingSparseEmbedder:
name: str = "sparse"
model_version: str = "stub-sparse-v1"
def embed_batch(self, texts: Sequence[str], *, query: bool = False) -> list[SparseVector]:
raise RuntimeError("boom")

View File

View File

@@ -0,0 +1,34 @@
"""Argument parsing for `python -m src.cli.provision_tenant`.
Only the argv -> arguments mapping is covered here; the provisioning behaviour
itself needs a real database and lives in
`tests/integration/postgres/test_provisioning.py`.
"""
import pytest
from src.application.tenants import DEFAULT_SCOPES
from src.cli.provision_tenant import _parse_args
pytestmark = pytest.mark.unit
def test_parse_args_defaults_scopes_and_domains() -> None:
args = _parse_args(["--slug", "acme"])
assert args.slug == "acme"
assert args.name is None
assert args.key_name == "bootstrap"
assert tuple(args.scopes.split(",")) == DEFAULT_SCOPES
assert args.domains == []
def test_parse_args_collects_repeated_domain_flags() -> None:
args = _parse_args(["--slug", "acme", "--domain", "fire", "--domain", "life"])
assert args.domains == ["fire", "life"]
def test_parse_args_requires_a_slug() -> None:
with pytest.raises(SystemExit):
_parse_args(["--domain", "fire"])

View File

@@ -0,0 +1,106 @@
"""Dual-sink logging config and the static environment processor (ADR-0011).
`configure_logging` mutates global logging state (`logging.config.dictConfig`,
`structlog.configure`), so these tests assert on the *handler configuration it
builds*, plus one end-to-end capture per sink, rather than trying to isolate
process-global state across tests.
"""
import json
import logging
import pytest
import structlog
from src.config import AppLimitSettings, LoggingSettings
from src.infrastructure.observability.logging import configure_logging
pytestmark = pytest.mark.unit
# Cross-test pollution from configure_logging() (cache_logger_on_first_use=True
# etc.) is reset by the autouse fixture in tests/conftest.py after every test,
# not just this module's -- these tests call the real configure_logging()
# directly and need the same cleanup any other test does.
def test_configure_logging_without_file_path_registers_only_console(tmp_path) -> None:
configure_logging(LoggingSettings(file_path=None), AppLimitSettings())
root = logging.getLogger()
handler_names = {type(h).__name__ for h in root.handlers}
assert handler_names == {"StreamHandler"}
def test_configure_logging_with_file_path_adds_a_rotating_file_handler(tmp_path) -> None:
log_file = tmp_path / "app.log"
configure_logging(
LoggingSettings(file_path=str(log_file), file_max_bytes=1024, file_backup_count=2),
AppLimitSettings(),
)
root = logging.getLogger()
handler_names = {type(h).__name__ for h in root.handlers}
assert handler_names == {"StreamHandler", "RotatingFileHandler"}
def test_file_sink_is_json_even_when_console_is_not(tmp_path, capsys) -> None:
"""The two sinks render independently: console stays human-readable while
the file stays JSON, regardless of LOG_JSON_FORMAT.
"""
log_file = tmp_path / "app.log"
configure_logging(
LoggingSettings(json_format=False, file_path=str(log_file)),
AppLimitSettings(),
)
structlog.get_logger("test").info("logging.dual_sink.test", widget_id="abc123")
console_output = capsys.readouterr().out
file_output = log_file.read_text().strip()
# Console: human-readable, not parseable JSON.
with pytest.raises(json.JSONDecodeError):
json.loads(console_output)
assert "logging.dual_sink.test" in console_output
# File: valid JSON with the same event.
file_event = json.loads(file_output)
assert file_event["event"] == "logging.dual_sink.test"
assert file_event["widget_id"] == "abc123"
def test_every_event_carries_env_and_service_version(tmp_path, capsys) -> None:
"""Static environment context, not a per-request contextvar -- it must
show up on an event with no request in flight.
"""
configure_logging(
LoggingSettings(json_format=True, file_path=None),
AppLimitSettings(env="staging", service_version="abc1234"),
)
structlog.get_logger("test").info("logging.env_context.test")
event = json.loads(capsys.readouterr().out.strip())
assert event["env"] == "staging"
assert event["service_version"] == "abc1234"
def test_env_context_survives_request_contextvar_clearing(tmp_path, capsys) -> None:
"""The bug this design avoids: if env/service_version were bound via
contextvars before a request, RequestIdMiddleware's clear_contextvars()
would wipe them. They must still appear after a clear.
"""
configure_logging(
LoggingSettings(json_format=True, file_path=None),
AppLimitSettings(env="prod", service_version="v42"),
)
structlog.contextvars.clear_contextvars()
structlog.contextvars.bind_contextvars(request_id="req-1")
structlog.get_logger("test").info("logging.post_clear.test")
event = json.loads(capsys.readouterr().out.strip())
assert event["env"] == "prod"
assert event["service_version"] == "v42"
assert event["request_id"] == "req-1"