Why:
- The failure modes worth testing are races, and they are cheap to force
against the fake and expensive to observe anywhere else.
Changes:
- Unit tests for both boundaries, the repeat delete, a missing neighbour, the
traversal property after several deletes, and — via a repository that bumps a
rival's version before each apply — both the partial-apply repair and the
unconvergent 409.
- One new shared contract scenario (a multi-point batch applies every patch,
including nulling a pointer) so it runs against the fake and real Qdrant.
- HTTP tests against real Postgres and Qdrant for relinking, soft-not-hard
delete, cross-tenant 404, and scope enforcement on both routes.
- create_source_file factory for tests addressing a file without uploading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- The isolation and pagination guarantees are the ones that fail silently, so
they need tests that would actually notice.
Changes:
- API tests over real Postgres and Qdrant together, because the invariant worth
testing spans both: the tenant Postgres derived is the only one Qdrant is ever
queried with.
- Pagination holds when a point is inserted behind the cursor mid-listing --
the defect an offset cursor would have.
- A route-order guard, since /{point_id} declared first turns count into a 422
and nothing else in the suite would catch it.
- Unit tests for the query service against the fake, including the Arabic to
Persian letterform fold and raising rather than returning None.
Impact:
- Suite goes to 325 passed, 3 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- Two parallel test files let a fake drift more permissive than the store it
stands in for, so unit tests stay green while production diverges. Plan 002
Phase 1's exit criterion is precisely that the two agree.
Changes:
- One scenario suite in tests/support/point_contract.py, run against
FakePointRepository (unit) and QdrantPointRepository (integration). A
divergence fails one of the two runs rather than hiding.
- The fake models the behaviours services branch on: the implied is_active read
filter, value-based cursor pagination, and a stale version guard that matches
nothing rather than raising -- the no-op Qdrant's filtered set_payload actually
has, and the reason a service must read back to know its write landed.
- Patched points are re-validated rather than model_copy'd, so the fake holds a
datetime where a read from real Qdrant returns one.
- The seeded corpus gives each tenant its own file: point IDs derive from
file_id plus chunk_index alone, so two tenants in one file would collide on a
single ID and the fixture would assert an impossible state.
Impact:
- 15 scenarios pass against both implementations.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- nothing over HTTP could create the first tenant: every /v1 route needs an API
key, and a key cannot exist before its tenant. The service was unusable by a
human without hand-written SQL.
Changes:
- add `provision_tenant`, owning tenant reuse-or-create, key generation and
hashing, and domain registration in one transaction
- expose it as `python -m src.cli.provision_tenant`, alongside
`alembic upgrade head` and `qdrant_bootstrap`
- add `tenants.get_by_slug`/`create` and `api_keys.create`
- log `tenant.provisioned` / `api_key.provisioned` with the key prefix only
Impact:
- a third deployment step; the plaintext key is printed once and never logged
or stored (ADR-0011, ADR-0009)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- Wanted human-readable console output while developing locally, without
losing a machine-parseable log for later grepping/parsing. A single
renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
(build/region/instance) independent of any one request.
Changes:
- configure_logging() now builds two independent handlers: console (always
on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
(LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
sinks locally"; "bind process-level environment context once at startup").
Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
both call sites (lifespan, qdrant_bootstrap CLI) updated.
Why:
- Ingested chunks need to become searchable Qdrant points before the upload
response returns, with tenant/domain isolation and a safe re-ingestion
story per ADR-0001/0017.
Changes:
- src/application/points/: index_chunks() is the sole entry point, owning
payload construction, batched/bounded-concurrency upserts
(upsert_concurrency semaphore), and a soft-delete sweep for points a
shorter re-ingestion leaves behind. The sweep runs only after every upsert
in the attempt succeeds, so a failed attempt can leave a stale prefix but
never removes content from a working index.
- PointStorage port (application/ports/) + QdrantPointStorage adapter
(infrastructure/qdrant/points.py), keeping the qdrant_client SDK out of
application code per ADR-0015.
- FakePointStorage test double for exercising the ordering/idempotency
guarantees without a real Qdrant.
Why:
- ADR-0001's Qdrant payload records embedding_model_version so a future model
swap can identify which chunks need re-embedding. The embedder is what
knows which model produced its vectors, so it reports this rather than the
call site reconstructing it from settings.
Changes:
- DenseEmbedder/SparseEmbedder protocols gain a model_version: str attribute.
- OpenAICompatibleEmbedder reports its configured model; Bm25SparseEmbedder
reports its analyzer (bm25-<analyzer>).
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
the inline upload path, with process-wide capacity/timeout/chunk-limit
guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
evaluation lab, which benchmarked them against the real Farsi corpus
(bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
at native 3072-dim), closing open items in ADR-0001/ADR-0005.
Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
helpers, and dense/sparse adapters (analyzers.py, bm25.py,
openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
(503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).
Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Implements plan 001 Phase 3's upload orchestration.
Changes:
- ObjectStorage port and MinIO adapter, thread-offloaded per ADR-0017.
- Tenant-scoped repositories for api_keys, source_files, ingestion_jobs.
- upload_source_file implementing the two-transaction shape with
(tenant_id, domain, content_sha256) idempotency.
Impact:
- This phase stores bytes only -- chunks_indexed is always 0 until
Phase 4/5 add parsing/embedding.
Why:
- The package exposed 8 modules directly, pushing source-type dispatch and
the ADR-0017 thread-offload obligation onto every caller.
Changes:
- Add parse_and_chunk_document as the sole public entry point.
- Demote the individual parsers to internal/test-only.
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
