Why:
- nothing over HTTP could create the first tenant: every /v1 route needs an API
key, and a key cannot exist before its tenant. The service was unusable by a
human without hand-written SQL.
Changes:
- add `provision_tenant`, owning tenant reuse-or-create, key generation and
hashing, and domain registration in one transaction
- expose it as `python -m src.cli.provision_tenant`, alongside
`alembic upgrade head` and `qdrant_bootstrap`
- add `tenants.get_by_slug`/`create` and `api_keys.create`
- log `tenant.provisioned` / `api_key.provisioned` with the key prefix only
Impact:
- a third deployment step; the plaintext key is printed once and never logged
or stored (ADR-0011, ADR-0009)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- resolve_auth_context() runs on every authenticated request and logged
nothing; four distinct rejection reasons (malformed/unknown/inactive/
expired key, inactive tenant) were all invisible.
- The domain-allowlist rejection in upload_source_file() happens before any
ingestion_jobs row exists, so it wasn't covered by the job-level
ingestion.job.failed event either -- a rejected upload left zero trace.
- Four of upload_source_file()'s five failure branches (parse_failed,
chunk_limit_exceeded, embedding_failed, index_failed) called
_mark_job_failed(), which wrote to Postgres but never logged; only
storage_failed and timeout had an ad-hoc logger.warning duplicated at their
own call sites.
Changes:
- auth/service.py: auth.succeeded / auth.failed (with a reason field per
rejection type), matching ADR-0011's own event catalog.
- domains/service.py: domain.rejected on the allowlist check;
domain.created / domain.updated / domain.status_changed on the three
mutations.
- files/upload.py: centralized failure logging inside _mark_job_failed
(every failure branch already calls it, so logging there once closes all
five branches instead of duplicating a log call at each site) as
ingestion.job.failed; added ingestion.job.started; renamed the ad-hoc
files.upload.succeeded to ingestion.job.completed for catalog consistency.
Impact:
- None to request/response behavior -- log events only.
Why:
- Wanted human-readable console output while developing locally, without
losing a machine-parseable log for later grepping/parsing. A single
renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
(build/region/instance) independent of any one request.
Changes:
- configure_logging() now builds two independent handlers: console (always
on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
(LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
sinks locally"; "bind process-level environment context once at startup").
Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
both call sites (lifespan, qdrant_bootstrap CLI) updated.
Why:
- Domain values are denormalized into every Qdrant point payload. Without
validation, an unregistered or typo'd domain (e.g. "fier" for "fire")
silently creates a new partition that retrieval never queries — the file
ends up invisible rather than rejected. Tenants also need independently
sized domain sets (one may run 14 insurance lines, another 6), which rules
out an enum.
Changes:
- tenant_domains table (migration 41335d162de8) + repository, unique on
(tenant_id, domain).
- src/application/domains/: ensure_domain_allowed() is the strict-allowlist
check now run inside upload_source_file()'s first transaction, before any
MinIO object, job row, or Qdrant point is written.
- /v1/domains (list/create/patch/disable/enable) gated on its own
domains:read/domains:write scopes, deliberately separate from files:write
so an upload key cannot create partitions. domain itself is immutable
(denormalized into every point payload); only display_name is editable.
Disable blocks new uploads without touching already-indexed points.
Impact:
- BREAKING: POST /v1/files now rejects any domain without an active
tenant_domains row (400, unknown_domain). A domain must be created via
POST /v1/domains before the first upload to it.
Why:
- The chunks collection is now created by an explicit deployment step
(qdrant_bootstrap), not at startup, which means a process can boot against
a healthy Qdrant that has no collection at all. /readyz's previous check
only called get_collections(), so it reported ready in that state — the
misconfiguration stayed invisible until the first upload failed with a 502
after already paying for the MinIO write and embedding round trips.
Changes:
- ping_qdrant() now checks collection_exists(collection) instead of just
reachability.
Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
— chunks were parsed and embedded but never written to Qdrant, so nothing
was actually searchable after upload.
Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
(error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
chunks_indexed now reflect the real indexing result instead of a hardcoded
zero.
- Wired PointStorage through AppResources/lifespan/the files router.
Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
Why:
- Ingested chunks need to become searchable Qdrant points before the upload
response returns, with tenant/domain isolation and a safe re-ingestion
story per ADR-0001/0017.
Changes:
- src/application/points/: index_chunks() is the sole entry point, owning
payload construction, batched/bounded-concurrency upserts
(upsert_concurrency semaphore), and a soft-delete sweep for points a
shorter re-ingestion leaves behind. The sweep runs only after every upsert
in the attempt succeeds, so a failed attempt can leave a stale prefix but
never removes content from a working index.
- PointStorage port (application/ports/) + QdrantPointStorage adapter
(infrastructure/qdrant/points.py), keeping the qdrant_client SDK out of
application code per ADR-0015.
- FakePointStorage test double for exercising the ordering/idempotency
guarantees without a real Qdrant.
Why:
- ADR-0001's Qdrant payload records embedding_model_version so a future model
swap can identify which chunks need re-embedding. The embedder is what
knows which model produced its vectors, so it reports this rather than the
call site reconstructing it from settings.
Changes:
- DenseEmbedder/SparseEmbedder protocols gain a model_version: str attribute.
- OpenAICompatibleEmbedder reports its configured model; Bm25SparseEmbedder
reports its analyzer (bm25-<analyzer>).
Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
sparse, late_interaction) and payload indexes defined at creation time per
ADR-0001; sparse/multivector fields cannot be added to an existing
collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
setup(), so it is a deployment step instead.
Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
idempotent and schema-verifying (raises on dimension/modifier mismatch
rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
(python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.
Impact:
- Deployments must run the new bootstrap command before the first upload;
see ADR-0001's new "Collection provisioning" section.
Why:
- No setting in this app ever actually read from .env: only the outer
Settings declared env_file=".env", and pydantic-settings does not cascade
that to nested BaseSettings classes. Every previously-correct local value
was coincidence (.env.example defaults matching class defaults). Found by
testing EMBEDDING_OPENAI_API_KEY against the live OpenAI API.
Changes:
- Every nested settings class now declares env_file=".env" itself.
- Settings.__init__/EmbeddingSettings.__init__ explicitly thread an
_env_file override to every nested constructor, so overriding it (as
tests do) reaches the whole tree, not just the outer class.
- env_ignore_empty=True everywhere, since the fix surfaced a second bug:
a blank env var (e.g. EMBEDDING_OPENAI_DIMENSIONS=) failed to parse as
int | None instead of falling back to the field default.
Impact:
- Real deployments setting env vars directly (Docker Compose) are
unaffected. Local .env-file development now actually works.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
the inline upload path, with process-wide capacity/timeout/chunk-limit
guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
evaluation lab, which benchmarked them against the real Farsi corpus
(bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
at native 3072-dim), closing open items in ADR-0001/ADR-0005.
Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
helpers, and dense/sparse adapters (analyzers.py, bm25.py,
openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
(503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).
Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Wires the ADR-0008 error envelope, per-request correlation id, and the
/v1/files routes into the app.
Changes:
- Extend AppResources/lifespan with the ingestion CapacityLimiter and
ObjectStorage adapter.
Impact:
- /v1 now exposes routes for the first time.
Why:
- Implements plan 001 Phase 3's upload orchestration.
Changes:
- ObjectStorage port and MinIO adapter, thread-offloaded per ADR-0017.
- Tenant-scoped repositories for api_keys, source_files, ingestion_jobs.
- upload_source_file implementing the two-transaction shape with
(tenant_id, domain, content_sha256) idempotency.
Impact:
- This phase stores bytes only -- chunks_indexed is always 0 until
Phase 4/5 add parsing/embedding.
Why:
- ADR-0017 names the bound INGESTION_MAX_UPLOAD_SIZE_MB; the code had it
as APP_MAX_UPLOAD_SIZE_MB. Reconciled to match the ADR, grouped with the
other three ADR-0017 bounds on IngestionSettings.
Why:
- The package exposed 8 modules directly, pushing source-type dispatch and
the ADR-0017 thread-offload obligation onto every caller.
Changes:
- Add parse_and_chunk_document as the sole public entry point.
- Demote the individual parsers to internal/test-only.
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
Why:
- first runnable slice of ADR-0012's resource-lifetime rules and ADR-0015's package layout: app-lifetime clients built once in the lifespan, released via explicit dependencies.
Changes:
- Settings split into per-domain nested settings (postgres/minio/ingestion/qdrant/app/logging); FastAPI app wired with /healthz, /readyz and a /v1 router; Postgres/MinIO/Qdrant adapters and SQLAlchemy models for tenants, API keys, source files, ingestion jobs/events.
Add the initial src/ package (empty FastAPI entrypoint, Settings stub),
an .env.example placeholder, and expand .gitignore for test/coverage
artifacts, environment files, and local Claude Code overrides. Also
documents the Alembic async bootstrap command in ADR-0009.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
