Why:
- ADR-0002's keyword search needs a full-text index on content, which
collection.py deliberately deferred to plan 002. is_active and chunk_index
were unindexed while ingestion was the only reader; every /v1/points read path
filters on them.
Changes:
- content gets a TEXT index with the multilingual tokenizer, which segments
Persian correctly where the word tokenizer mishandles ZWNJ-joined compounds.
No stemmer or stopword list: content is already letter-folded by
normalize_persian_text at ingest, and the ranked Farsi lexical path is the
benchmarked BM25 sparse vector, not this index.
- Tests assert content is TEXT rather than KEYWORD -- a keyword index would only
match an entire chunk verbatim, which never happens and fails silently.
- Adds a test that a missing index is added to an already-live collection.
Impact:
- Requires re-running `python -m src.cli.qdrant_bootstrap`. Payload indexes are
additive, so no collection rebuild and no re-embedding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- ADR-0002 already answered three of the four questions plan 002 listed as
"decisions needed"; the fourth -- what happens to a manually edited point when
its file is re-uploaded -- was left for a Phase 6 test to force. Deciding it in
code rather than in the ADR would invert this repo's rule.
Changes:
- ADR-0002 gains "Re-ingestion versus manual edits": the new file wins,
surviving points are overwritten with an incremented version, absent points
are flagged inactive rather than removed, and manually created points sit past
the ingested chunk_index range so the existing sweep covers them.
- Plan 002's stale decisions section becomes a pointer table; its audit scope is
pinned to both ADR-0009 tables.
Impact:
- Clobbered edits are recoverable from point_audit_events, not from Qdrant: the
deterministic point ID cannot hold both versions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- the slice had no operator documentation: no tuning guidance, no failure
procedure, no statement of the proxy timeout requirement.
Changes:
- add docs/runbook.md: startup, deployment steps, provisioning, the INGESTION_*
tuning table with each bound's status code, the proxy read-timeout rule,
/healthz vs /readyz, failure investigation by real event name plus job/event
SQL, retry semantics, and alert thresholds
- link it from the README and note provisioning there
- mark plan 001 Phase 6 done and refresh CLAUDE.md's status paragraph
Impact:
- documentation only
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- ADR-0016 reserves Compose for a serialized smoke test of the running web
process. Nothing else exercises the deployment steps, a real HTTP server, or
the real logging configuration, which every in-process test no-ops.
Changes:
- scripts/smoke.sh brings up Compose, runs both bootstrap steps, provisions a
throwaway tenant, starts uvicorn, and drives the test against it
- the test skips unless SMOKE_BASE_URL is set, so `uv run pytest` never invokes
Compose; it asserts the ADR-0011 JSON log sink
Impact:
- a pre-release gate, not a per-PR one
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- the slice's reliability invariants (ADR-0016) had no end-to-end coverage.
Changes:
- 13 Testcontainers-based tests: duplicate upload, retry after a failed job,
cross-tenant 404, unregistered domain, capacity 503, timeout 504, parse 400,
real Qdrant 502, missing scope 403, and both readiness states
- only the dense embedders are faked (ADR-0016 bars live providers); they
return the pinned 768/3072 dimensions
- the capacity test uses a committing sessionmaker, since the shared-connection
fixture cannot serve concurrent sessions
Impact:
- runs in the default `uv run pytest`; needs Docker, like every integration test
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- nothing over HTTP could create the first tenant: every /v1 route needs an API
key, and a key cannot exist before its tenant. The service was unusable by a
human without hand-written SQL.
Changes:
- add `provision_tenant`, owning tenant reuse-or-create, key generation and
hashing, and domain registration in one transaction
- expose it as `python -m src.cli.provision_tenant`, alongside
`alembic upgrade head` and `qdrant_bootstrap`
- add `tenants.get_by_slug`/`create` and `api_keys.create`
- log `tenant.provisioned` / `api_key.provisioned` with the key prefix only
Impact:
- a third deployment step; the plaintext key is printed once and never logged
or stored (ADR-0011, ADR-0009)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- tests/e2e/ cannot reach fixtures defined in a per-boundary conftest, and
`pytest_plugins` is only honoured in the root conftest.
Changes:
- move the Postgres/MinIO/Qdrant container fixtures into
tests/support/containers.py and register it as a root plugin
- fold MinIO bucket creation into `minio_settings`; an autouse fixture in a
globally registered plugin would pull a container into unit runs
- add a `postgres_settings` fixture so a component can be built from it directly
Impact:
- no behavior change; `pytest -m unit` still needs no Docker
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- resolve_auth_context() runs on every authenticated request and logged
nothing; four distinct rejection reasons (malformed/unknown/inactive/
expired key, inactive tenant) were all invisible.
- The domain-allowlist rejection in upload_source_file() happens before any
ingestion_jobs row exists, so it wasn't covered by the job-level
ingestion.job.failed event either -- a rejected upload left zero trace.
- Four of upload_source_file()'s five failure branches (parse_failed,
chunk_limit_exceeded, embedding_failed, index_failed) called
_mark_job_failed(), which wrote to Postgres but never logged; only
storage_failed and timeout had an ad-hoc logger.warning duplicated at their
own call sites.
Changes:
- auth/service.py: auth.succeeded / auth.failed (with a reason field per
rejection type), matching ADR-0011's own event catalog.
- domains/service.py: domain.rejected on the allowlist check;
domain.created / domain.updated / domain.status_changed on the three
mutations.
- files/upload.py: centralized failure logging inside _mark_job_failed
(every failure branch already calls it, so logging there once closes all
five branches instead of duplicating a log call at each site) as
ingestion.job.failed; added ingestion.job.started; renamed the ad-hoc
files.upload.succeeded to ingestion.job.completed for catalog consistency.
Impact:
- None to request/response behavior -- log events only.
Why:
- Any test using the client/api_client fixtures runs the app's real lifespan,
which calls the production configure_logging() -- setting
cache_logger_on_first_use=True (ADR-0011). That permanently monkeypatches
the .bind method on whichever module-level
logger = structlog.get_logger(__name__) instance is used first.
structlog.reset_defaults() only resets *global* config, not that
per-instance mutation, so once triggered, structlog.testing.capture_logs()
silently stops intercepting events in every test that runs afterward in the
same pytest process -- order-dependent flakiness with no useful failure
message (assertions just see an empty list).
Changes:
- Added two autouse fixtures: one no-ops configure_logging for tests that
spin up the app via LifespanManager (they test HTTP behavior, not logging
output, so they don't need the real thing), one resets structlog defaults
after every test as defense in depth.
Impact:
- Test-only; makes capture_logs()-based assertions reliable regardless of
test execution order.
Why:
- Wanted human-readable console output while developing locally, without
losing a machine-parseable log for later grepping/parsing. A single
renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
(build/region/instance) independent of any one request.
Changes:
- configure_logging() now builds two independent handlers: console (always
on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
(LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
sinks locally"; "bind process-level environment context once at startup").
Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
both call sites (lifespan, qdrant_bootstrap CLI) updated.
Why:
- Domain values are denormalized into every Qdrant point payload. Without
validation, an unregistered or typo'd domain (e.g. "fier" for "fire")
silently creates a new partition that retrieval never queries — the file
ends up invisible rather than rejected. Tenants also need independently
sized domain sets (one may run 14 insurance lines, another 6), which rules
out an enum.
Changes:
- tenant_domains table (migration 41335d162de8) + repository, unique on
(tenant_id, domain).
- src/application/domains/: ensure_domain_allowed() is the strict-allowlist
check now run inside upload_source_file()'s first transaction, before any
MinIO object, job row, or Qdrant point is written.
- /v1/domains (list/create/patch/disable/enable) gated on its own
domains:read/domains:write scopes, deliberately separate from files:write
so an upload key cannot create partitions. domain itself is immutable
(denormalized into every point payload); only display_name is editable.
Disable blocks new uploads without touching already-indexed points.
Impact:
- BREAKING: POST /v1/files now rejects any domain without an active
tenant_domains row (400, unknown_domain). A domain must be created via
POST /v1/domains before the first upload to it.
Why:
- The chunks collection is now created by an explicit deployment step
(qdrant_bootstrap), not at startup, which means a process can boot against
a healthy Qdrant that has no collection at all. /readyz's previous check
only called get_collections(), so it reported ready in that state — the
misconfiguration stayed invisible until the first upload failed with a 502
after already paying for the MinIO write and embedding round trips.
Changes:
- ping_qdrant() now checks collection_exists(collection) instead of just
reachability.
Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
— chunks were parsed and embedded but never written to Qdrant, so nothing
was actually searchable after upload.
Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
(error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
chunks_indexed now reflect the real indexing result instead of a hardcoded
zero.
- Wired PointStorage through AppResources/lifespan/the files router.
Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
Why:
- Ingested chunks need to become searchable Qdrant points before the upload
response returns, with tenant/domain isolation and a safe re-ingestion
story per ADR-0001/0017.
Changes:
- src/application/points/: index_chunks() is the sole entry point, owning
payload construction, batched/bounded-concurrency upserts
(upsert_concurrency semaphore), and a soft-delete sweep for points a
shorter re-ingestion leaves behind. The sweep runs only after every upsert
in the attempt succeeds, so a failed attempt can leave a stale prefix but
never removes content from a working index.
- PointStorage port (application/ports/) + QdrantPointStorage adapter
(infrastructure/qdrant/points.py), keeping the qdrant_client SDK out of
application code per ADR-0015.
- FakePointStorage test double for exercising the ordering/idempotency
guarantees without a real Qdrant.
Why:
- ADR-0001's Qdrant payload records embedding_model_version so a future model
swap can identify which chunks need re-embedding. The embedder is what
knows which model produced its vectors, so it reports this rather than the
call site reconstructing it from settings.
Changes:
- DenseEmbedder/SparseEmbedder protocols gain a model_version: str attribute.
- OpenAICompatibleEmbedder reports its configured model; Bm25SparseEmbedder
reports its analyzer (bm25-<analyzer>).
Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
sparse, late_interaction) and payload indexes defined at creation time per
ADR-0001; sparse/multivector fields cannot be added to an existing
collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
setup(), so it is a deployment step instead.
Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
idempotent and schema-verifying (raises on dimension/modifier mismatch
rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
(python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.
Impact:
- Deployments must run the new bootstrap command before the first upload;
see ADR-0001's new "Collection provisioning" section.
Why:
- Testcontainers' session-scoped container startup (~25s on a cold Docker
cache) was charged against the global 10s pytest-timeout budget, causing
every integration test to fail regardless of its own runtime.
Changes:
- Set timeout_func_only = true so the budget applies to the test function
only, not fixture setup.
Why:
- No setting in this app ever actually read from .env: only the outer
Settings declared env_file=".env", and pydantic-settings does not cascade
that to nested BaseSettings classes. Every previously-correct local value
was coincidence (.env.example defaults matching class defaults). Found by
testing EMBEDDING_OPENAI_API_KEY against the live OpenAI API.
Changes:
- Every nested settings class now declares env_file=".env" itself.
- Settings.__init__/EmbeddingSettings.__init__ explicitly thread an
_env_file override to every nested constructor, so overriding it (as
tests do) reaches the whole tree, not just the outer class.
- env_ignore_empty=True everywhere, since the fix surfaced a second bug:
a blank env var (e.g. EMBEDDING_OPENAI_DIMENSIONS=) failed to parse as
int | None instead of falling back to the field default.
Impact:
- Real deployments setting env vars directly (Docker Compose) are
unaffected. Local .env-file development now actually works.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
the inline upload path, with process-wide capacity/timeout/chunk-limit
guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
evaluation lab, which benchmarked them against the real Farsi corpus
(bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
at native 3072-dim), closing open items in ADR-0001/ADR-0005.
Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
helpers, and dense/sparse adapters (analyzers.py, bm25.py,
openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
(503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).
Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Code under test (auth resolution, the two-phase upload) opens more than
one session per operation; the existing fixture only exposed one
rolled-back session.
Changes:
- Add a db_sessionmaker fixture sharing one outer transaction.
- Pin loop_scope="session" -- without it, a second async test against the
session-scoped Postgres container fails with "Event loop is closed."
Why:
- Wires the ADR-0008 error envelope, per-request correlation id, and the
/v1/files routes into the app.
Changes:
- Extend AppResources/lifespan with the ingestion CapacityLimiter and
ObjectStorage adapter.
Impact:
- /v1 now exposes routes for the first time.
Why:
- Implements plan 001 Phase 3's upload orchestration.
Changes:
- ObjectStorage port and MinIO adapter, thread-offloaded per ADR-0017.
- Tenant-scoped repositories for api_keys, source_files, ingestion_jobs.
- upload_source_file implementing the two-transaction shape with
(tenant_id, domain, content_sha256) idempotency.
Impact:
- This phase stores bytes only -- chunks_indexed is always 0 until
Phase 4/5 add parsing/embedding.
Why:
- ADR-0017 names the bound INGESTION_MAX_UPLOAD_SIZE_MB; the code had it
as APP_MAX_UPLOAD_SIZE_MB. Reconciled to match the ADR, grouped with the
other three ADR-0017 bounds on IngestionSettings.
Why:
- The package exposed 8 modules directly, pushing source-type dispatch and
the ADR-0017 thread-offload obligation onto every caller.
Changes:
- Add parse_and_chunk_document as the sole public entry point.
- Demote the individual parsers to internal/test-only.
Testcontainers reports the container host as `localhost`, which resolves
to ::1 before 127.0.0.1 on this machine. Docker publishes the mapped port
on IPv4 only, and the IPv6 SYN is dropped rather than refused, so asyncpg
blocked on the first address until its connect timeout instead of falling
back to the second -- the migration fixture hung rather than failing.
127.0.0.1 connects in 0.07s where localhost timed out at 15s; the
integration suite now runs in 3.6s, inside the 10s pytest-timeout.
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
alembic/env.py previously always overwrote sqlalchemy.url from
Settings().postgres.dsn, which made it impossible for a test fixture to
point Alembic at a Testcontainers-managed database. Now env.py only
sets it when unset, and a new integration suite runs `alembic upgrade
head` against a real Postgres container per ADR-0016 (no create_all()).
Why:
- first runnable slice of ADR-0012's resource-lifetime rules and ADR-0015's package layout: app-lifetime clients built once in the lifespan, released via explicit dependencies.
Changes:
- Settings split into per-domain nested settings (postgres/minio/ingestion/qdrant/app/logging); FastAPI app wired with /healthz, /readyz and a /v1 router; Postgres/MinIO/Qdrant adapters and SQLAlchemy models for tenants, API keys, source files, ingestion jobs/events.
Why:
- a diagnostic quietly suppressed (noqa, type: ignore, Any, getattr, broad except, config downgrades) is functionally worse than one left failing.
Changes:
- hook now scans added lines per file/config edit for suppression patterns and reports them, and remembers prior diagnostics so one vanishing in the same edit that added a suppression marker is flagged rather than treated as fixed.
Why:
- ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path.
Changes:
- new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it.
Impact:
- no broker, outbox, or worker code; rabbitmq test marker removed.
Add the Explore subagent definition and a PostToolUse hook that runs
Ruff and ty on Python file edits, plus the settings.json wiring it in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add the initial src/ package (empty FastAPI entrypoint, Settings stub),
an .env.example placeholder, and expand .gitignore for test/coverage
artifacts, environment files, and local Claude Code overrides. Also
documents the Alembic async bootstrap command in ADR-0009.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add the README section and .env.langfuse.example template for running
Langfuse locally via Compose. The Compose file itself is tracked
separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Switch the durable ingestion/maintenance job-dispatch broker decision from
NATS JetStream to RabbitMQ (aio-pika), rewriting ADR-0014 and propagating
the terminology change through ADR-0015, ADR-0016, and the ingestion
vertical-slice plan. Adds aio-pika as a runtime dependency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Establish explicit ownership and lifetime rules for application, request/job, and operation-scoped resources.
Changes:
- Define FastAPI lifespan ownership for engines, clients, graphs, and pools.
- Define dependency-managed sessions and request context.
- Require explicit transaction boundaries and dependency passing.
- Prohibit shared global SQLAlchemy sessions and import-time network clients.
Impact:
- Application resources are created and closed by their process owner.
- Request and job resources must not be shared across concurrent units of work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why:
- Establish the versioned FastAPI boundary for authentication, tenant isolation, chat runs, file ingestion, point management, and health checks.
Changes:
- Define /v1 routers, bearer-token authentication, scopes, response envelopes, error conventions, and job-shaped ingestion responses.
- Rename the older indicative /chunks routes to /points while preserving the existing Qdrant payload and CRUD semantics.
- Define tenant injection and concurrency requirements at the HTTP boundary.
Impact:
- The REST API is owned by ADR-0008 when older endpoint examples differ.
- Clients should use /v1/points and /v1/threads resource paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
