Why:
- Both are contracts callers depend on, not implementation details, and neither
was written down. This repo treats the ADR as the source of truth rather than
letting code diverge from it silently.
Changes:
- Record that the search query is folded the same way ingested content was, and
why the alternative fails in the worst available way: an exact-looking query
returning nothing, with no error and nothing in the logs to distinguish it
from a genuine miss.
- Record that listing requires file_id and paginates by order_id value, and why
an offset cursor repeats an already-served row under a concurrent insert.
- State that results carry no relevance score and no ranked order, so callers
cannot read array position as relevance.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- points:read and points:write are named in ADR-0009 but were absent from
DEFAULT_SCOPES, so a provisioned key could not reach the read paths that
follow. The runbook documented only files:write and domains:write, which
understated what a default key can now do.
Changes:
- Add points:read and points:write to DEFAULT_SCOPES.
- Document the full scope table in the runbook, calling out that points:read
grants the text of every chunk of every file -- so an upload-only key gets
files:write alone.
Impact:
- Keys issued before this change keep their existing scopes; provisioning does
not backfill. Reissue or widen an existing key explicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- ADR-0002 already answered three of the four questions plan 002 listed as
"decisions needed"; the fourth -- what happens to a manually edited point when
its file is re-uploaded -- was left for a Phase 6 test to force. Deciding it in
code rather than in the ADR would invert this repo's rule.
Changes:
- ADR-0002 gains "Re-ingestion versus manual edits": the new file wins,
surviving points are overwritten with an incremented version, absent points
are flagged inactive rather than removed, and manually created points sit past
the ingested chunk_index range so the existing sweep covers them.
- Plan 002's stale decisions section becomes a pointer table; its audit scope is
pinned to both ADR-0009 tables.
Impact:
- Clobbered edits are recoverable from point_audit_events, not from Qdrant: the
deterministic point ID cannot hold both versions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- the slice had no operator documentation: no tuning guidance, no failure
procedure, no statement of the proxy timeout requirement.
Changes:
- add docs/runbook.md: startup, deployment steps, provisioning, the INGESTION_*
tuning table with each bound's status code, the proxy read-timeout rule,
/healthz vs /readyz, failure investigation by real event name plus job/event
SQL, retry semantics, and alert thresholds
- link it from the README and note provisioning there
- mark plan 001 Phase 6 done and refresh CLAUDE.md's status paragraph
Impact:
- documentation only
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- Wanted human-readable console output while developing locally, without
losing a machine-parseable log for later grepping/parsing. A single
renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
(build/region/instance) independent of any one request.
Changes:
- configure_logging() now builds two independent handlers: console (always
on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
(LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
sinks locally"; "bind process-level environment context once at startup").
Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
both call sites (lifespan, qdrant_bootstrap CLI) updated.
Why:
- Domain values are denormalized into every Qdrant point payload. Without
validation, an unregistered or typo'd domain (e.g. "fier" for "fire")
silently creates a new partition that retrieval never queries — the file
ends up invisible rather than rejected. Tenants also need independently
sized domain sets (one may run 14 insurance lines, another 6), which rules
out an enum.
Changes:
- tenant_domains table (migration 41335d162de8) + repository, unique on
(tenant_id, domain).
- src/application/domains/: ensure_domain_allowed() is the strict-allowlist
check now run inside upload_source_file()'s first transaction, before any
MinIO object, job row, or Qdrant point is written.
- /v1/domains (list/create/patch/disable/enable) gated on its own
domains:read/domains:write scopes, deliberately separate from files:write
so an upload key cannot create partitions. domain itself is immutable
(denormalized into every point payload); only display_name is editable.
Disable blocks new uploads without touching already-indexed points.
Impact:
- BREAKING: POST /v1/files now rejects any domain without an active
tenant_domains row (400, unknown_domain). A domain must be created via
POST /v1/domains before the first upload to it.
Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
— chunks were parsed and embedded but never written to Qdrant, so nothing
was actually searchable after upload.
Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
(error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
chunks_indexed now reflect the real indexing result instead of a hardcoded
zero.
- Wired PointStorage through AppResources/lifespan/the files router.
Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
Why:
- The chunks collection needs four named vectors (dense_nomic, dense_openai,
sparse, late_interaction) and payload indexes defined at creation time per
ADR-0001; sparse/multivector fields cannot be added to an existing
collection without recreating it, so schema drift here is expensive.
- Creating it at FastAPI startup would mirror the DDL-at-boot anti-pattern
ADR-0009 already rejects for Postgres and ADR-0012 rejects for LangGraph's
setup(), so it is a deployment step instead.
Changes:
- src/infrastructure/qdrant/collection.py: ensure_chunks_collection(),
idempotent and schema-verifying (raises on dimension/modifier mismatch
rather than silently accepting a misconfigured collection).
- src/cli/qdrant_bootstrap.py: the operator entry point
(python -m src.cli.qdrant_bootstrap).
- QdrantSettings gains collection/upsert_batch_size/upsert_concurrency.
Impact:
- Deployments must run the new bootstrap command before the first upload;
see ADR-0001's new "Collection provisioning" section.
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
the inline upload path, with process-wide capacity/timeout/chunk-limit
guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
evaluation lab, which benchmarked them against the real Farsi corpus
(bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
at native 3072-dim), closing open items in ADR-0001/ADR-0005.
Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
helpers, and dense/sparse adapters (analyzers.py, bm25.py,
openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
(503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).
Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
Why:
- ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path.
Changes:
- new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it.
Impact:
- no broker, outbox, or worker code; rabbitmq test marker removed.
Add the initial src/ package (empty FastAPI entrypoint, Settings stub),
an .env.example placeholder, and expand .gitignore for test/coverage
artifacts, environment files, and local Claude Code overrides. Also
documents the Alembic async bootstrap command in ADR-0009.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Switch the durable ingestion/maintenance job-dispatch broker decision from
NATS JetStream to RabbitMQ (aio-pika), rewriting ADR-0014 and propagating
the terminology change through ADR-0015, ADR-0016, and the ingestion
vertical-slice plan. Adds aio-pika as a runtime dependency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Why:
- Establish explicit ownership and lifetime rules for application, request/job, and operation-scoped resources.
Changes:
- Define FastAPI lifespan ownership for engines, clients, graphs, and pools.
- Define dependency-managed sessions and request context.
- Require explicit transaction boundaries and dependency passing.
- Prohibit shared global SQLAlchemy sessions and import-time network clients.
Impact:
- Application resources are created and closed by their process owner.
- Request and job resources must not be shared across concurrent units of work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why:
- Establish the versioned FastAPI boundary for authentication, tenant isolation, chat runs, file ingestion, point management, and health checks.
Changes:
- Define /v1 routers, bearer-token authentication, scopes, response envelopes, error conventions, and job-shaped ingestion responses.
- Rename the older indicative /chunks routes to /points while preserving the existing Qdrant payload and CRUD semantics.
- Define tenant injection and concurrency requirements at the HTTP boundary.
Impact:
- The REST API is owned by ADR-0008 when older endpoint examples differ.
- Clients should use /v1/points and /v1/threads resource paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why:
- Establish the application-owned relational source of truth for tenants, authentication, ingestion, audit, graph runs, LLM usage, and feedback.
Changes:
- Define SQLAlchemy 2.x and Alembic conventions.
- Specify tenant, API-key, ingestion, audit, graph-run, pricing, usage, and feedback tables.
- Document indexing, retention, privacy, and multitenancy rules.
Impact:
- Postgres schema changes must be implemented through Alembic migrations.
- FastAPI must not run DDL during startup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why:
- Design the LangGraph-based insurance chatbot: retrieval-grounded
answering, grounded clarification, and handoff to a human assistant
owned by the existing main backend.
Changes:
- ADR-0006: graph nodes/edges, escalation triggers, and the
backend-owned handoff signal contract.
- ADR-0007: Postgres checkpointer, thread model (thread_id as the
shared identifier), history trimming strategy, and cross-thread
user memory via the Store.
Impact:
- Design-only; no code changes yet.
