Why:
- ADR-0002 already answered three of the four questions plan 002 listed as
"decisions needed"; the fourth -- what happens to a manually edited point when
its file is re-uploaded -- was left for a Phase 6 test to force. Deciding it in
code rather than in the ADR would invert this repo's rule.
Changes:
- ADR-0002 gains "Re-ingestion versus manual edits": the new file wins,
surviving points are overwritten with an incremented version, absent points
are flagged inactive rather than removed, and manually created points sit past
the ingested chunk_index range so the existing sweep covers them.
- Plan 002's stale decisions section becomes a pointer table; its audit scope is
pinned to both ADR-0009 tables.
Impact:
- Clobbered edits are recoverable from point_audit_events, not from Qdrant: the
deterministic point ID cannot hold both versions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- the slice had no operator documentation: no tuning guidance, no failure
procedure, no statement of the proxy timeout requirement.
Changes:
- add docs/runbook.md: startup, deployment steps, provisioning, the INGESTION_*
tuning table with each bound's status code, the proxy read-timeout rule,
/healthz vs /readyz, failure investigation by real event name plus job/event
SQL, retry semantics, and alert thresholds
- link it from the README and note provisioning there
- mark plan 001 Phase 6 done and refresh CLAUDE.md's status paragraph
Impact:
- documentation only
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Why:
- POST /v1/files was reporting chunks_indexed=0/points_created=0 unconditionally
— chunks were parsed and embedded but never written to Qdrant, so nothing
was actually searchable after upload.
Changes:
- upload_source_file() now calls index_chunks() after embedding, inside the
same INGESTION_TIMEOUT_SECONDS window, and marks the job failed
(error_code=index_failed, 502) if it raises.
- Job counters (points_created, points_soft_deleted) and the response's
chunks_indexed now reflect the real indexing result instead of a hardcoded
zero.
- Wired PointStorage through AppResources/lifespan/the files router.
Impact:
- A successful upload is now searchable in Qdrant by the time 201 returns.
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
the inline upload path, with process-wide capacity/timeout/chunk-limit
guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
evaluation lab, which benchmarked them against the real Farsi corpus
(bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
at native 3072-dim), closing open items in ADR-0001/ADR-0005.
Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
helpers, and dense/sparse adapters (analyzers.py, bm25.py,
openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
(503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).
Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
Why:
- ingestion is inline in the request instead of dispatched through RabbitMQ/outbox/worker; ADR-0014 is superseded (not deleted) and named as the design to adopt once ingestion needs to move off the request path.
Changes:
- new ADR-0017 plus amendments to every ADR/plan that referenced the job-shaped/broker contract, so none silently contradict it.
Impact:
- no broker, outbox, or worker code; rabbitmq test marker removed.
Switch the durable ingestion/maintenance job-dispatch broker decision from
NATS JetStream to RabbitMQ (aio-pika), rewriting ADR-0014 and propagating
the terminology change through ADR-0015, ADR-0016, and the ingestion
vertical-slice plan. Adds aio-pika as a runtime dependency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
