Commit Graph

2 Commits

Author SHA1 Message Date
Ali Zarinkolah
9e8987968c feat(observability): add dual local logging sinks and static environment context
Why:
- Wanted human-readable console output while developing locally, without
  losing a machine-parseable log for later grepping/parsing. A single
  renderer chosen by a flag can't do both at once.
- ADR-0011 had no way to correlate an issue with a specific deployment
  (build/region/instance) independent of any one request.

Changes:
- configure_logging() now builds two independent handlers: console (always
  on, colored unless LOG_JSON_FORMAT=true) and an optional rotating JSON file
  (LOG_FILE_PATH, unset by default) -- the same structlog event fans out to
  both, so call sites are unaffected.
- A static structlog processor binds env/service_version onto every event.
  Deliberately not a contextvar: RequestIdMiddleware's clear_contextvars()
  would wipe a value bound there before the first request.
- New settings: APP_SERVICE_VERSION, LOG_FILE_PATH/LOG_FILE_MAX_BYTES/
  LOG_FILE_BACKUP_COUNT.
- ADR-0011 amended with both decisions ("console and file are independent
  sinks locally"; "bind process-level environment context once at startup").

Impact:
- configure_logging() signature changed to (logging_settings, app_settings);
  both call sites (lifespan, qdrant_bootstrap CLI) updated.
2026-08-20 19:20:27 +03:30
5c0a5938f8 feat(ingestion): add bounded, benchmark-aligned embedding execution
Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
  the inline upload path, with process-wide capacity/timeout/chunk-limit
  guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
  evaluation lab, which benchmarked them against the real Farsi corpus
  (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
  at native 3072-dim), closing open items in ADR-0001/ADR-0005.

Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
  helpers, and dense/sparse adapters (analyzers.py, bm25.py,
  openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
  (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
  every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
  creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).

Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-19 17:13:32 +03:30