feat(ingestion): add bounded, benchmark-aligned embedding execution

Why:
- Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into
  the inline upload path, with process-wide capacity/timeout/chunk-limit
  guards (ADR-0017).
- The BM25 analyzer and dense-model config are ported from the `emet`
  evaluation lab, which benchmarked them against the real Farsi corpus
  (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large
  at native 3072-dim), closing open items in ADR-0001/ADR-0005.

Changes:
- New: embedding ports, orchestration (embed_chunks), request-bounds
  helpers, and dense/sparse adapters (analyzers.py, bm25.py,
  openai_compatible.py).
- upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY
  (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413);
  every failure path still writes a terminal job row.
- Lifespan builds and warms both dense embedders at startup (fail-soft) and
  creates the sparse embedder and concurrency semaphore.
- httpx moves from dev to main dependencies (adapters use it directly).

Impact:
- Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0.
- New EMBEDDING_* env vars documented in .env.example; safe defaults.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md

@@ -2,9 +2,10 @@
 
 ## Status
 
-Proposed — the fusion/rerank *shape* and reranker model are decided; the
-final BM25 analyzer and the commercial license status of the reranker are
-still open per the follow-up items below.
+Proposed — the fusion/rerank *shape*, the reranker model, and (as of the
+`emet` benchmark, see "Benchmark outcome" below) the **BM25 analyzer** are
+decided. The commercial license status of the reranker remains open per the
+follow-up items below.
 
 ## Context
 
@@ -100,6 +101,50 @@ entirely to the analyzer stage, not the ranking formula:
 consistent with Farsi's high density of function words (ezafe particles,
 prepositions, common verbs) adding TF/IDF noise if left in.
 
+### 3a. Benchmark outcome: `bm25-fa-norm-stop` confirmed, and where the BM25 math runs
+
+The `emet` evaluation lab (`~/code/talie/emet`) ran the four-variant
+comparison above against the real Farsi corpus and confirmed
+**`bm25-fa-norm-stop`** as the winner. It is the only sparse variant promoted
+into emet's hybrid matrix (`emet/hybrid.yaml`). This closes follow-up item 4
+below.
+
+The winning analyzer is a specific, reproducible artifact, ported into
+`src/infrastructure/embedding/analyzers.py` and verified token-for-token
+against emet's implementation. Its details are load-bearing:
+
+- Unicode **NFC** (not NFKC), then ZWNJ → space, then Persian/Arabic-Indic
+  digits → ASCII, then `ي→ی ك→ک ة→ه ؤ→و إ→ا أ→ا`.
+- Tokenizer `[^\W_]+`, which **keeps digits**. This matters for an insurance
+  corpus: policy numbers, dates, and amounts are exactly the terms lexical
+  retrieval should match, and the digit folding above means a query in ASCII
+  digits matches a document authored in Persian ones.
+- A 51-entry stopword set (40 Persian/Arabic + 11 English, the corpus being
+  mixed-script). Deliberately not a full `hazm` list.
+- No stemming — `fa_norm_stem` was the losing arm.
+
+**The BM25 formula is split across two systems, deliberately.** The client
+applies term-frequency saturation, including the `k`/`b` document-length
+normalization; **IDF is supplied by Qdrant** via `modifier="idf"` on the
+sparse vector field, computed from collection-wide statistics rather than
+from a fixed client-side corpus.
+
+That split is a correctness trap worth stating plainly: a `chunks` collection
+created *without* `modifier="idf"` will score these vectors as saturated term
+frequencies with no IDF weighting at all — no error, no warning, just
+materially worse lexical retrieval. The collection bootstrap must set it.
+
+Document and query encoding are asymmetric in exactly one term: documents
+carry the `b` length normalization, queries do not (standard BM25 practice).
+Both sides must therefore encode through the same implementation, which is
+why the sparse port carries a `query` flag rather than leaving retrieval to
+grow a second, silently divergent encoder.
+
+Term → sparse-index mapping is `blake2b(token, digest_size=8) % (2**31 - 1)`,
+a pure hash with no vocabulary table, so it needs no shared state and stays
+identical across processes and between ingest and query time. Changing the
+hash orphans every stored sparse vector: that is a re-ingestion, not a deploy.
+
 ### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula
 
 These are standard, well-validated defaults (Trotman, Puurula & Burgess,
@@ -159,8 +204,21 @@ comparison and `b` sweep in the follow-ups below.
 3. Run an ablation: single dense model + sparse + rerank vs. the current
    dual-dense-model + sparse + rerank setup, on real Farsi queries, to
    justify (or drop) the second dense vector (`dense_openai`).
-4. Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation to
-   determine whether gains come from stopword removal, stemming, or both.
+4. ~~Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation~~ —
+   **done**, see "Benchmark outcome" above. `fa_norm_stop` won; stemming was
+   not adopted.
 5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document
    length varies significantly across the corpus (short chat messages vs.
    long articles) and `0.75` is a generic default, not corpus-tuned.
+6. **Recalibrate `avg_len`.** The client-side `b` term needs an average
+   document length in *analyzer tokens*. The ported value (256.0) is emet's
+   own placeholder, and emet measured it over short Q&A records rather than
+   this service's ~400-token chunks, so it is very likely miscalibrated here.
+   Exposed as `EMBEDDING_SPARSE_AVG_LEN` so it can be corrected from real
+   corpus statistics without a code change.
+7. **Re-benchmark the analyzer with diacritic stripping.** `fa_norm_stop`
+   does not remove harakat or tatweel, so `ســلام` and `سلام` are distinct
+   terms. `src/application/ingestion/normalization.py` already strips both
+   for chunk *content*; extending that to the analyzer is plausibly an
+   improvement but would deviate from the measured configuration, so it
+   belongs in an emet run rather than an unmeasured edit.