# 0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus ## Status Proposed — the fusion/rerank *shape*, the reranker model, and (as of the `emet` benchmark, see "Benchmark outcome" below) the **BM25 analyzer** are decided. The commercial license status of the reranker remains open per the follow-up items below. ## Context ADR-0003 fixed the retrieval *shape* (dense_nomic + dense_openai + sparse prefetch → RRF fusion → late-interaction rerank) but deliberately left two things open: which model powers `late_interaction`, and the sparse-side analyzer detail behind `bm25-fa-norm-stop` (ADR-0001). This ADR resolves those, driven by the corpus being **Farsi (Persian)** — a morphologically rich, low-resource language for most public embedding/rerank benchmarks and for Qdrant's own hosted tooling. Two things specific to Farsi drove this investigation rather than picking a generic default: - Qdrant's hosted `Qdrant/bm25` FastEmbed model's documented supported- language list does not include Farsi (`fa`) stemming — using it as-is would silently apply no stemming rule, or the wrong one, for this corpus. - Farsi's morphology (verb conjugation, ezafe constructions, high-frequency function words) makes pure BM25/keyword signals noisier than in English, which raised the bar on how load-bearing the rerank stage needs to be — treated here as close to mandatory for quality, not optional. ## Decision ### 1. Fusion: RRF by default, weighted RRF as a fallback Confirms ADR-0003's fusion stage combines all three prefetch results (`dense_nomic`, `dense_openai`, `sparse`) via **RRF** by default. If one signal (typically sparse, on Farsi text) is observed to dominate the fused ranking unexpectedly, **weighted RRF** is the fallback — not a switch to a different fusion algorithm. This refines, not replaces, ADR-0003's fusion decision. ### 2. Late-interaction reranker: jina-colbert-v2 Two multilingual late-interaction (ColBERT-style) options were evaluated against the requirement of confirmed Farsi support: | Model | Farsi support | License | Local hosting | |---|---|---|---| | **jina-colbert-v2** | Confirmed — `fa` explicitly listed among 89 supported languages | Conflicting: HF repo metadata says `cc-by-4.0`; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only | Possible via PyLate/RAGatouille; requires `flash_attn` (CUDA GPU effectively required); reported loading issues via generic `transformers.AutoModel` | | **LFM2-ColBERT-350M** | Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es) | — | — | | BGE-M3 (alternative, not adopted) | Strong Farsi performance in the FaMTEB benchmark | Apache-2.0 — unambiguous, commercial-friendly | CPU-runnable, no `flash_attn` dependency | **jina-colbert-v2 is adopted** as the `late_interaction` reranker: it has explicitly confirmed Farsi support among its 89 languages, which is the primary requirement for this corpus. This means: - A **CUDA GPU is now a required dependency** for the rerank stage (`flash_attn`), not optional infrastructure — this changes the self-hosted Docker deployment assumption from ADR-0001/0003, which had not committed to GPU hosting. - The **commercial license is unresolved** — HF repo metadata states `cc-by-4.0` while Jina's own announcement states CC BY-NC-4.0 (non-commercial, with commercial use only via Jina's paid API/AWS/Azure offering). This must be confirmed directly with Jina **before** this project ships commercially on a self-hosted jina-colbert-v2 model; if Jina confirms the non-commercial reading, self-hosting it commercially is not an option and the fallback is Jina's paid hosted API or BGE-M3 as a substitute reranker (same `late_interaction` vector shape, no schema change needed either way). - Loading it outside PyLate/RAGatouille (e.g. generic `transformers.AutoModel`) has reported issues — plan to load it through one of those two libraries, not a raw `transformers` call. `late_interaction` continues to use `hnsw_config: m=0` (per ADR-0001, now made explicit: HNSW indexing is disabled for this vector because it is used only for reranking already-fetched candidates via MAX_SIM, never for independent ANN search — the recommended pattern for rerank-only multivectors) plus on-disk storage. ### 3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi normalization/stopword/stemming computed outside Qdrant Cloud Inference) is used instead of Qdrant's hosted `Qdrant/bm25` FastEmbed model, specifically because that hosted model's documented language list omits Farsi. The sparse vector is uploaded with `modifier="idf"`, the standard BM25 sparse vector configuration. Four analyzer variants were benchmarked, all sharing identical BM25 scoring parameters (`k=1.2`, `b=0.75`) so any accuracy difference is attributable entirely to the analyzer stage, not the ranking formula: | Analyzer | Description | |---|---| | `bm25-raw` | no normalization | | `bm25-fa-norm` | Farsi normalization only | | `bm25-fa-norm-stop` | normalization + stopword removal — **current best performer** | | `bm25-fa-norm-stem` | normalization + stemming | `bm25-fa-norm-stop` is confirmed as the sparse analyzer named in ADR-0001, consistent with Farsi's high density of function words (ezafe particles, prepositions, common verbs) adding TF/IDF noise if left in. ### 3a. Benchmark outcome: `bm25-fa-norm-stop` confirmed, and where the BM25 math runs The `emet` evaluation lab (`~/code/talie/emet`) ran the four-variant comparison above against the real Farsi corpus and confirmed **`bm25-fa-norm-stop`** as the winner. It is the only sparse variant promoted into emet's hybrid matrix (`emet/hybrid.yaml`). This closes follow-up item 4 below. The winning analyzer is a specific, reproducible artifact, ported into `src/infrastructure/embedding/analyzers.py` and verified token-for-token against emet's implementation. Its details are load-bearing: - Unicode **NFC** (not NFKC), then ZWNJ → space, then Persian/Arabic-Indic digits → ASCII, then `ي→ی ك→ک ة→ه ؤ→و إ→ا أ→ا`. - Tokenizer `[^\W_]+`, which **keeps digits**. This matters for an insurance corpus: policy numbers, dates, and amounts are exactly the terms lexical retrieval should match, and the digit folding above means a query in ASCII digits matches a document authored in Persian ones. - A 51-entry stopword set (40 Persian/Arabic + 11 English, the corpus being mixed-script). Deliberately not a full `hazm` list. - No stemming — `fa_norm_stem` was the losing arm. **The BM25 formula is split across two systems, deliberately.** The client applies term-frequency saturation, including the `k`/`b` document-length normalization; **IDF is supplied by Qdrant** via `modifier="idf"` on the sparse vector field, computed from collection-wide statistics rather than from a fixed client-side corpus. That split is a correctness trap worth stating plainly: a `chunks` collection created *without* `modifier="idf"` will score these vectors as saturated term frequencies with no IDF weighting at all — no error, no warning, just materially worse lexical retrieval. The collection bootstrap must set it. Document and query encoding are asymmetric in exactly one term: documents carry the `b` length normalization, queries do not (standard BM25 practice). Both sides must therefore encode through the same implementation, which is why the sparse port carries a `query` flag rather than leaving retrieval to grow a second, silently divergent encoder. Term → sparse-index mapping is `blake2b(token, digest_size=8) % (2**31 - 1)`, a pure hash with no vocabulary table, so it needs no shared state and stays identical across processes and between ingest and query time. Changing the hash orphans every stored sparse vector: that is a re-ingestion, not a deploy. ### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula These are standard, well-validated defaults (Trotman, Puurula & Burgess, 2014) and are not the source of the observed analyzer-variant accuracy differences — so formula tuning is deprioritized in favor of the analyzer comparison and `b` sweep in the follow-ups below. ## Consequences ### Positive - Resolves ADR-0003's two deferred decisions (reranker model, sparse analyzer detail) with a corpus-specific rationale instead of a generic default. - jina-colbert-v2 is the only evaluated option with explicitly confirmed Farsi support, directly matching this project's primary requirement. - Isolating BM25 formula parameters from analyzer choice gives a clean, defensible experimental basis for the `bm25-fa-norm-stop` selection. ### Negative - Introduces a hard GPU dependency (`flash_attn`/CUDA) for the rerank stage that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU capacity, not just Docker on commodity hardware. - Commercial license status is unresolved; shipping this commercially on a self-hosted jina-colbert-v2 model without confirming licensing with Jina is a legal risk, not just a technical one. - Loading path is constrained to PyLate/RAGatouille due to reported `transformers.AutoModel` issues — an extra library dependency and less flexibility than a standard `transformers` load would give. - `bm25-fa-norm-stop` is provisional until compared directly against `bm25-fa-norm-stem` — Persian stemming can over-collapse distinct words (irregular verb conjugation, Arabic-loanword plurals), so the current "best performer" result could shift. ## Alternatives Considered - **BGE-M3 as the reranker**: not adopted — Apache-2.0 license and CPU-runnability are attractive and it remains the fallback if jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2 was prioritized for its explicit Farsi support. - **LFM2-ColBERT-350M**: rejected outright — no Farsi support among its 8 supported languages. - **Qdrant's hosted `Qdrant/bm25` FastEmbed model**: rejected — its documented language support does not include Farsi stemming; using it would risk silently wrong or absent stemming for this corpus. - **Treating "RRF vs. rerank" as either/or**: rejected — RRF alone can't resolve disagreement between two distinct dense embedding spaces plus a noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank pipeline from ADR-0003 is kept, not replaced with either component alone. ## Follow-up / Open Items 1. Confirm jina-colbert-v2's commercial license status directly with Jina before commercial deployment; fall back to Jina's paid hosted API or BGE-M3 if the non-commercial reading is confirmed. 2. Provision GPU capacity for self-hosted jina-colbert-v2 (`flash_attn` requires CUDA) as part of the deployment plan, not an afterthought. 3. Run an ablation: single dense model + sparse + rerank vs. the current dual-dense-model + sparse + rerank setup, on real Farsi queries, to justify (or drop) the second dense vector (`dense_openai`). 4. ~~Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation~~ — **done**, see "Benchmark outcome" above. `fa_norm_stop` won; stemming was not adopted. 5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document length varies significantly across the corpus (short chat messages vs. long articles) and `0.75` is a generic default, not corpus-tuned. 6. **Recalibrate `avg_len`.** The client-side `b` term needs an average document length in *analyzer tokens*. The ported value (256.0) is emet's own placeholder, and emet measured it over short Q&A records rather than this service's ~400-token chunks, so it is very likely miscalibrated here. Exposed as `EMBEDDING_SPARSE_AVG_LEN` so it can be corrected from real corpus statistics without a code change. 7. **Re-benchmark the analyzer with diacritic stripping.** `fa_norm_stop` does not remove harakat or tatweel, so `ســلام` and `سلام` are distinct terms. `src/application/ingestion/normalization.py` already strips both for chunk *content*; extending that to the analyzer is plausibly an improvement but would deviate from the measured configuration, so it belongs in an emet run rather than an unmeasured edit.