chatbot_v3/docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md

0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus

Status

Proposed — the fusion/rerank shape, the reranker model, and (as of the emet benchmark, see "Benchmark outcome" below) the BM25 analyzer are decided. The commercial license status of the reranker remains open per the follow-up items below.

Context

ADR-0003 fixed the retrieval shape (dense_nomic + dense_openai + sparse prefetch → RRF fusion → late-interaction rerank) but deliberately left two things open: which model powers late_interaction, and the sparse-side analyzer detail behind bm25-fa-norm-stop (ADR-0001). This ADR resolves those, driven by the corpus being Farsi (Persian) — a morphologically rich, low-resource language for most public embedding/rerank benchmarks and for Qdrant's own hosted tooling.

Two things specific to Farsi drove this investigation rather than picking a generic default:

• Qdrant's hosted Qdrant/bm25 FastEmbed model's documented supported- language list does not include Farsi (fa) stemming — using it as-is would silently apply no stemming rule, or the wrong one, for this corpus.
• Farsi's morphology (verb conjugation, ezafe constructions, high-frequency function words) makes pure BM25/keyword signals noisier than in English, which raised the bar on how load-bearing the rerank stage needs to be — treated here as close to mandatory for quality, not optional.

Decision

1. Fusion: RRF by default, weighted RRF as a fallback

Confirms ADR-0003's fusion stage combines all three prefetch results (dense_nomic, dense_openai, sparse) via RRF by default. If one signal (typically sparse, on Farsi text) is observed to dominate the fused ranking unexpectedly, weighted RRF is the fallback — not a switch to a different fusion algorithm. This refines, not replaces, ADR-0003's fusion decision.

2. Late-interaction reranker: jina-colbert-v2

Two multilingual late-interaction (ColBERT-style) options were evaluated against the requirement of confirmed Farsi support:

Model	Farsi support	License	Local hosting
jina-colbert-v2	Confirmed — fa explicitly listed among 89 supported languages	Conflicting: HF repo metadata says cc-by-4.0; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only	Possible via PyLate/RAGatouille; requires flash_attn (CUDA GPU effectively required); reported loading issues via generic transformers.AutoModel
LFM2-ColBERT-350M	Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es)	—	—
BGE-M3 (alternative, not adopted)	Strong Farsi performance in the FaMTEB benchmark	Apache-2.0 — unambiguous, commercial-friendly	CPU-runnable, no flash_attn dependency

jina-colbert-v2 is adopted as the late_interaction reranker: it has explicitly confirmed Farsi support among its 89 languages, which is the primary requirement for this corpus. This means:

• A CUDA GPU is now a required dependency for the rerank stage (flash_attn), not optional infrastructure — this changes the self-hosted Docker deployment assumption from ADR-0001/0003, which had not committed to GPU hosting.
• The commercial license is unresolved — HF repo metadata states cc-by-4.0 while Jina's own announcement states CC BY-NC-4.0 (non-commercial, with commercial use only via Jina's paid API/AWS/Azure offering). This must be confirmed directly with Jina before this project ships commercially on a self-hosted jina-colbert-v2 model; if Jina confirms the non-commercial reading, self-hosting it commercially is not an option and the fallback is Jina's paid hosted API or BGE-M3 as a substitute reranker (same late_interaction vector shape, no schema change needed either way).
• Loading it outside PyLate/RAGatouille (e.g. generic transformers.AutoModel) has reported issues — plan to load it through one of those two libraries, not a raw transformers call.

late_interaction continues to use hnsw_config: m=0 (per ADR-0001, now made explicit: HNSW indexing is disabled for this vector because it is used only for reranking already-fetched candidates via MAX_SIM, never for independent ANN search — the recommended pattern for rerank-only multivectors) plus on-disk storage.

3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model

Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi normalization/stopword/stemming computed outside Qdrant Cloud Inference) is used instead of Qdrant's hosted Qdrant/bm25 FastEmbed model, specifically because that hosted model's documented language list omits Farsi. The sparse vector is uploaded with modifier="idf", the standard BM25 sparse vector configuration.

Four analyzer variants were benchmarked, all sharing identical BM25 scoring parameters (k=1.2, b=0.75) so any accuracy difference is attributable entirely to the analyzer stage, not the ranking formula:

Analyzer	Description
bm25-raw	no normalization
bm25-fa-norm	Farsi normalization only
bm25-fa-norm-stop	normalization + stopword removal — current best performer
bm25-fa-norm-stem	normalization + stemming

bm25-fa-norm-stop is confirmed as the sparse analyzer named in ADR-0001, consistent with Farsi's high density of function words (ezafe particles, prepositions, common verbs) adding TF/IDF noise if left in.

3a. Benchmark outcome: bm25-fa-norm-stop confirmed, and where the BM25 math runs

The emet evaluation lab (~/code/talie/emet) ran the four-variant comparison above against the real Farsi corpus and confirmed bm25-fa-norm-stop as the winner. It is the only sparse variant promoted into emet's hybrid matrix (emet/hybrid.yaml). This closes follow-up item 4 below.

The winning analyzer is a specific, reproducible artifact, ported into src/infrastructure/embedding/analyzers.py and verified token-for-token against emet's implementation. Its details are load-bearing:

• Unicode NFC (not NFKC), then ZWNJ → space, then Persian/Arabic-Indic digits → ASCII, then ي→ی ك→ک ة→ه ؤ→و إ→ا أ→ا.
• Tokenizer [^\W_]+, which keeps digits. This matters for an insurance corpus: policy numbers, dates, and amounts are exactly the terms lexical retrieval should match, and the digit folding above means a query in ASCII digits matches a document authored in Persian ones.
• A 51-entry stopword set (40 Persian/Arabic + 11 English, the corpus being mixed-script). Deliberately not a full hazm list.
• No stemming — fa_norm_stem was the losing arm.

The BM25 formula is split across two systems, deliberately. The client applies term-frequency saturation, including the k/b document-length normalization; IDF is supplied by Qdrant via modifier="idf" on the sparse vector field, computed from collection-wide statistics rather than from a fixed client-side corpus.

That split is a correctness trap worth stating plainly: a chunks collection created without modifier="idf" will score these vectors as saturated term frequencies with no IDF weighting at all — no error, no warning, just materially worse lexical retrieval. The collection bootstrap must set it.

Document and query encoding are asymmetric in exactly one term: documents carry the b length normalization, queries do not (standard BM25 practice). Both sides must therefore encode through the same implementation, which is why the sparse port carries a query flag rather than leaving retrieval to grow a second, silently divergent encoder.

Term → sparse-index mapping is blake2b(token, digest_size=8) % (2**31 - 1), a pure hash with no vocabulary table, so it needs no shared state and stays identical across processes and between ingest and query time. Changing the hash orphans every stored sparse vector: that is a re-ingestion, not a deploy.

4. BM25 parameters: keep k=1.2, b=0.75; tune analyzer, not formula

These are standard, well-validated defaults (Trotman, Puurula & Burgess, 2014) and are not the source of the observed analyzer-variant accuracy differences — so formula tuning is deprioritized in favor of the analyzer comparison and b sweep in the follow-ups below.

Consequences

Positive

• Resolves ADR-0003's two deferred decisions (reranker model, sparse analyzer detail) with a corpus-specific rationale instead of a generic default.
• jina-colbert-v2 is the only evaluated option with explicitly confirmed Farsi support, directly matching this project's primary requirement.
• Isolating BM25 formula parameters from analyzer choice gives a clean, defensible experimental basis for the bm25-fa-norm-stop selection.

Negative

• Introduces a hard GPU dependency (flash_attn/CUDA) for the rerank stage that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU capacity, not just Docker on commodity hardware.
• Commercial license status is unresolved; shipping this commercially on a self-hosted jina-colbert-v2 model without confirming licensing with Jina is a legal risk, not just a technical one.
• Loading path is constrained to PyLate/RAGatouille due to reported transformers.AutoModel issues — an extra library dependency and less flexibility than a standard transformers load would give.
• bm25-fa-norm-stop is provisional until compared directly against bm25-fa-norm-stem — Persian stemming can over-collapse distinct words (irregular verb conjugation, Arabic-loanword plurals), so the current "best performer" result could shift.

Alternatives Considered

• BGE-M3 as the reranker: not adopted — Apache-2.0 license and CPU-runnability are attractive and it remains the fallback if jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2 was prioritized for its explicit Farsi support.
• LFM2-ColBERT-350M: rejected outright — no Farsi support among its 8 supported languages.
• Qdrant's hosted Qdrant/bm25 FastEmbed model: rejected — its documented language support does not include Farsi stemming; using it would risk silently wrong or absent stemming for this corpus.
• Treating "RRF vs. rerank" as either/or: rejected — RRF alone can't resolve disagreement between two distinct dense embedding spaces plus a noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank pipeline from ADR-0003 is kept, not replaced with either component alone.

Follow-up / Open Items

1. Confirm jina-colbert-v2's commercial license status directly with Jina before commercial deployment; fall back to Jina's paid hosted API or BGE-M3 if the non-commercial reading is confirmed.
2. Provision GPU capacity for self-hosted jina-colbert-v2 (flash_attn requires CUDA) as part of the deployment plan, not an afterthought.
3. Run an ablation: single dense model + sparse + rerank vs. the current dual-dense-model + sparse + rerank setup, on real Farsi queries, to justify (or drop) the second dense vector (dense_openai).
4. Compare bm25-fa-norm-stop vs. bm25-fa-norm-stem in isolation — done, see "Benchmark outcome" above. fa_norm_stop won; stemming was not adopted.
5. Sweep BM25 b (e.g. 0.5–0.9) for the winning analyzer, since document length varies significantly across the corpus (short chat messages vs. long articles) and 0.75 is a generic default, not corpus-tuned.
6. Recalibrate avg_len. The client-side b term needs an average document length in analyzer tokens. The ported value (256.0) is emet's own placeholder, and emet measured it over short Q&A records rather than this service's ~400-token chunks, so it is very likely miscalibrated here. Exposed as EMBEDDING_SPARSE_AVG_LEN so it can be corrected from real corpus statistics without a code change.
7. Re-benchmark the analyzer with diacritic stripping. fa_norm_stop does not remove harakat or tatweel, so ســلام and سلام are distinct terms. src/application/ingestion/normalization.py already strips both for chunk content; extending that to the analyzer is plausibly an improvement but would deviate from the measured configuration, so it belongs in an emet run rather than an unmeasured edit.