chatbot_v3/docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md

0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus

Status

Proposed — the fusion/rerank shape and reranker model are decided; the final BM25 analyzer and the commercial license status of the reranker are still open per the follow-up items below.

Context

ADR-0003 fixed the retrieval shape (dense_nomic + dense_openai + sparse prefetch → RRF fusion → late-interaction rerank) but deliberately left two things open: which model powers late_interaction, and the sparse-side analyzer detail behind bm25-fa-norm-stop (ADR-0001). This ADR resolves those, driven by the corpus being Farsi (Persian) — a morphologically rich, low-resource language for most public embedding/rerank benchmarks and for Qdrant's own hosted tooling.

Two things specific to Farsi drove this investigation rather than picking a generic default:

• Qdrant's hosted Qdrant/bm25 FastEmbed model's documented supported- language list does not include Farsi (fa) stemming — using it as-is would silently apply no stemming rule, or the wrong one, for this corpus.
• Farsi's morphology (verb conjugation, ezafe constructions, high-frequency function words) makes pure BM25/keyword signals noisier than in English, which raised the bar on how load-bearing the rerank stage needs to be — treated here as close to mandatory for quality, not optional.

Decision

1. Fusion: RRF by default, weighted RRF as a fallback

Confirms ADR-0003's fusion stage combines all three prefetch results (dense_nomic, dense_openai, sparse) via RRF by default. If one signal (typically sparse, on Farsi text) is observed to dominate the fused ranking unexpectedly, weighted RRF is the fallback — not a switch to a different fusion algorithm. This refines, not replaces, ADR-0003's fusion decision.

2. Late-interaction reranker: jina-colbert-v2

Two multilingual late-interaction (ColBERT-style) options were evaluated against the requirement of confirmed Farsi support:

Model	Farsi support	License	Local hosting
jina-colbert-v2	Confirmed — fa explicitly listed among 89 supported languages	Conflicting: HF repo metadata says cc-by-4.0; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only	Possible via PyLate/RAGatouille; requires flash_attn (CUDA GPU effectively required); reported loading issues via generic transformers.AutoModel
LFM2-ColBERT-350M	Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es)	—	—
BGE-M3 (alternative, not adopted)	Strong Farsi performance in the FaMTEB benchmark	Apache-2.0 — unambiguous, commercial-friendly	CPU-runnable, no flash_attn dependency

jina-colbert-v2 is adopted as the late_interaction reranker: it has explicitly confirmed Farsi support among its 89 languages, which is the primary requirement for this corpus. This means:

• A CUDA GPU is now a required dependency for the rerank stage (flash_attn), not optional infrastructure — this changes the self-hosted Docker deployment assumption from ADR-0001/0003, which had not committed to GPU hosting.
• The commercial license is unresolved — HF repo metadata states cc-by-4.0 while Jina's own announcement states CC BY-NC-4.0 (non-commercial, with commercial use only via Jina's paid API/AWS/Azure offering). This must be confirmed directly with Jina before this project ships commercially on a self-hosted jina-colbert-v2 model; if Jina confirms the non-commercial reading, self-hosting it commercially is not an option and the fallback is Jina's paid hosted API or BGE-M3 as a substitute reranker (same late_interaction vector shape, no schema change needed either way).
• Loading it outside PyLate/RAGatouille (e.g. generic transformers.AutoModel) has reported issues — plan to load it through one of those two libraries, not a raw transformers call.

late_interaction continues to use hnsw_config: m=0 (per ADR-0001, now made explicit: HNSW indexing is disabled for this vector because it is used only for reranking already-fetched candidates via MAX_SIM, never for independent ANN search — the recommended pattern for rerank-only multivectors) plus on-disk storage.

3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model

Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi normalization/stopword/stemming computed outside Qdrant Cloud Inference) is used instead of Qdrant's hosted Qdrant/bm25 FastEmbed model, specifically because that hosted model's documented language list omits Farsi. The sparse vector is uploaded with modifier="idf", the standard BM25 sparse vector configuration.

Four analyzer variants were benchmarked, all sharing identical BM25 scoring parameters (k=1.2, b=0.75) so any accuracy difference is attributable entirely to the analyzer stage, not the ranking formula:

Analyzer	Description
bm25-raw	no normalization
bm25-fa-norm	Farsi normalization only
bm25-fa-norm-stop	normalization + stopword removal — current best performer
bm25-fa-norm-stem	normalization + stemming

bm25-fa-norm-stop is confirmed as the sparse analyzer named in ADR-0001, consistent with Farsi's high density of function words (ezafe particles, prepositions, common verbs) adding TF/IDF noise if left in.

4. BM25 parameters: keep k=1.2, b=0.75; tune analyzer, not formula

These are standard, well-validated defaults (Trotman, Puurula & Burgess, 2014) and are not the source of the observed analyzer-variant accuracy differences — so formula tuning is deprioritized in favor of the analyzer comparison and b sweep in the follow-ups below.

Consequences

Positive

• Resolves ADR-0003's two deferred decisions (reranker model, sparse analyzer detail) with a corpus-specific rationale instead of a generic default.
• jina-colbert-v2 is the only evaluated option with explicitly confirmed Farsi support, directly matching this project's primary requirement.
• Isolating BM25 formula parameters from analyzer choice gives a clean, defensible experimental basis for the bm25-fa-norm-stop selection.

Negative

• Introduces a hard GPU dependency (flash_attn/CUDA) for the rerank stage that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU capacity, not just Docker on commodity hardware.
• Commercial license status is unresolved; shipping this commercially on a self-hosted jina-colbert-v2 model without confirming licensing with Jina is a legal risk, not just a technical one.
• Loading path is constrained to PyLate/RAGatouille due to reported transformers.AutoModel issues — an extra library dependency and less flexibility than a standard transformers load would give.
• bm25-fa-norm-stop is provisional until compared directly against bm25-fa-norm-stem — Persian stemming can over-collapse distinct words (irregular verb conjugation, Arabic-loanword plurals), so the current "best performer" result could shift.

Alternatives Considered

• BGE-M3 as the reranker: not adopted — Apache-2.0 license and CPU-runnability are attractive and it remains the fallback if jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2 was prioritized for its explicit Farsi support.
• LFM2-ColBERT-350M: rejected outright — no Farsi support among its 8 supported languages.
• Qdrant's hosted Qdrant/bm25 FastEmbed model: rejected — its documented language support does not include Farsi stemming; using it would risk silently wrong or absent stemming for this corpus.
• Treating "RRF vs. rerank" as either/or: rejected — RRF alone can't resolve disagreement between two distinct dense embedding spaces plus a noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank pipeline from ADR-0003 is kept, not replaced with either component alone.

Follow-up / Open Items

1. Confirm jina-colbert-v2's commercial license status directly with Jina before commercial deployment; fall back to Jina's paid hosted API or BGE-M3 if the non-commercial reading is confirmed.
2. Provision GPU capacity for self-hosted jina-colbert-v2 (flash_attn requires CUDA) as part of the deployment plan, not an afterthought.
3. Run an ablation: single dense model + sparse + rerank vs. the current dual-dense-model + sparse + rerank setup, on real Farsi queries, to justify (or drop) the second dense vector (dense_openai).
4. Compare bm25-fa-norm-stop vs. bm25-fa-norm-stem in isolation to determine whether gains come from stopword removal, stemming, or both.
5. Sweep BM25 b (e.g. 0.5–0.9) for the winning analyzer, since document length varies significantly across the corpus (short chat messages vs. long articles) and 0.75 is a generic default, not corpus-tuned.