chatbot_v3/docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md

# 0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus

## Status

Proposed — the fusion/rerank *shape* and reranker model are decided; the
final BM25 analyzer and the commercial license status of the reranker are
still open per the follow-up items below.

## Context

ADR-0003 fixed the retrieval *shape* (dense_nomic + dense_openai + sparse
prefetch → RRF fusion → late-interaction rerank) but deliberately left two
things open: which model powers `late_interaction`, and the sparse-side
analyzer detail behind `bm25-fa-norm-stop` (ADR-0001). This ADR resolves
those, driven by the corpus being **Farsi (Persian)** — a morphologically
rich, low-resource language for most public embedding/rerank benchmarks and
for Qdrant's own hosted tooling.

Two things specific to Farsi drove this investigation rather than picking a
generic default:

- Qdrant's hosted `Qdrant/bm25` FastEmbed model's documented supported-
  language list does not include Farsi (`fa`) stemming — using it as-is
  would silently apply no stemming rule, or the wrong one, for this corpus.
- Farsi's morphology (verb conjugation, ezafe constructions, high-frequency
  function words) makes pure BM25/keyword signals noisier than in English,
  which raised the bar on how load-bearing the rerank stage needs to be —
  treated here as close to mandatory for quality, not optional.

## Decision

### 1. Fusion: RRF by default, weighted RRF as a fallback

Confirms ADR-0003's fusion stage combines all three prefetch results
(`dense_nomic`, `dense_openai`, `sparse`) via **RRF** by default. If one
signal (typically sparse, on Farsi text) is observed to dominate the fused
ranking unexpectedly, **weighted RRF** is the fallback — not a switch to a
different fusion algorithm. This refines, not replaces, ADR-0003's fusion
decision.

### 2. Late-interaction reranker: jina-colbert-v2

Two multilingual late-interaction (ColBERT-style) options were evaluated
against the requirement of confirmed Farsi support:

| Model | Farsi support | License | Local hosting |
|---|---|---|---|
| **jina-colbert-v2** | Confirmed — `fa` explicitly listed among 89 supported languages | Conflicting: HF repo metadata says `cc-by-4.0`; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only | Possible via PyLate/RAGatouille; requires `flash_attn` (CUDA GPU effectively required); reported loading issues via generic `transformers.AutoModel` |
| **LFM2-ColBERT-350M** | Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es) | — | — |
| BGE-M3 (alternative, not adopted) | Strong Farsi performance in the FaMTEB benchmark | Apache-2.0 — unambiguous, commercial-friendly | CPU-runnable, no `flash_attn` dependency |

**jina-colbert-v2 is adopted** as the `late_interaction` reranker: it has
explicitly confirmed Farsi support among its 89 languages, which is the
primary requirement for this corpus. This means:

- A **CUDA GPU is now a required dependency** for the rerank stage
  (`flash_attn`), not optional infrastructure — this changes the self-hosted
  Docker deployment assumption from ADR-0001/0003, which had not committed
  to GPU hosting.
- The **commercial license is unresolved** — HF repo metadata states
  `cc-by-4.0` while Jina's own announcement states CC BY-NC-4.0
  (non-commercial, with commercial use only via Jina's paid API/AWS/Azure
  offering). This must be confirmed directly with Jina **before** this
  project ships commercially on a self-hosted jina-colbert-v2 model; if
  Jina confirms the non-commercial reading, self-hosting it commercially is
  not an option and the fallback is Jina's paid hosted API or BGE-M3 as a
  substitute reranker (same `late_interaction` vector shape, no schema
  change needed either way).
- Loading it outside PyLate/RAGatouille (e.g. generic
  `transformers.AutoModel`) has reported issues — plan to load it through
  one of those two libraries, not a raw `transformers` call.

`late_interaction` continues to use `hnsw_config: m=0` (per ADR-0001,
now made explicit: HNSW indexing is disabled for this vector because it is
used only for reranking already-fetched candidates via MAX_SIM, never for
independent ANN search — the recommended pattern for rerank-only
multivectors) plus on-disk storage.

### 3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model

Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi
normalization/stopword/stemming computed outside Qdrant Cloud Inference) is
used instead of Qdrant's hosted `Qdrant/bm25` FastEmbed model, specifically
because that hosted model's documented language list omits Farsi. The
sparse vector is uploaded with `modifier="idf"`, the standard BM25 sparse
vector configuration.

Four analyzer variants were benchmarked, all sharing identical BM25 scoring
parameters (`k=1.2`, `b=0.75`) so any accuracy difference is attributable
entirely to the analyzer stage, not the ranking formula:

| Analyzer | Description |
|---|---|
| `bm25-raw` | no normalization |
| `bm25-fa-norm` | Farsi normalization only |
| `bm25-fa-norm-stop` | normalization + stopword removal — **current best performer** |
| `bm25-fa-norm-stem` | normalization + stemming |

`bm25-fa-norm-stop` is confirmed as the sparse analyzer named in ADR-0001,
consistent with Farsi's high density of function words (ezafe particles,
prepositions, common verbs) adding TF/IDF noise if left in.

### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula

These are standard, well-validated defaults (Trotman, Puurula & Burgess,
2014) and are not the source of the observed analyzer-variant accuracy
differences — so formula tuning is deprioritized in favor of the analyzer
comparison and `b` sweep in the follow-ups below.

## Consequences

### Positive
- Resolves ADR-0003's two deferred decisions (reranker model, sparse
  analyzer detail) with a corpus-specific rationale instead of a generic
  default.
- jina-colbert-v2 is the only evaluated option with explicitly confirmed
  Farsi support, directly matching this project's primary requirement.
- Isolating BM25 formula parameters from analyzer choice gives a clean,
  defensible experimental basis for the `bm25-fa-norm-stop` selection.

### Negative
- Introduces a hard GPU dependency (`flash_attn`/CUDA) for the rerank stage
  that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU
  capacity, not just Docker on commodity hardware.
- Commercial license status is unresolved; shipping this commercially on a
  self-hosted jina-colbert-v2 model without confirming licensing with Jina
  is a legal risk, not just a technical one.
- Loading path is constrained to PyLate/RAGatouille due to reported
  `transformers.AutoModel` issues — an extra library dependency and less
  flexibility than a standard `transformers` load would give.
- `bm25-fa-norm-stop` is provisional until compared directly against
  `bm25-fa-norm-stem` — Persian stemming can over-collapse distinct words
  (irregular verb conjugation, Arabic-loanword plurals), so the current
  "best performer" result could shift.

## Alternatives Considered

- **BGE-M3 as the reranker**: not adopted — Apache-2.0 license and
  CPU-runnability are attractive and it remains the fallback if
  jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2
  was prioritized for its explicit Farsi support.
- **LFM2-ColBERT-350M**: rejected outright — no Farsi support among its 8
  supported languages.
- **Qdrant's hosted `Qdrant/bm25` FastEmbed model**: rejected — its
  documented language support does not include Farsi stemming; using it
  would risk silently wrong or absent stemming for this corpus.
- **Treating "RRF vs. rerank" as either/or**: rejected — RRF alone can't
  resolve disagreement between two distinct dense embedding spaces plus a
  noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank
  pipeline from ADR-0003 is kept, not replaced with either component alone.

## Follow-up / Open Items

1. Confirm jina-colbert-v2's commercial license status directly with Jina
   before commercial deployment; fall back to Jina's paid hosted API or
   BGE-M3 if the non-commercial reading is confirmed.
2. Provision GPU capacity for self-hosted jina-colbert-v2 (`flash_attn`
   requires CUDA) as part of the deployment plan, not an afterthought.
3. Run an ablation: single dense model + sparse + rerank vs. the current
   dual-dense-model + sparse + rerank setup, on real Farsi queries, to
   justify (or drop) the second dense vector (`dense_openai`).
4. Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation to
   determine whether gains come from stopword removal, stemming, or both.
5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document
   length varies significantly across the corpus (short chat messages vs.
   long articles) and `0.75` is a generic default, not corpus-tuned.