initial commit
This commit is contained in:
166
docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md
Normal file
166
docs/adr/0005-reranking-model-and-sparse-analyzer-selection.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# 0005. Reranking model and sparse (BM25) analyzer selection for the Farsi corpus
|
||||
|
||||
## Status
|
||||
|
||||
Proposed — the fusion/rerank *shape* and reranker model are decided; the
|
||||
final BM25 analyzer and the commercial license status of the reranker are
|
||||
still open per the follow-up items below.
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0003 fixed the retrieval *shape* (dense_nomic + dense_openai + sparse
|
||||
prefetch → RRF fusion → late-interaction rerank) but deliberately left two
|
||||
things open: which model powers `late_interaction`, and the sparse-side
|
||||
analyzer detail behind `bm25-fa-norm-stop` (ADR-0001). This ADR resolves
|
||||
those, driven by the corpus being **Farsi (Persian)** — a morphologically
|
||||
rich, low-resource language for most public embedding/rerank benchmarks and
|
||||
for Qdrant's own hosted tooling.
|
||||
|
||||
Two things specific to Farsi drove this investigation rather than picking a
|
||||
generic default:
|
||||
|
||||
- Qdrant's hosted `Qdrant/bm25` FastEmbed model's documented supported-
|
||||
language list does not include Farsi (`fa`) stemming — using it as-is
|
||||
would silently apply no stemming rule, or the wrong one, for this corpus.
|
||||
- Farsi's morphology (verb conjugation, ezafe constructions, high-frequency
|
||||
function words) makes pure BM25/keyword signals noisier than in English,
|
||||
which raised the bar on how load-bearing the rerank stage needs to be —
|
||||
treated here as close to mandatory for quality, not optional.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Fusion: RRF by default, weighted RRF as a fallback
|
||||
|
||||
Confirms ADR-0003's fusion stage combines all three prefetch results
|
||||
(`dense_nomic`, `dense_openai`, `sparse`) via **RRF** by default. If one
|
||||
signal (typically sparse, on Farsi text) is observed to dominate the fused
|
||||
ranking unexpectedly, **weighted RRF** is the fallback — not a switch to a
|
||||
different fusion algorithm. This refines, not replaces, ADR-0003's fusion
|
||||
decision.
|
||||
|
||||
### 2. Late-interaction reranker: jina-colbert-v2
|
||||
|
||||
Two multilingual late-interaction (ColBERT-style) options were evaluated
|
||||
against the requirement of confirmed Farsi support:
|
||||
|
||||
| Model | Farsi support | License | Local hosting |
|
||||
|---|---|---|---|
|
||||
| **jina-colbert-v2** | Confirmed — `fa` explicitly listed among 89 supported languages | Conflicting: HF repo metadata says `cc-by-4.0`; Jina's own announcement states CC BY-NC-4.0 (non-commercial), commercial use via paid API/AWS/Azure only | Possible via PyLate/RAGatouille; requires `flash_attn` (CUDA GPU effectively required); reported loading issues via generic `transformers.AutoModel` |
|
||||
| **LFM2-ColBERT-350M** | Not supported (8 languages: en, ar, zh, fr, de, ja, ko, es) | — | — |
|
||||
| BGE-M3 (alternative, not adopted) | Strong Farsi performance in the FaMTEB benchmark | Apache-2.0 — unambiguous, commercial-friendly | CPU-runnable, no `flash_attn` dependency |
|
||||
|
||||
**jina-colbert-v2 is adopted** as the `late_interaction` reranker: it has
|
||||
explicitly confirmed Farsi support among its 89 languages, which is the
|
||||
primary requirement for this corpus. This means:
|
||||
|
||||
- A **CUDA GPU is now a required dependency** for the rerank stage
|
||||
(`flash_attn`), not optional infrastructure — this changes the self-hosted
|
||||
Docker deployment assumption from ADR-0001/0003, which had not committed
|
||||
to GPU hosting.
|
||||
- The **commercial license is unresolved** — HF repo metadata states
|
||||
`cc-by-4.0` while Jina's own announcement states CC BY-NC-4.0
|
||||
(non-commercial, with commercial use only via Jina's paid API/AWS/Azure
|
||||
offering). This must be confirmed directly with Jina **before** this
|
||||
project ships commercially on a self-hosted jina-colbert-v2 model; if
|
||||
Jina confirms the non-commercial reading, self-hosting it commercially is
|
||||
not an option and the fallback is Jina's paid hosted API or BGE-M3 as a
|
||||
substitute reranker (same `late_interaction` vector shape, no schema
|
||||
change needed either way).
|
||||
- Loading it outside PyLate/RAGatouille (e.g. generic
|
||||
`transformers.AutoModel`) has reported issues — plan to load it through
|
||||
one of those two libraries, not a raw `transformers` call.
|
||||
|
||||
`late_interaction` continues to use `hnsw_config: m=0` (per ADR-0001,
|
||||
now made explicit: HNSW indexing is disabled for this vector because it is
|
||||
used only for reranking already-fetched candidates via MAX_SIM, never for
|
||||
independent ANN search — the recommended pattern for rerank-only
|
||||
multivectors) plus on-disk storage.
|
||||
|
||||
### 3. Sparse retrieval: custom BM25 pipeline, not Qdrant's hosted model
|
||||
|
||||
Confirms ADR-0001's choice: the project's own BM25 pipeline (Farsi
|
||||
normalization/stopword/stemming computed outside Qdrant Cloud Inference) is
|
||||
used instead of Qdrant's hosted `Qdrant/bm25` FastEmbed model, specifically
|
||||
because that hosted model's documented language list omits Farsi. The
|
||||
sparse vector is uploaded with `modifier="idf"`, the standard BM25 sparse
|
||||
vector configuration.
|
||||
|
||||
Four analyzer variants were benchmarked, all sharing identical BM25 scoring
|
||||
parameters (`k=1.2`, `b=0.75`) so any accuracy difference is attributable
|
||||
entirely to the analyzer stage, not the ranking formula:
|
||||
|
||||
| Analyzer | Description |
|
||||
|---|---|
|
||||
| `bm25-raw` | no normalization |
|
||||
| `bm25-fa-norm` | Farsi normalization only |
|
||||
| `bm25-fa-norm-stop` | normalization + stopword removal — **current best performer** |
|
||||
| `bm25-fa-norm-stem` | normalization + stemming |
|
||||
|
||||
`bm25-fa-norm-stop` is confirmed as the sparse analyzer named in ADR-0001,
|
||||
consistent with Farsi's high density of function words (ezafe particles,
|
||||
prepositions, common verbs) adding TF/IDF noise if left in.
|
||||
|
||||
### 4. BM25 parameters: keep `k=1.2`, `b=0.75`; tune analyzer, not formula
|
||||
|
||||
These are standard, well-validated defaults (Trotman, Puurula & Burgess,
|
||||
2014) and are not the source of the observed analyzer-variant accuracy
|
||||
differences — so formula tuning is deprioritized in favor of the analyzer
|
||||
comparison and `b` sweep in the follow-ups below.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Resolves ADR-0003's two deferred decisions (reranker model, sparse
|
||||
analyzer detail) with a corpus-specific rationale instead of a generic
|
||||
default.
|
||||
- jina-colbert-v2 is the only evaluated option with explicitly confirmed
|
||||
Farsi support, directly matching this project's primary requirement.
|
||||
- Isolating BM25 formula parameters from analyzer choice gives a clean,
|
||||
defensible experimental basis for the `bm25-fa-norm-stop` selection.
|
||||
|
||||
### Negative
|
||||
- Introduces a hard GPU dependency (`flash_attn`/CUDA) for the rerank stage
|
||||
that ADR-0001/0003 hadn't assumed — self-hosted deployment now needs GPU
|
||||
capacity, not just Docker on commodity hardware.
|
||||
- Commercial license status is unresolved; shipping this commercially on a
|
||||
self-hosted jina-colbert-v2 model without confirming licensing with Jina
|
||||
is a legal risk, not just a technical one.
|
||||
- Loading path is constrained to PyLate/RAGatouille due to reported
|
||||
`transformers.AutoModel` issues — an extra library dependency and less
|
||||
flexibility than a standard `transformers` load would give.
|
||||
- `bm25-fa-norm-stop` is provisional until compared directly against
|
||||
`bm25-fa-norm-stem` — Persian stemming can over-collapse distinct words
|
||||
(irregular verb conjugation, Arabic-loanword plurals), so the current
|
||||
"best performer" result could shift.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- **BGE-M3 as the reranker**: not adopted — Apache-2.0 license and
|
||||
CPU-runnability are attractive and it remains the fallback if
|
||||
jina-colbert-v2's license is confirmed non-commercial, but jina-colbert-v2
|
||||
was prioritized for its explicit Farsi support.
|
||||
- **LFM2-ColBERT-350M**: rejected outright — no Farsi support among its 8
|
||||
supported languages.
|
||||
- **Qdrant's hosted `Qdrant/bm25` FastEmbed model**: rejected — its
|
||||
documented language support does not include Farsi stemming; using it
|
||||
would risk silently wrong or absent stemming for this corpus.
|
||||
- **Treating "RRF vs. rerank" as either/or**: rejected — RRF alone can't
|
||||
resolve disagreement between two distinct dense embedding spaces plus a
|
||||
noisy Farsi BM25 signal, so the two-stage prefetch-fusion-then-rerank
|
||||
pipeline from ADR-0003 is kept, not replaced with either component alone.
|
||||
|
||||
## Follow-up / Open Items
|
||||
|
||||
1. Confirm jina-colbert-v2's commercial license status directly with Jina
|
||||
before commercial deployment; fall back to Jina's paid hosted API or
|
||||
BGE-M3 if the non-commercial reading is confirmed.
|
||||
2. Provision GPU capacity for self-hosted jina-colbert-v2 (`flash_attn`
|
||||
requires CUDA) as part of the deployment plan, not an afterthought.
|
||||
3. Run an ablation: single dense model + sparse + rerank vs. the current
|
||||
dual-dense-model + sparse + rerank setup, on real Farsi queries, to
|
||||
justify (or drop) the second dense vector (`dense_openai`).
|
||||
4. Compare `bm25-fa-norm-stop` vs. `bm25-fa-norm-stem` in isolation to
|
||||
determine whether gains come from stopword removal, stemming, or both.
|
||||
5. Sweep BM25 `b` (e.g. 0.5–0.9) for the winning analyzer, since document
|
||||
length varies significantly across the corpus (short chat messages vs.
|
||||
long articles) and `0.75` is a generic default, not corpus-tuned.
|
||||
Reference in New Issue
Block a user