docs(research): add farsi embedding model research knowledge base

Body:
        Why:
        - Insurance RAG pipeline needs Farsi-optimized embeddings
        - Current local baseline (Nomic Embed) is English-only

        Changes:
        - 8 cloud model profiles with curl + python test commands
        - 5 local model profiles with VRAM, serving platform notes
        - Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
        - Decision report with ranked options and recommendations
        - Full comparison table (15 models × 10 columns)

        Impact:
        - Knowledge base for embedding model selection
        - Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
        - Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
2026-07-20 15:00:11 +03:30
parent 72d8a3ebe4
commit 6c9cebaaa0
13 changed files with 1296 additions and 0 deletions

View File

@@ -0,0 +1,92 @@
---
last_updated: 2026-07-20
tags: [embedding, benchmarks, farsi, persian, miracl, mmarco, famte, mteb]
source: import-knowledge
---
# Persian Embedding Benchmarks — What Exists and What Doesn't
## Active Benchmarks
| Benchmark | Has Farsi? | Task Type | URL | Status |
|-----------|-----------|-----------|-----|--------|
| **MIRACL** | ✅ Yes (1 of 18 languages) | Passage Retrieval (nDCG@10) | [HF: castorini/mirACL](https://huggingface.co/datasets/castorini/mirACL) | Active. Primary Persian retrieval benchmark. |
| **MTEB** | ✅ Via MIRACL subtask | Embedding benchmark suite | [HF Spaces: mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard) | Integrates MIRACL-Farsi. Per-language breakdowns available. |
| **FaMTE** | ✅ Yes | Multi-task NLU incl. STS | [GitHub: pnlp-co/FaMTE](https://github.com/pnlp-co/FaMTE) | Persian-only. Repo may be relocated (404 during research). Includes STS relevant to embedding quality. |
## Benchmarks That Do NOT Include Farsi
| Benchmark | Languages | Farsi? |
|-----------|-----------|--------|
| **mMARCO** | 14 languages (EN, ZH, FR, DE, etc.) | ❌ No |
| **Mr. TyDi** | 11 languages (AR, EN, FI, DE, etc.) | ❌ No |
## Key Takeaway
**MIRACL (Farsi) is the primary publicly available Persian retrieval benchmark.** Most embedding models do not publish isolated Farsi scores in their model cards — you need to run your own eval.
## Loading MIRACL-Farsi
```python
from datasets import load_dataset
# Load Farsi subset
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
print(f"Number of queries: {len(dataset)}")
print(f"Sample: {dataset[0]}")
```
## Using MIRACL-Farsi for Evaluation
```python
from datasets import load_dataset
import numpy as np
from sentence_transformers import SentenceTransformer
# Load data
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
model = SentenceTransformer("BAAI/bge-m3")
# Embed queries and documents
queries = [item["query"] for item in dataset]
documents = [item["positive_document"] for item in dataset]
query_embeddings = model.encode(queries, show_progress_bar=True)
doc_embeddings = model.encode(documents, show_progress_bar=True)
# Compute cosine similarity and MRR
def mrr(query_embs, doc_embs, k=10):
scores = query_embs @ doc_embs.T
rr = []
for i, row in enumerate(scores):
ranked = np.argsort(row)[::-1][:k]
for rank, idx in enumerate(ranked):
if idx == i: # assuming 1:1 query-doc mapping
rr.append(1 / (rank + 1))
break
else:
rr.append(0)
return np.mean(rr)
print(f"MRR@10: {mrr(query_embeddings, doc_embeddings):.4f}")
```
## Building Your Own Insurance Eval Set
Since Farsi-specific benchmarks are sparse, build your own:
1. Start with your 100-row Q&A CSV
2. Map each question to the relevant chunk IDs in your corpus
3. For each question, embed with candidate model, rank all chunks by cosine similarity
4. Measure Recall@5 (is the right chunk in top-5?) and MRR (rank of the right chunk)
This gives you the most reliable signal for YOUR specific use case.
## TODO
- [ ] Download MIRACL-Farsi validation set
- [ ] Run baseline eval with OpenAI text-embedding-3-large
- [ ] Run eval with BGE-M3
- [ ] Run eval with gte-multilingual-base
- [ ] Map 100-row Q&A CSV to chunk IDs for insurance-domain eval