docs(research): add farsi embedding model research knowledge base
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
92
embedding_models/benchmarks/README.md
Normal file
92
embedding_models/benchmarks/README.md
Normal file
@@ -0,0 +1,92 @@
|
||||
---
|
||||
last_updated: 2026-07-20
|
||||
tags: [embedding, benchmarks, farsi, persian, miracl, mmarco, famte, mteb]
|
||||
source: import-knowledge
|
||||
---
|
||||
|
||||
# Persian Embedding Benchmarks — What Exists and What Doesn't
|
||||
|
||||
## Active Benchmarks
|
||||
|
||||
| Benchmark | Has Farsi? | Task Type | URL | Status |
|
||||
|-----------|-----------|-----------|-----|--------|
|
||||
| **MIRACL** | ✅ Yes (1 of 18 languages) | Passage Retrieval (nDCG@10) | [HF: castorini/mirACL](https://huggingface.co/datasets/castorini/mirACL) | Active. Primary Persian retrieval benchmark. |
|
||||
| **MTEB** | ✅ Via MIRACL subtask | Embedding benchmark suite | [HF Spaces: mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard) | Integrates MIRACL-Farsi. Per-language breakdowns available. |
|
||||
| **FaMTE** | ✅ Yes | Multi-task NLU incl. STS | [GitHub: pnlp-co/FaMTE](https://github.com/pnlp-co/FaMTE) | Persian-only. Repo may be relocated (404 during research). Includes STS relevant to embedding quality. |
|
||||
|
||||
## Benchmarks That Do NOT Include Farsi
|
||||
|
||||
| Benchmark | Languages | Farsi? |
|
||||
|-----------|-----------|--------|
|
||||
| **mMARCO** | 14 languages (EN, ZH, FR, DE, etc.) | ❌ No |
|
||||
| **Mr. TyDi** | 11 languages (AR, EN, FI, DE, etc.) | ❌ No |
|
||||
|
||||
## Key Takeaway
|
||||
|
||||
**MIRACL (Farsi) is the primary publicly available Persian retrieval benchmark.** Most embedding models do not publish isolated Farsi scores in their model cards — you need to run your own eval.
|
||||
|
||||
## Loading MIRACL-Farsi
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
|
||||
# Load Farsi subset
|
||||
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
|
||||
|
||||
print(f"Number of queries: {len(dataset)}")
|
||||
print(f"Sample: {dataset[0]}")
|
||||
```
|
||||
|
||||
## Using MIRACL-Farsi for Evaluation
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
import numpy as np
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# Load data
|
||||
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
|
||||
model = SentenceTransformer("BAAI/bge-m3")
|
||||
|
||||
# Embed queries and documents
|
||||
queries = [item["query"] for item in dataset]
|
||||
documents = [item["positive_document"] for item in dataset]
|
||||
|
||||
query_embeddings = model.encode(queries, show_progress_bar=True)
|
||||
doc_embeddings = model.encode(documents, show_progress_bar=True)
|
||||
|
||||
# Compute cosine similarity and MRR
|
||||
def mrr(query_embs, doc_embs, k=10):
|
||||
scores = query_embs @ doc_embs.T
|
||||
rr = []
|
||||
for i, row in enumerate(scores):
|
||||
ranked = np.argsort(row)[::-1][:k]
|
||||
for rank, idx in enumerate(ranked):
|
||||
if idx == i: # assuming 1:1 query-doc mapping
|
||||
rr.append(1 / (rank + 1))
|
||||
break
|
||||
else:
|
||||
rr.append(0)
|
||||
return np.mean(rr)
|
||||
|
||||
print(f"MRR@10: {mrr(query_embeddings, doc_embeddings):.4f}")
|
||||
```
|
||||
|
||||
## Building Your Own Insurance Eval Set
|
||||
|
||||
Since Farsi-specific benchmarks are sparse, build your own:
|
||||
|
||||
1. Start with your 100-row Q&A CSV
|
||||
2. Map each question to the relevant chunk IDs in your corpus
|
||||
3. For each question, embed with candidate model, rank all chunks by cosine similarity
|
||||
4. Measure Recall@5 (is the right chunk in top-5?) and MRR (rank of the right chunk)
|
||||
|
||||
This gives you the most reliable signal for YOUR specific use case.
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Download MIRACL-Farsi validation set
|
||||
- [ ] Run baseline eval with OpenAI text-embedding-3-large
|
||||
- [ ] Run eval with BGE-M3
|
||||
- [ ] Run eval with gte-multilingual-base
|
||||
- [ ] Map 100-row Q&A CSV to chunk IDs for insurance-domain eval
|
||||
Reference in New Issue
Block a user