Research/embedding_models/benchmarks/README.md

last_updated, tags, source

last_updated	tags	source
2026-07-20	embedding benchmarks farsi persian miracl mmarco famte mteb	import-knowledge

Persian Embedding Benchmarks — What Exists and What Doesn't

Active Benchmarks

Benchmark	Has Farsi?	Task Type	URL	Status
MIRACL	✅ Yes (1 of 18 languages)	Passage Retrieval (nDCG@10)	HF: castorini/mirACL	Active. Primary Persian retrieval benchmark.
MTEB	✅ Via MIRACL subtask	Embedding benchmark suite	HF Spaces: mteb/leaderboard	Integrates MIRACL-Farsi. Per-language breakdowns available.
FaMTE	✅ Yes	Multi-task NLU incl. STS	GitHub: pnlp-co/FaMTE	Persian-only. Repo may be relocated (404 during research). Includes STS relevant to embedding quality.

Benchmarks That Do NOT Include Farsi

Benchmark	Languages	Farsi?
mMARCO	14 languages (EN, ZH, FR, DE, etc.)	❌ No
Mr. TyDi	11 languages (AR, EN, FI, DE, etc.)	❌ No

Key Takeaway

MIRACL (Farsi) is the primary publicly available Persian retrieval benchmark. Most embedding models do not publish isolated Farsi scores in their model cards — you need to run your own eval.

Loading MIRACL-Farsi

from datasets import load_dataset

# Load Farsi subset
dataset = load_dataset("castorini/mirACL", "fas", split="validation")

print(f"Number of queries: {len(dataset)}")
print(f"Sample: {dataset[0]}")

Using MIRACL-Farsi for Evaluation

from datasets import load_dataset
import numpy as np
from sentence_transformers import SentenceTransformer

# Load data
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
model = SentenceTransformer("BAAI/bge-m3")

# Embed queries and documents
queries = [item["query"] for item in dataset]
documents = [item["positive_document"] for item in dataset]

query_embeddings = model.encode(queries, show_progress_bar=True)
doc_embeddings = model.encode(documents, show_progress_bar=True)

# Compute cosine similarity and MRR
def mrr(query_embs, doc_embs, k=10):
    scores = query_embs @ doc_embs.T
    rr = []
    for i, row in enumerate(scores):
        ranked = np.argsort(row)[::-1][:k]
        for rank, idx in enumerate(ranked):
            if idx == i:  # assuming 1:1 query-doc mapping
                rr.append(1 / (rank + 1))
                break
        else:
            rr.append(0)
    return np.mean(rr)

print(f"MRR@10: {mrr(query_embeddings, doc_embeddings):.4f}")

Building Your Own Insurance Eval Set

Since Farsi-specific benchmarks are sparse, build your own:

1. Start with your 100-row Q&A CSV
2. Map each question to the relevant chunk IDs in your corpus
3. For each question, embed with candidate model, rank all chunks by cosine similarity
4. Measure Recall@5 (is the right chunk in top-5?) and MRR (rank of the right chunk)

This gives you the most reliable signal for YOUR specific use case.

TODO

• Download MIRACL-Farsi validation set
• Run baseline eval with OpenAI text-embedding-3-large
• Run eval with BGE-M3
• Run eval with gte-multilingual-base
• Map 100-row Q&A CSV to chunk IDs for insurance-domain eval