---
last_updated: 2026-07-20
tags: [embedding, benchmarks, farsi, persian, miracl, mmarco, famte, mteb]
source: import-knowledge
---

# Persian Embedding Benchmarks — What Exists and What Doesn't

## Active Benchmarks

| Benchmark | Has Farsi? | Task Type | URL | Status |
|-----------|-----------|-----------|-----|--------|
| **MIRACL** | ✅ Yes (1 of 18 languages) | Passage Retrieval (nDCG@10) | [HF: castorini/mirACL](https://huggingface.co/datasets/castorini/mirACL) | Active. Primary Persian retrieval benchmark. |
| **MTEB** | ✅ Via MIRACL subtask | Embedding benchmark suite | [HF Spaces: mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard) | Integrates MIRACL-Farsi. Per-language breakdowns available. |
| **FaMTE** | ✅ Yes | Multi-task NLU incl. STS | [GitHub: pnlp-co/FaMTE](https://github.com/pnlp-co/FaMTE) | Persian-only. Repo may be relocated (404 during research). Includes STS relevant to embedding quality. |

## Benchmarks That Do NOT Include Farsi

| Benchmark | Languages | Farsi? |
|-----------|-----------|--------|
| **mMARCO** | 14 languages (EN, ZH, FR, DE, etc.) | ❌ No |
| **Mr. TyDi** | 11 languages (AR, EN, FI, DE, etc.) | ❌ No |

## Key Takeaway

**MIRACL (Farsi) is the primary publicly available Persian retrieval benchmark.** Most embedding models do not publish isolated Farsi scores in their model cards — you need to run your own eval.

## Loading MIRACL-Farsi

```python
from datasets import load_dataset

# Load Farsi subset
dataset = load_dataset("castorini/mirACL", "fas", split="validation")

print(f"Number of queries: {len(dataset)}")
print(f"Sample: {dataset[0]}")
```

## Using MIRACL-Farsi for Evaluation

```python
from datasets import load_dataset
import numpy as np
from sentence_transformers import SentenceTransformer

# Load data
dataset = load_dataset("castorini/mirACL", "fas", split="validation")
model = SentenceTransformer("BAAI/bge-m3")

# Embed queries and documents
queries = [item["query"] for item in dataset]
documents = [item["positive_document"] for item in dataset]

query_embeddings = model.encode(queries, show_progress_bar=True)
doc_embeddings = model.encode(documents, show_progress_bar=True)

# Compute cosine similarity and MRR
def mrr(query_embs, doc_embs, k=10):
    scores = query_embs @ doc_embs.T
    rr = []
    for i, row in enumerate(scores):
        ranked = np.argsort(row)[::-1][:k]
        for rank, idx in enumerate(ranked):
            if idx == i:  # assuming 1:1 query-doc mapping
                rr.append(1 / (rank + 1))
                break
        else:
            rr.append(0)
    return np.mean(rr)

print(f"MRR@10: {mrr(query_embeddings, doc_embeddings):.4f}")
```

## Building Your Own Insurance Eval Set

Since Farsi-specific benchmarks are sparse, build your own:

1. Start with your 100-row Q&A CSV
2. Map each question to the relevant chunk IDs in your corpus
3. For each question, embed with candidate model, rank all chunks by cosine similarity
4. Measure Recall@5 (is the right chunk in top-5?) and MRR (rank of the right chunk)

This gives you the most reliable signal for YOUR specific use case.

## TODO

- [ ] Download MIRACL-Farsi validation set
- [ ] Run baseline eval with OpenAI text-embedding-3-large
- [ ] Run eval with BGE-M3
- [ ] Run eval with gte-multilingual-base
- [ ] Map 100-row Q&A CSV to chunk IDs for insurance-domain eval