docs(research): add farsi embedding model research knowledge base
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
73
embedding_models/mE5-large.md
Normal file
73
embedding_models/mE5-large.md
Normal file
@@ -0,0 +1,73 @@
|
||||
---
|
||||
last_updated: 2026-07-20
|
||||
tags: [embedding, multilingual-e5, local, farsi, insurance, rag]
|
||||
source: import-knowledge
|
||||
---
|
||||
|
||||
# intfloat/multilingual-e5-large
|
||||
|
||||
## Overview
|
||||
|
||||
- **Type**: Local
|
||||
- **Provider**: intfloat
|
||||
- **Parameters**: ~560M (24-layer, xlm-roberta-large init)
|
||||
- **VRAM (fp16)**: ~1.1 GB
|
||||
- **Embedding dimensions**: 1024
|
||||
- **Max input tokens**: 512
|
||||
- **Language coverage**: 100 languages including Farsi
|
||||
- **Serving platform**: sentence-transformers, TEI, Ollama (available)
|
||||
- **License**: MIT
|
||||
- **Model card**: https://huggingface.co/intfloat/multilingual-e5-large
|
||||
|
||||
## Pros
|
||||
|
||||
- Most widely used multilingual embedding model
|
||||
- Strong community support and documentation
|
||||
- 100 languages including Farsi
|
||||
- MIT license
|
||||
- Available on Ollama
|
||||
|
||||
## Cons
|
||||
|
||||
- **512 token max context** — requires aggressive chunking for insurance documents
|
||||
- No ColBERT/sparse support
|
||||
- Requires `"query: "` / `"passage: "` prefixes for optimal retrieval
|
||||
- Benchmarked on Mr. TyDi but Farsi is NOT one of the 11 Mr. TyDi languages
|
||||
|
||||
## ⚠️ 512 Token Limit
|
||||
|
||||
Insurance policy clauses and FAQ answers can easily exceed 512 tokens. You will need to chunk aggressively. Consider BGE-M3 (8K context) or gte-multilingual-base (8K context) instead.
|
||||
|
||||
## API Test — Ollama
|
||||
|
||||
```bash
|
||||
ollama pull multilingual-e5-large
|
||||
|
||||
curl -s http://localhost:11434/api/embeddings -d '{
|
||||
"model": "multilingual-e5-large",
|
||||
"prompt": "query: بیمه نامه شخص ثالث چیست؟"
|
||||
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'dims={len(d["embedding"])}')"
|
||||
```
|
||||
|
||||
**⚠️ Important**: Ollama may not auto-add the `"query: "` prefix. You must add it manually in the prompt for optimal retrieval quality.
|
||||
|
||||
## API Test — Python
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
model = SentenceTransformer("intfloat/multilingual-e5-large")
|
||||
|
||||
# ⚠️ Must add "query: " prefix for queries, "passage: " for documents
|
||||
query_embedding = model.encode("query: بیمه نامه شخص ثالث چیست؟")
|
||||
doc_embedding = model.encode("passage: بیمه نامه شخص ثالث شامل پوشش خسارات مالی و جانی است.")
|
||||
|
||||
print(f"Query dims: {len(query_embedding)}, Doc dims: {len(doc_embedding)}")
|
||||
# Expected: Query dims: 1024, Doc dims: 1024
|
||||
```
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Benchmark on MIRACL-Farsi subset
|
||||
- [ ] Test with aggressive chunking (max 500 tokens)
|
||||
- [ ] Compare with BGE-M3 on Farsi retrieval quality
|
||||
Reference in New Issue
Block a user