Mahdi Bazrafshan
6c9cebaaa0
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
128 lines
4.1 KiB
Markdown
128 lines
4.1 KiB
Markdown
---
|
|
last_updated: 2026-07-20
|
|
tags: [embedding, bge-m3, local, farsi, insurance, rag, triple-mode, recommended]
|
|
source: import-knowledge
|
|
---
|
|
|
|
# BAAI/bge-m3 — Top Local Recommendation ⭐
|
|
|
|
## Overview
|
|
|
|
- **Type**: Local
|
|
- **Provider**: BAAI (Beijing Academy of AI)
|
|
- **Parameters**: ~568M
|
|
- **VRAM (fp16)**: ~1.1 GB
|
|
- **Embedding dimensions**: 1024
|
|
- **Max input tokens**: 8,192
|
|
- **Language coverage**: 100+ languages including Farsi
|
|
- **ColBERT/late interaction**: YES (native multi-vector retrieval)
|
|
- **Sparse vectors**: YES (native lexical weight retrieval)
|
|
- **Serving platform**: FlagEmbedding library, sentence-transformers, TEI, or Ollama (available)
|
|
- **License**: MIT
|
|
- **Model card**: https://huggingface.co/BAAI/bge-m3
|
|
|
|
## Why It's #1
|
|
|
|
**Triple-mode retrieval** (dense + sparse + ColBERT) from a single model. This is uniquely suited to LangGraph + Qdrant hybrid search:
|
|
- **Dense**: Standard cosine similarity embeddings
|
|
- **Sparse**: BM25-like lexical retrieval without a separate sparse encoder
|
|
- **ColBERT**: Token-level late interaction for finer-grained matching
|
|
|
|
Only 1.1 GB VRAM on a 24 GB RTX 3090 — leaves room for other workloads. 8K context handles long insurance policy chunks.
|
|
|
|
## Pros
|
|
|
|
- Supports Farsi (100+ languages)
|
|
- Triple-mode: dense + sparse + ColBERT multi-vector
|
|
- Hybrid search ready — no separate sparse encoder needed
|
|
- 8,192 token context (can handle long insurance policy chunks)
|
|
- Only 1.1 GB VRAM — leaves headroom for other workloads
|
|
- MIT license — no commercial restrictions
|
|
- Available on Ollama, TEI, sentence-transformers
|
|
- Active development by BAAI (Beijing Academy of AI)
|
|
|
|
## Cons
|
|
|
|
- No published Farsi-specific retrieval benchmark scores
|
|
- Slightly less community adoption than multilingual-E5
|
|
- Triple-mode adds architectural complexity if only dense mode is used
|
|
|
|
## API Test — Ollama
|
|
|
|
```bash
|
|
# Pull model
|
|
ollama pull bge-m3
|
|
|
|
# Test with Farsi insurance query
|
|
curl -s http://localhost:11434/api/embeddings -d '{
|
|
"model": "bge-m3",
|
|
"prompt": "بیمه نامه شخص ثالث چیست؟"
|
|
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'dims={len(d["embedding"])}')"
|
|
```
|
|
|
|
## API Test — Python (sentence-transformers)
|
|
|
|
```python
|
|
from sentence_transformers import SentenceTransformer
|
|
|
|
model = SentenceTransformer("BAAI/bge-m3")
|
|
|
|
# Farsi insurance query
|
|
query = "شرایط لغو بیمه نامه چیست؟"
|
|
embedding = model.encode(query)
|
|
print(f"Dimensions: {len(embedding)}, Type: {type(embedding)}")
|
|
# Expected: Dimensions: 1024
|
|
```
|
|
|
|
## API Test — Python (FlagEmbedding for triple-mode)
|
|
|
|
```python
|
|
from FlagEmbedding import BGEM3FlagModel
|
|
|
|
model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)
|
|
|
|
# Encode with all three modes
|
|
sentences = ["بیمه نامه شخص ثالث چیست؟", "شرایط بیمه آتشسوزی"]
|
|
output = model.encode(sentences, return_dense=True, return_sparse=True, return_colbert_vecs=True)
|
|
|
|
print(f"Dense dims: {output['dense_vecs'].shape}")
|
|
print(f"Sparse tokens: {len(output['lexical_weights'][0])}")
|
|
print(f"ColBERT vecs: {len(output['colbert_vecs'][0])}")
|
|
```
|
|
|
|
## Serving Platform Notes
|
|
|
|
| Platform | Status | Notes |
|
|
|----------|--------|-------|
|
|
| **Ollama** ✅ | Available | Easiest. `ollama pull bge-m3` |
|
|
| **TEI** ✅ | Available | Best throughput for production |
|
|
| **sentence-transformers** ✅ | Available | Direct Python integration |
|
|
| **vLLM** | Not natively | vLLM is for LLM serving, not embeddings |
|
|
|
|
## Qdrant Integration
|
|
|
|
BGE-M3's sparse vectors work natively with Qdrant's sparse vector support:
|
|
|
|
```python
|
|
from qdrant_client import QdrantClient
|
|
from qdrant_client.models import VectorParams, Distance, SparseVectorParams
|
|
|
|
client = QdrantClient("localhost", port=6333)
|
|
|
|
# Create collection with both dense and sparse vectors
|
|
client.create_collection(
|
|
collection_name="insurance_docs",
|
|
vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
|
|
sparse_vectors_config={
|
|
"sparse": SparseVectorParams()
|
|
}
|
|
)
|
|
```
|
|
|
|
## TODO
|
|
|
|
- [ ] Benchmark on MIRACL-Farsi subset
|
|
- [ ] Compare dense-only vs triple-mode retrieval quality
|
|
- [ ] Test with Qdrant sparse vector indexing
|
|
- [ ] Measure inference latency on RTX 3090
|