docs(research): add farsi embedding model research knowledge base

Body:
        Why:
        - Insurance RAG pipeline needs Farsi-optimized embeddings
        - Current local baseline (Nomic Embed) is English-only

        Changes:
        - 8 cloud model profiles with curl + python test commands
        - 5 local model profiles with VRAM, serving platform notes
        - Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
        - Decision report with ranked options and recommendations
        - Full comparison table (15 models × 10 columns)

        Impact:
        - Knowledge base for embedding model selection
        - Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
        - Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
2026-07-20 15:00:11 +03:30
parent 72d8a3ebe4
commit 6c9cebaaa0
13 changed files with 1296 additions and 0 deletions

View File

@@ -0,0 +1,98 @@
---
last_updated: 2026-07-20
tags: [embedding, cohere, cloud, farsi, insurance, rag, enterprise]
source: import-knowledge
---
# Cohere — embed-v4.0
## Overview
- **Type**: Cloud
- **Provider**: Cohere
- **Price**: Enterprise only (Model Vault: ~$4.00$5.00/hr dedicated instance) — legacy embed-multilingual-v3 was ~$0.10/1M tokens ([source](https://cohere.com/pricing))
- **Max input tokens**: 512,000
- **Embedding dimensions**: 256, 512, 1024, or 1536 (configurable)
- **Language coverage**: 100+ languages including Farsi
- **Persian benchmark**: Cohere models score well on MIRACL-Farsi on the MTEB leaderboard. No isolated Farsi numbers published by Cohere.
## Why It's #1 Cloud Pick
**512K context window** — largest of any embedding API. Can embed full insurance documents without chunking. Strong Farsi support (100+ languages). MIRACL-Farsi scores on MTEB leaderboard.
**Caveat**: Enterprise-only pricing. No public per-token API anymore.
## Pros
- 100+ languages with explicit Farsi support
- 512K context — can embed full documents, no chunking needed
- Strong MIRACL-Farsi scores
- v2 API with input_type distinction (search_query vs search_document)
- Configurable dimensions (2561536)
## Cons
- Enterprise-only pricing (Model Vault: ~$45/hr dedicated instance)
- No public per-token API pricing — requires sales engagement
- Smaller ecosystem than OpenAI
## API Test — Curl
**Endpoint**: `POST https://api.cohere.ai/v2/embed`
**Auth**: `Authorization: Bearer $COHERE_API_KEY`
**Get key**: https://dashboard.cohere.com/api-keys
```bash
curl -s https://api.cohere.ai/v2/embed -H "Content-Type: application/json" -H "Authorization: Bearer $COHERE_API_KEY" -d '{
"model": "embed-v4.0",
"texts": ["بیمه نامه شخص ثالث چیست؟"],
"input_type": "search_query",
"embedding_types": ["float"]
}' | python3 -m json.tool
```
### ⚠️ Key Differences from OpenAI
| Aspect | OpenAI | Cohere |
|--------|--------|--------|
| Input field | `"input"` (string/array) | `"texts"` (array) |
| Input type | N/A | Required: `"search_query"` or `"search_document"` |
| Output format | `data[0].embedding` | `data.embeddings.float[0]` |
| Token count | `usage.total_tokens` | `meta.billed_units.input_tokens` |
### Indexing Documents (use search_document)
```bash
curl -s https://api.cohere.ai/v2/embed -H "Content-Type: application/json" -H "Authorization: Bearer $COHERE_API_KEY" -d '{
"model": "embed-v4.0",
"texts": ["بیمه نامه شخص ثالث شامل پوشش خسارات مالی و جانی است."],
"input_type": "search_document",
"embedding_types": ["float"]
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'dims={len(d["data"]["embeddings"]["float"][0])}, tokens={d["meta"]["billed_units"]["input_tokens"]}')"
```
## API Test — Python
```python
import cohere
co = cohere.ClientV2(api_key="YOUR_COHERE_API_KEY") # or use COHERE_API_KEY env var
response = co.embed(
model="embed-v4.0",
input_type="search_query",
texts=["بیمه نامه شخص ثالث چیست؟"],
embedding_types=["float"],
output_dimension=1024
)
embedding = response.embeddings.float[0]
print(f"Dimensions: {len(embedding)}")
```
## TODO
- [ ] Contact Cohere sales for enterprise pricing
- [ ] Benchmark on MIRACL-Farsi subset
- [ ] Compare with OpenAI text-embedding-3-large on Farsi queries
- [ ] Verify on-premises deployment options