docs(research): add farsi embedding model research knowledge base
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
98
embedding_models/cohere.md
Normal file
98
embedding_models/cohere.md
Normal file
@@ -0,0 +1,98 @@
|
||||
---
|
||||
last_updated: 2026-07-20
|
||||
tags: [embedding, cohere, cloud, farsi, insurance, rag, enterprise]
|
||||
source: import-knowledge
|
||||
---
|
||||
|
||||
# Cohere — embed-v4.0
|
||||
|
||||
## Overview
|
||||
|
||||
- **Type**: Cloud
|
||||
- **Provider**: Cohere
|
||||
- **Price**: Enterprise only (Model Vault: ~$4.00–$5.00/hr dedicated instance) — legacy embed-multilingual-v3 was ~$0.10/1M tokens ([source](https://cohere.com/pricing))
|
||||
- **Max input tokens**: 512,000
|
||||
- **Embedding dimensions**: 256, 512, 1024, or 1536 (configurable)
|
||||
- **Language coverage**: 100+ languages including Farsi
|
||||
- **Persian benchmark**: Cohere models score well on MIRACL-Farsi on the MTEB leaderboard. No isolated Farsi numbers published by Cohere.
|
||||
|
||||
## Why It's #1 Cloud Pick
|
||||
|
||||
**512K context window** — largest of any embedding API. Can embed full insurance documents without chunking. Strong Farsi support (100+ languages). MIRACL-Farsi scores on MTEB leaderboard.
|
||||
|
||||
**Caveat**: Enterprise-only pricing. No public per-token API anymore.
|
||||
|
||||
## Pros
|
||||
|
||||
- 100+ languages with explicit Farsi support
|
||||
- 512K context — can embed full documents, no chunking needed
|
||||
- Strong MIRACL-Farsi scores
|
||||
- v2 API with input_type distinction (search_query vs search_document)
|
||||
- Configurable dimensions (256–1536)
|
||||
|
||||
## Cons
|
||||
|
||||
- Enterprise-only pricing (Model Vault: ~$4–5/hr dedicated instance)
|
||||
- No public per-token API pricing — requires sales engagement
|
||||
- Smaller ecosystem than OpenAI
|
||||
|
||||
## API Test — Curl
|
||||
|
||||
**Endpoint**: `POST https://api.cohere.ai/v2/embed`
|
||||
**Auth**: `Authorization: Bearer $COHERE_API_KEY`
|
||||
**Get key**: https://dashboard.cohere.com/api-keys
|
||||
|
||||
```bash
|
||||
curl -s https://api.cohere.ai/v2/embed -H "Content-Type: application/json" -H "Authorization: Bearer $COHERE_API_KEY" -d '{
|
||||
"model": "embed-v4.0",
|
||||
"texts": ["بیمه نامه شخص ثالث چیست؟"],
|
||||
"input_type": "search_query",
|
||||
"embedding_types": ["float"]
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### ⚠️ Key Differences from OpenAI
|
||||
|
||||
| Aspect | OpenAI | Cohere |
|
||||
|--------|--------|--------|
|
||||
| Input field | `"input"` (string/array) | `"texts"` (array) |
|
||||
| Input type | N/A | Required: `"search_query"` or `"search_document"` |
|
||||
| Output format | `data[0].embedding` | `data.embeddings.float[0]` |
|
||||
| Token count | `usage.total_tokens` | `meta.billed_units.input_tokens` |
|
||||
|
||||
### Indexing Documents (use search_document)
|
||||
|
||||
```bash
|
||||
curl -s https://api.cohere.ai/v2/embed -H "Content-Type: application/json" -H "Authorization: Bearer $COHERE_API_KEY" -d '{
|
||||
"model": "embed-v4.0",
|
||||
"texts": ["بیمه نامه شخص ثالث شامل پوشش خسارات مالی و جانی است."],
|
||||
"input_type": "search_document",
|
||||
"embedding_types": ["float"]
|
||||
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'dims={len(d["data"]["embeddings"]["float"][0])}, tokens={d["meta"]["billed_units"]["input_tokens"]}')"
|
||||
```
|
||||
|
||||
## API Test — Python
|
||||
|
||||
```python
|
||||
import cohere
|
||||
|
||||
co = cohere.ClientV2(api_key="YOUR_COHERE_API_KEY") # or use COHERE_API_KEY env var
|
||||
|
||||
response = co.embed(
|
||||
model="embed-v4.0",
|
||||
input_type="search_query",
|
||||
texts=["بیمه نامه شخص ثالث چیست؟"],
|
||||
embedding_types=["float"],
|
||||
output_dimension=1024
|
||||
)
|
||||
|
||||
embedding = response.embeddings.float[0]
|
||||
print(f"Dimensions: {len(embedding)}")
|
||||
```
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Contact Cohere sales for enterprise pricing
|
||||
- [ ] Benchmark on MIRACL-Farsi subset
|
||||
- [ ] Compare with OpenAI text-embedding-3-large on Farsi queries
|
||||
- [ ] Verify on-premises deployment options
|
||||
Reference in New Issue
Block a user