docs(research): add farsi embedding model research knowledge base

Body:
        Why:
        - Insurance RAG pipeline needs Farsi-optimized embeddings
        - Current local baseline (Nomic Embed) is English-only

        Changes:
        - 8 cloud model profiles with curl + python test commands
        - 5 local model profiles with VRAM, serving platform notes
        - Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
        - Decision report with ranked options and recommendations
        - Full comparison table (15 models × 10 columns)

        Impact:
        - Knowledge base for embedding model selection
        - Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
        - Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
2026-07-20 15:00:11 +03:30
parent 72d8a3ebe4
commit 6c9cebaaa0
13 changed files with 1296 additions and 0 deletions

View File

@@ -0,0 +1,97 @@
---
last_updated: 2026-07-20
tags: [embedding, voyage-ai, cloud, farsi, insurance, rag, free-tier]
source: import-knowledge
---
# Voyage AI — voyage-4-large & voyage-4
## Overview
Voyage AI's latest embedding models. Founded by OpenAI alumni. 200M free tokens — risk-free testing.
## voyage-4-large *(recommended)*
- **Type**: Cloud
- **Provider**: Voyage AI
- **Price**: $0.12 per 1M input tokens (200M free tokens included) ([source](https://docs.voyageai.com/docs/pricing))
- **Max input tokens**: 32,000
- **Embedding dimensions**: 1024
- **Language coverage**: Multilingual
- **Persian benchmark**: No public Persian-specific scores. MTEB multilingual aggregate available.
- **Notes**: Slightly cheaper than OpenAI text-embedding-3-large ($0.12 vs $0.13). 32K context — 4× OpenAI's 8K. voyage-3-large and voyage-3 are now legacy.
## voyage-4 *(mid-tier)*
- **Type**: Cloud
- **Provider**: Voyage AI
- **Price**: $0.06 per 1M input tokens (200M free tokens included) ([source](https://docs.voyageai.com/docs/pricing))
- **Max input tokens**: 32,000
- **Embedding dimensions**: 1024
- **Language coverage**: Multilingual
- **Persian benchmark**: No public Persian-specific scores.
## Pros
- $0.12/1M tokens — slightly cheaper than OpenAI
- 200M tokens free — enough for testing + months of low-volume traffic
- 32K context — 4× OpenAI's context window
- Strong MTEB multilingual scores
- Simple API (OpenAI-compatible format)
## Cons
- No published Farsi-specific scores
- Smaller company — less battle-tested at enterprise scale
- Requires separate API key management
## API Test — Curl
**Endpoint**: `POST https://api.voyageai.com/v1/embeddings`
**Auth**: `Authorization: Bearer $VOYAGE_API_KEY`
**Get key**: https://voyage.ai/api-keys
**Free tier**: 200M tokens
### voyage-4-large
```bash
curl -s https://api.voyageai.com/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $VOYAGE_API_KEY" -d '{
"model": "voyage-4-large",
"input": ["بیمه نامه شخص ثالث چیست؟"]
}' | python3 -m json.tool
```
### voyage-4
```bash
curl -s https://api.voyageai.com/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $VOYAGE_API_KEY" -d '{
"model": "voyage-4",
"input": ["بیمه نامه شخص ثالث چیست؟"]
}' | python3 -m json.tool
```
**Response shape**: `data[0].embedding` = float array, `usage.total_tokens` = token count
## API Test — Python
```python
import voyageai
vo = voyageai.Client(api_key="YOUR_VOYAGE_API_KEY") # or uses VOYAGE_API_KEY env var
result = vo.embed(
["بیمه نامه شخص ثالث چیست؟"],
model="voyage-4-large",
input_type="document"
)
embedding = result.embeddings[0]
print(f"Dimensions: {len(embedding)}, Tokens: {result.total_tokens}")
```
## TODO
- [ ] Run 100-question eval set against voyage-4-large (200M free tokens)
- [ ] Benchmark on MIRACL-Farsi subset
- [ ] Compare with OpenAI text-embedding-3-large on Farsi queries
- [ ] Calculate cost at production volume