docs(research): add farsi embedding model research knowledge base
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
97
embedding_models/voyage.md
Normal file
97
embedding_models/voyage.md
Normal file
@@ -0,0 +1,97 @@
|
||||
---
|
||||
last_updated: 2026-07-20
|
||||
tags: [embedding, voyage-ai, cloud, farsi, insurance, rag, free-tier]
|
||||
source: import-knowledge
|
||||
---
|
||||
|
||||
# Voyage AI — voyage-4-large & voyage-4
|
||||
|
||||
## Overview
|
||||
|
||||
Voyage AI's latest embedding models. Founded by OpenAI alumni. 200M free tokens — risk-free testing.
|
||||
|
||||
## voyage-4-large *(recommended)*
|
||||
|
||||
- **Type**: Cloud
|
||||
- **Provider**: Voyage AI
|
||||
- **Price**: $0.12 per 1M input tokens (200M free tokens included) ([source](https://docs.voyageai.com/docs/pricing))
|
||||
- **Max input tokens**: 32,000
|
||||
- **Embedding dimensions**: 1024
|
||||
- **Language coverage**: Multilingual
|
||||
- **Persian benchmark**: No public Persian-specific scores. MTEB multilingual aggregate available.
|
||||
- **Notes**: Slightly cheaper than OpenAI text-embedding-3-large ($0.12 vs $0.13). 32K context — 4× OpenAI's 8K. voyage-3-large and voyage-3 are now legacy.
|
||||
|
||||
## voyage-4 *(mid-tier)*
|
||||
|
||||
- **Type**: Cloud
|
||||
- **Provider**: Voyage AI
|
||||
- **Price**: $0.06 per 1M input tokens (200M free tokens included) ([source](https://docs.voyageai.com/docs/pricing))
|
||||
- **Max input tokens**: 32,000
|
||||
- **Embedding dimensions**: 1024
|
||||
- **Language coverage**: Multilingual
|
||||
- **Persian benchmark**: No public Persian-specific scores.
|
||||
|
||||
## Pros
|
||||
|
||||
- $0.12/1M tokens — slightly cheaper than OpenAI
|
||||
- 200M tokens free — enough for testing + months of low-volume traffic
|
||||
- 32K context — 4× OpenAI's context window
|
||||
- Strong MTEB multilingual scores
|
||||
- Simple API (OpenAI-compatible format)
|
||||
|
||||
## Cons
|
||||
|
||||
- No published Farsi-specific scores
|
||||
- Smaller company — less battle-tested at enterprise scale
|
||||
- Requires separate API key management
|
||||
|
||||
## API Test — Curl
|
||||
|
||||
**Endpoint**: `POST https://api.voyageai.com/v1/embeddings`
|
||||
**Auth**: `Authorization: Bearer $VOYAGE_API_KEY`
|
||||
**Get key**: https://voyage.ai/api-keys
|
||||
**Free tier**: 200M tokens
|
||||
|
||||
### voyage-4-large
|
||||
|
||||
```bash
|
||||
curl -s https://api.voyageai.com/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $VOYAGE_API_KEY" -d '{
|
||||
"model": "voyage-4-large",
|
||||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### voyage-4
|
||||
|
||||
```bash
|
||||
curl -s https://api.voyageai.com/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $VOYAGE_API_KEY" -d '{
|
||||
"model": "voyage-4",
|
||||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
**Response shape**: `data[0].embedding` = float array, `usage.total_tokens` = token count
|
||||
|
||||
## API Test — Python
|
||||
|
||||
```python
|
||||
import voyageai
|
||||
|
||||
vo = voyageai.Client(api_key="YOUR_VOYAGE_API_KEY") # or uses VOYAGE_API_KEY env var
|
||||
|
||||
result = vo.embed(
|
||||
["بیمه نامه شخص ثالث چیست؟"],
|
||||
model="voyage-4-large",
|
||||
input_type="document"
|
||||
)
|
||||
|
||||
embedding = result.embeddings[0]
|
||||
print(f"Dimensions: {len(embedding)}, Tokens: {result.total_tokens}")
|
||||
```
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Run 100-question eval set against voyage-4-large (200M free tokens)
|
||||
- [ ] Benchmark on MIRACL-Farsi subset
|
||||
- [ ] Compare with OpenAI text-embedding-3-large on Farsi queries
|
||||
- [ ] Calculate cost at production volume
|
||||
Reference in New Issue
Block a user