Mahdi Bazrafshan
6c9cebaaa0
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
123 lines
4.0 KiB
Markdown
123 lines
4.0 KiB
Markdown
---
|
||
last_updated: 2026-07-20
|
||
tags: [embedding, openai, cloud, farsi, insurance, rag]
|
||
source: import-knowledge
|
||
---
|
||
|
||
# OpenAI — text-embedding-3-large & text-embedding-3-small
|
||
|
||
## Overview
|
||
|
||
OpenAI's embedding API is the current baseline for our RAG pipeline. Two models available: text-embedding-3-large (higher quality) and text-embedding-3-small (cheaper).
|
||
|
||
## text-embedding-3-large *(current baseline)*
|
||
|
||
- **Type**: Cloud
|
||
- **Provider**: OpenAI
|
||
- **Price**: $0.13 per 1M input tokens ([source](https://openai.com/api/pricing/))
|
||
- **Max input tokens**: 8,191
|
||
- **Embedding dimensions**: 256, 1024, or 3072 (configurable via `dimensions` param)
|
||
- **Language coverage**: Multilingual (trained on multilingual corpus including Farsi)
|
||
- **Persian benchmark**: No public Persian-specific retrieval benchmark score published by OpenAI. MTEB aggregate available; MIRACL-Farsi subtask scores exist on the MTEB leaderboard but not officially published by OpenAI.
|
||
- **Notes**: Good general multilingual performance but not specifically optimized for Farsi. Pricing is mid-range among cloud options. Already integrated — zero migration cost.
|
||
|
||
## text-embedding-3-small *(cheaper alternative)*
|
||
|
||
- **Type**: Cloud
|
||
- **Provider**: OpenAI
|
||
- **Price**: $0.02 per 1M input tokens ([source](https://openai.com/api/pricing/))
|
||
- **Max input tokens**: 8,191
|
||
- **Embedding dimensions**: 512 or 1536 (configurable)
|
||
- **Language coverage**: Multilingual
|
||
- **Persian benchmark**: No public Persian-specific scores.
|
||
- **Notes**: 6.5× cheaper than text-embedding-3-large but generally lower retrieval quality. Useful as a cost-optimized fallback, not as a quality upgrade.
|
||
|
||
## API Test — Curl
|
||
|
||
**Endpoint**: `POST https://api.openai.com/v1/embeddings`
|
||
**Auth**: `Authorization: Bearer $OPENAI_API_KEY`
|
||
**Get key**: https://platform.openai.com/api-keys
|
||
|
||
### text-embedding-3-large
|
||
|
||
```bash
|
||
curl -s https://api.openai.com/v1/embeddings \
|
||
-H "Content-Type: application/json" \
|
||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||
-d '{
|
||
"model": "text-embedding-3-large",
|
||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||
}' | python3 -m json.tool
|
||
```
|
||
|
||
### text-embedding-3-small
|
||
|
||
```bash
|
||
curl -s https://api.openai.com/v1/embeddings \
|
||
-H "Content-Type: application/json" \
|
||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||
-d '{
|
||
"model": "text-embedding-3-small",
|
||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||
}' | python3 -m json.tool
|
||
```
|
||
|
||
**Response shape**: `data[0].embedding` = float array, `usage.total_tokens` = token count
|
||
|
||
## API Test — Python
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI() # uses OPENAI_API_KEY env var
|
||
|
||
# Test with a Farsi insurance query
|
||
response = client.embeddings.create(
|
||
model="text-embedding-3-large",
|
||
input=["بیمه نامه شخص ثالث چیست؟"]
|
||
)
|
||
|
||
embedding = response.data[0].embedding
|
||
tokens = response.usage.total_tokens
|
||
print(f"Dimensions: {len(embedding)}, Tokens: {tokens}")
|
||
# Expected: Dimensions: 3072, Tokens: 6
|
||
```
|
||
|
||
## Python Test — Compare Both Models
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
import time
|
||
|
||
client = OpenAI()
|
||
query = "شرایط لغو بیمه نامه چیست؟"
|
||
|
||
for model in ["text-embedding-3-large", "text-embedding-3-small"]:
|
||
start = time.time()
|
||
resp = client.embeddings.create(model=model, input=[query])
|
||
elapsed = time.time() - start
|
||
dims = len(resp.data[0].embedding)
|
||
tokens = resp.usage.total_tokens
|
||
print(f"{model}: dims={dims}, tokens={tokens}, time={elapsed:.2f}s")
|
||
```
|
||
|
||
## Response Format
|
||
|
||
| Field | Path |
|
||
|-------|------|
|
||
| Embedding vector | `data[0].embedding` |
|
||
| Token count | `usage.total_tokens` |
|
||
|
||
## Key Differences from Other Providers
|
||
|
||
- Uses standard `input` field (string or array)
|
||
- Configurable dimensions — can reduce from 3072 to 1024 or 256 to save storage/latency
|
||
- 8K context limit — may require aggressive chunking for long insurance documents
|
||
- Data sent to US servers — not on-premises
|
||
|
||
## TODO
|
||
|
||
- [ ] Benchmark on MIRACL-Farsi subset
|
||
- [ ] Test dimension reduction (3072 → 1024) impact on retrieval quality
|
||
- [ ] Calculate monthly cost at production volume
|