docs(research): add farsi embedding model research knowledge base
Body:
Why:
- Insurance RAG pipeline needs Farsi-optimized embeddings
- Current local baseline (Nomic Embed) is English-only
Changes:
- 8 cloud model profiles with curl + python test commands
- 5 local model profiles with VRAM, serving platform notes
- Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
- Decision report with ranked options and recommendations
- Full comparison table (15 models × 10 columns)
Impact:
- Knowledge base for embedding model selection
- Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
- Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
122
embedding_models/openai.md
Normal file
122
embedding_models/openai.md
Normal file
@@ -0,0 +1,122 @@
|
||||
---
|
||||
last_updated: 2026-07-20
|
||||
tags: [embedding, openai, cloud, farsi, insurance, rag]
|
||||
source: import-knowledge
|
||||
---
|
||||
|
||||
# OpenAI — text-embedding-3-large & text-embedding-3-small
|
||||
|
||||
## Overview
|
||||
|
||||
OpenAI's embedding API is the current baseline for our RAG pipeline. Two models available: text-embedding-3-large (higher quality) and text-embedding-3-small (cheaper).
|
||||
|
||||
## text-embedding-3-large *(current baseline)*
|
||||
|
||||
- **Type**: Cloud
|
||||
- **Provider**: OpenAI
|
||||
- **Price**: $0.13 per 1M input tokens ([source](https://openai.com/api/pricing/))
|
||||
- **Max input tokens**: 8,191
|
||||
- **Embedding dimensions**: 256, 1024, or 3072 (configurable via `dimensions` param)
|
||||
- **Language coverage**: Multilingual (trained on multilingual corpus including Farsi)
|
||||
- **Persian benchmark**: No public Persian-specific retrieval benchmark score published by OpenAI. MTEB aggregate available; MIRACL-Farsi subtask scores exist on the MTEB leaderboard but not officially published by OpenAI.
|
||||
- **Notes**: Good general multilingual performance but not specifically optimized for Farsi. Pricing is mid-range among cloud options. Already integrated — zero migration cost.
|
||||
|
||||
## text-embedding-3-small *(cheaper alternative)*
|
||||
|
||||
- **Type**: Cloud
|
||||
- **Provider**: OpenAI
|
||||
- **Price**: $0.02 per 1M input tokens ([source](https://openai.com/api/pricing/))
|
||||
- **Max input tokens**: 8,191
|
||||
- **Embedding dimensions**: 512 or 1536 (configurable)
|
||||
- **Language coverage**: Multilingual
|
||||
- **Persian benchmark**: No public Persian-specific scores.
|
||||
- **Notes**: 6.5× cheaper than text-embedding-3-large but generally lower retrieval quality. Useful as a cost-optimized fallback, not as a quality upgrade.
|
||||
|
||||
## API Test — Curl
|
||||
|
||||
**Endpoint**: `POST https://api.openai.com/v1/embeddings`
|
||||
**Auth**: `Authorization: Bearer $OPENAI_API_KEY`
|
||||
**Get key**: https://platform.openai.com/api-keys
|
||||
|
||||
### text-embedding-3-large
|
||||
|
||||
```bash
|
||||
curl -s https://api.openai.com/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-d '{
|
||||
"model": "text-embedding-3-large",
|
||||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### text-embedding-3-small
|
||||
|
||||
```bash
|
||||
curl -s https://api.openai.com/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-d '{
|
||||
"model": "text-embedding-3-small",
|
||||
"input": ["بیمه نامه شخص ثالث چیست؟"]
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
**Response shape**: `data[0].embedding` = float array, `usage.total_tokens` = token count
|
||||
|
||||
## API Test — Python
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI() # uses OPENAI_API_KEY env var
|
||||
|
||||
# Test with a Farsi insurance query
|
||||
response = client.embeddings.create(
|
||||
model="text-embedding-3-large",
|
||||
input=["بیمه نامه شخص ثالث چیست؟"]
|
||||
)
|
||||
|
||||
embedding = response.data[0].embedding
|
||||
tokens = response.usage.total_tokens
|
||||
print(f"Dimensions: {len(embedding)}, Tokens: {tokens}")
|
||||
# Expected: Dimensions: 3072, Tokens: 6
|
||||
```
|
||||
|
||||
## Python Test — Compare Both Models
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
import time
|
||||
|
||||
client = OpenAI()
|
||||
query = "شرایط لغو بیمه نامه چیست؟"
|
||||
|
||||
for model in ["text-embedding-3-large", "text-embedding-3-small"]:
|
||||
start = time.time()
|
||||
resp = client.embeddings.create(model=model, input=[query])
|
||||
elapsed = time.time() - start
|
||||
dims = len(resp.data[0].embedding)
|
||||
tokens = resp.usage.total_tokens
|
||||
print(f"{model}: dims={dims}, tokens={tokens}, time={elapsed:.2f}s")
|
||||
```
|
||||
|
||||
## Response Format
|
||||
|
||||
| Field | Path |
|
||||
|-------|------|
|
||||
| Embedding vector | `data[0].embedding` |
|
||||
| Token count | `usage.total_tokens` |
|
||||
|
||||
## Key Differences from Other Providers
|
||||
|
||||
- Uses standard `input` field (string or array)
|
||||
- Configurable dimensions — can reduce from 3072 to 1024 or 256 to save storage/latency
|
||||
- 8K context limit — may require aggressive chunking for long insurance documents
|
||||
- Data sent to US servers — not on-premises
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Benchmark on MIRACL-Farsi subset
|
||||
- [ ] Test dimension reduction (3072 → 1024) impact on retrieval quality
|
||||
- [ ] Calculate monthly cost at production volume
|
||||
Reference in New Issue
Block a user