docs(research): add farsi embedding model research knowledge base

Body:
        Why:
        - Insurance RAG pipeline needs Farsi-optimized embeddings
        - Current local baseline (Nomic Embed) is English-only

        Changes:
        - 8 cloud model profiles with curl + python test commands
        - 5 local model profiles with VRAM, serving platform notes
        - Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
        - Decision report with ranked options and recommendations
        - Full comparison table (15 models × 10 columns)

        Impact:
        - Knowledge base for embedding model selection
        - Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
        - Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
2026-07-20 15:00:11 +03:30
parent 72d8a3ebe4
commit 6c9cebaaa0
13 changed files with 1296 additions and 0 deletions

View File

@@ -0,0 +1,99 @@
---
last_updated: 2026-07-20
tags: [embedding, google, vertex-ai, cloud, farsi, insurance, rag]
source: import-knowledge
---
# Google Vertex AI — Embedding Models
## Models
### Gemini Embedding *(newest)*
- **Type**: Cloud
- **Provider**: Google Vertex AI
- **Price**: $0.15 per 1M input tokens (online) / $0.12 per 1M (batch) ([source](https://cloud.google.com/vertex-ai/generative-ai/pricing))
- **Max input tokens**: 8,192
- **Embedding dimensions**: 768
- **Language coverage**: Multilingual (Gemini family)
### text-embedding-004
- **Price**: ~$0.10 per 1M tokens equivalent ($0.025 per 1,000 characters) ([source](https://cloud.google.com/vertex-ai/generative-ai/pricing))
- **Language coverage**: English-primary with some multilingual
### text-multilingual-embedding-002 *(Farsi-relevant)*
- **Price**: ~$0.10 per 1M tokens equivalent (same pricing as above)
- **Language coverage**: Multilingual — this is the relevant variant for Farsi
**Note**: Legacy Google embedding models priced per character, not per token (~4 chars/token).
## Pros
- Google ecosystem integration
- Batch pricing available (20% discount)
- Gemini Embedding is Google's latest
## Cons
- Most complex setup (requires GCP project, billing, Vertex AI API)
- No Farsi-specific scores published
- Per-character pricing for legacy models (confusing)
- $0.15/1M tokens for Gemini Embedding — most expensive option
## API Test — Curl
**Auth**: `gcloud auth print-access-token` or service account JSON
**Setup**: Requires GCP project with Vertex AI API enabled
### Gemini Embedding
```bash
export GCP_TOKEN=$(gcloud auth print-access-token)
curl -s "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/us-central1/publishers/google/models/gemini-embedding-exp-03-07:predict" -H "Content-Type: application/json" -H "Authorization: Bearer $GCP_TOKEN" -d '{
"instances": [
{"content": "بیمه نامه شخص ثالث چیست؟"}
]
}' | python3 -m json.tool
```
### text-multilingual-embedding-002
```bash
curl -s "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/us-central1/publishers/google/models/text-multilingual-embedding-002:predict" -H "Content-Type: application/json" -H "Authorization: Bearer $GCP_TOKEN" -d '{
"instances": [
{"content": "بیمه نامه شخص ثالث چیست؟"}
]
}' | python3 -m json.tool
```
### text-embedding-004
```bash
curl -s "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/us-central1/publishers/google/models/text-embedding-004:predict" -H "Content-Type: application/json" -H "Authorization: Bearer $GCP_TOKEN" -d '{
"instances": [
{"content": "بیمه نامه شخص ثالث چیست؟"}
]
}' | python3 -m json.tool
```
**Response shape**: `predictions[0].embeddings.values` = float array (different from OpenAI!)
## API Test — Python
```python
from google.cloud import aiplatform
aiplatform.init(project="YOUR_PROJECT_ID", location="us-central1")
model = aiplatform.Endpoint("projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID")
response = model.predict(instances=[{"content": "بیمه نامه شخص ثالث چیست؟"}])
embedding = response.predictions[0]["embeddings"]["values"]
print(f"Dimensions: {len(embedding)}")
```
## TODO
- [ ] Set up GCP project if not already available
- [ ] Benchmark text-multilingual-embedding-002 on Farsi queries
- [ ] Compare with OpenAI on Farsi retrieval quality