docs(research): add farsi embedding model research knowledge base

Body:
        Why:
        - Insurance RAG pipeline needs Farsi-optimized embeddings
        - Current local baseline (Nomic Embed) is English-only

        Changes:
        - 8 cloud model profiles with curl + python test commands
        - 5 local model profiles with VRAM, serving platform notes
        - Persian benchmark landscape (MIRACL-Fa, mMARCO, FaMTE, MTEB)
        - Decision report with ranked options and recommendations
        - Full comparison table (15 models × 10 columns)

        Impact:
        - Knowledge base for embedding model selection
        - Top picks: BGE-M3 (local), Cohere embed-v4.0 (cloud)
        - Critical: Nomic Embed must be replaced (English-only)
This commit is contained in:
2026-07-20 15:00:11 +03:30
parent 72d8a3ebe4
commit 6c9cebaaa0
13 changed files with 1296 additions and 0 deletions

111
embedding_models/jina.md Normal file
View File

@@ -0,0 +1,111 @@
---
last_updated: 2026-07-20
tags: [embedding, jina, cloud, local, farsi, insurance, rag, cheap]
source: import-knowledge
---
# Jina AI — jina-embeddings-v3
## Overview
Available as both cloud API and local model. Cheapest cloud option with explicit Farsi support. **⚠️ Local model has non-commercial license (CC-BY-NC-4.0).**
## Cloud API
- **Type**: Cloud
- **Provider**: Jina AI
- **Price**: ~$0.02 per 1M tokens ([source](https://jina.ai/pricing/))
- **Max input tokens**: 8,192
- **Embedding dimensions**: 1024 (Matryoshka — resizable)
- **Language coverage**: 89 languages including Farsi
- **Persian benchmark**: No public Persian-specific scores. Strong MTEB multilingual.
## Local Model
- **Parameters**: ~568M
- **VRAM (fp16)**: ~1.1 GB
- **License**: CC-BY-NC-4.0 ⚠️ (non-commercial!) — **verify for enterprise use**
- **Model card**: https://huggingface.co/jinaai/jina-embeddings-v3
## Pros
- Cheapest cloud option: ~$0.02/1M tokens
- 89 languages including Farsi
- Task-specific adapters (retrieval, clustering, classification)
- Matryoshka support (resizable dimensions)
- 8K context
- ColBERT/late interaction support
## Cons
- Local model is CC-BY-NC (non-commercial) — verify for enterprise
- Smaller company — less enterprise track record
- Pricing page not transparent (SPA, not standard display)
## API Test — Curl
**Endpoint**: `POST https://api.jina.ai/v1/embeddings`
**Auth**: `Authorization: Bearer $JINA_API_KEY`
**Get key**: https://jina.ai/api-keys
### For queries
```bash
curl -s https://api.jina.ai/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $JINA_API_KEY" -d '{
"model": "jina-embeddings-v3",
"input": ["بیمه نامه شخص ثالث چیست؟"],
"task": "retrieval.query"
}' | python3 -m json.tool
```
### For document chunks (indexing)
```bash
curl -s https://api.jina.ai/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $JINA_API_KEY" -d '{
"model": "jina-embeddings-v3",
"input": ["بیمه نامه شخص ثالث شامل پوشش خسارات مالی و جانی است."],
"task": "retrieval.passage"
}' | python3 -m json.tool
```
### With Matryoshka dimension reduction
```bash
curl -s https://api.jina.ai/v1/embeddings -H "Content-Type: application/json" -H "Authorization: Bearer $JINA_API_KEY" -d '{
"model": "jina-embeddings-v3",
"input": ["بیمه نامه شخص ثالث چیست؟"],
"task": "retrieval.query",
"dimensions": 512
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'dims={len(d["data"][0]["embedding"])}')"
```
**Key differences from OpenAI**:
- Use `task: "retrieval.query"` for queries, `task: "retrieval.passage"` for chunks
- Optional `dimensions` param for Matryoshka truncation
- Optional `late_chunking: true` for late-interaction-style results
**Response shape**: `data[0].embedding` = float array, `usage.total_tokens` = token count
## API Test — Python
```python
from jina import Client
client = Client(api_key="YOUR_JINA_API_KEY") # or uses JINA_API_KEY env var
result = client.embed(
inputs=["بیمه نامه شخص ثالث چیست?"],
model="jina-embeddings-v3",
task="retrieval.query"
)
embedding = result.embeddings[0].embedding
print(f"Dimensions: {len(embedding)}")
```
## TODO
- [ ] Verify enterprise licensing terms for local model (CC-BY-NC)
- [ ] Test cloud API with 100-question eval set
- [ ] Benchmark on MIRACL-Farsi subset
- [ ] Compare with OpenAI and Cohere on Farsi queries