67 lines
2.4 KiB
Markdown
67 lines
2.4 KiB
Markdown
# ADR 0003: Dual Embedding Model Strategy
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
The system operates in an environment with intermittent internet connectivity, including government-imposed blackouts lasting up to 90 days. The retrieval system must function fully offline with zero external API dependencies during these periods.
|
|
|
|
Documents and queries are bilingual (Persian/Farsi and English). Embedding models must handle both languages effectively.
|
|
|
|
## Decision
|
|
|
|
Store **two embeddings per chunk** in Qdrant using named vectors:
|
|
|
|
| Model | Use Case | Type |
|
|
|-------|----------|------|
|
|
| `text-embedding-3-large` (OpenAI) | Primary retrieval when internet available | Cloud |
|
|
| `nomic-embed-text` | Fallback during blackouts, fully local | On-prem |
|
|
|
|
### Qdrant Configuration
|
|
```python
|
|
# Named vectors in Qdrant collection
|
|
collection_config = {
|
|
"vectors": {
|
|
"openai": {"size": 3072, "distance": "Cosine"},
|
|
"nomic": {"size": 768, "distance": "Cosine"}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Query-Time Behavior
|
|
- **Online:** Query with OpenAI embeddings, retrieve using `openai` vector
|
|
- **Offline:** Query with nomic embeddings, retrieve using `nomic` vector
|
|
- **A/B testing:** Can compare retrieval quality between models in real-time
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
- **Resilience** — system works identically online and offline
|
|
- **No re-indexing** — switch models at query time, not ingestion time
|
|
- **A/B testing** — can measure retrieval quality difference between models
|
|
- **Graceful degradation** — fallback is automatic when cloud APIs unavailable
|
|
|
|
### Negative
|
|
- **2x storage** — two vectors per chunk
|
|
- **2x ingestion time** — must embed with both models during indexing
|
|
- **Complexity** — query logic must select correct vector at runtime
|
|
|
|
### Neutral
|
|
- Requires monitoring to detect when cloud APIs become unavailable
|
|
- May need to tune retrieval parameters (top_k, score threshold) per model
|
|
|
|
## Generation Layer
|
|
|
|
During offline periods, answer generation uses **Qwen 2.5** running locally. This provides:
|
|
- Bilingual support (Persian/English)
|
|
- Fully on-prem inference
|
|
- No external API dependencies
|
|
|
|
## Alternatives Considered
|
|
|
|
1. **Single model (nomic only)** — simpler but loses quality of `text-embedding-3-large` when online
|
|
2. **Single model (multilingual-e5)** — good bilingual support but not tested in production
|
|
3. **Re-embed on switch** — would require re-indexing entire corpus during blackout (impractical)
|