98 lines
2.7 KiB
Markdown
98 lines
2.7 KiB
Markdown
# ADR 0011: Inference Infrastructure
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
The system needs local inference for:
|
|
- **LLM generation:** Qwen 2.5 (bilingual Persian/English)
|
|
- **Local embeddings:** Nomic-embed-text
|
|
- **Local reranking:** BGE Reranker
|
|
|
|
Hardware available: NVIDIA RTX 3090 (24GB VRAM)
|
|
|
|
## Decision
|
|
|
|
### Local Inference Stack
|
|
|
|
| Component | Tool | Model | VRAM Usage |
|
|
|-----------|------|-------|------------|
|
|
| LLM | Ollama | Qwen 2.5 | ~12-16GB (depends on quantization) |
|
|
| Embeddings | Ollama or direct | Nomic-embed-text | ~1-2GB |
|
|
| Reranker | Python + PyTorch | BGE Reranker | ~2-4GB |
|
|
|
|
### Ollama Configuration
|
|
|
|
```bash
|
|
# Pull Qwen 2.5
|
|
ollama pull qwen2.5:14b # or 7b for lower VRAM
|
|
|
|
# API endpoint
|
|
OLLAMA_HOST=0.0.0.0:11434 ollama serve
|
|
```
|
|
|
|
**Qwen 2.5 model selection:**
|
|
- `qwen2.5:7b` — fits comfortably, fast inference
|
|
- `qwen2.5:14b` — better quality, still fits in 24GB
|
|
- `qwen2.5:32b` — would require quantization or exceed VRAM
|
|
|
|
Recommendation: Start with `qwen2.5:14b` for balance of quality and speed.
|
|
|
|
### Nomic Embedding
|
|
|
|
Option 1: Via Ollama
|
|
```bash
|
|
ollama pull nomic-embed-text
|
|
```
|
|
|
|
Option 2: Direct with sentence-transformers (faster for batch)
|
|
```python
|
|
from sentence_transformers import SentenceTransformer
|
|
model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5')
|
|
```
|
|
|
|
### BGE Reranker
|
|
|
|
Run directly with PyTorch:
|
|
```python
|
|
from sentence_transformers import CrossEncoder
|
|
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')
|
|
```
|
|
|
|
### VRAM Allocation (24GB Total)
|
|
|
|
| Component | VRAM | Notes |
|
|
|-----------|------|-------|
|
|
| Qwen 2.5 14b | ~14GB | Loaded once, kept warm |
|
|
| BGE Reranker | ~3GB | Loaded on demand or kept warm |
|
|
| Nomic Embedding | ~1GB | Can share with reranker |
|
|
| Overhead | ~2GB | CUDA, framework |
|
|
| **Total** | ~20GB | Fits within 24GB with headroom |
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
- **Single GPU** — all local inference fits on one RTX 3090
|
|
- **Ollama simplicity** — easy model management and API
|
|
- **Low latency** — local inference, no network calls during blackouts
|
|
- **Bilingual support** — Qwen 2.5 and BGE-m3 handle Persian well
|
|
|
|
### Negative
|
|
- **Model limits** — can't run largest models (70b+) without quantization
|
|
- **Concurrent load** — heavy concurrent requests may queue
|
|
- **Power/heat** — GPU runs continuously under load
|
|
|
|
### Neutral
|
|
- May need to tune Ollama's `num_gpu` and `num_ctx` parameters
|
|
- Consider quantized models (GGUF) for efficiency
|
|
|
|
## Implementation Notes
|
|
|
|
- Run Ollama as systemd service for auto-restart
|
|
- Monitor GPU memory with `nvidia-smi`
|
|
- Consider GPU temperature monitoring and throttling
|
|
- Implement request queuing if concurrent load is high
|
|
- Test with expected load to validate VRAM allocation
|