# ADR 0011: Inference Infrastructure

## Status

Accepted

## Context

The system needs local inference for:
- **LLM generation:** Qwen 2.5 (bilingual Persian/English)
- **Local embeddings:** Nomic-embed-text
- **Local reranking:** BGE Reranker

Hardware available: NVIDIA RTX 3090 (24GB VRAM)

## Decision

### Local Inference Stack

| Component | Tool | Model | VRAM Usage |
|-----------|------|-------|------------|
| LLM | Ollama | Qwen 2.5 | ~12-16GB (depends on quantization) |
| Embeddings | Ollama or direct | Nomic-embed-text | ~1-2GB |
| Reranker | Python + PyTorch | BGE Reranker | ~2-4GB |

### Ollama Configuration

```bash
# Pull Qwen 2.5
ollama pull qwen2.5:14b  # or 7b for lower VRAM

# API endpoint
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```

**Qwen 2.5 model selection:**
- `qwen2.5:7b` — fits comfortably, fast inference
- `qwen2.5:14b` — better quality, still fits in 24GB
- `qwen2.5:32b` — would require quantization or exceed VRAM

Recommendation: Start with `qwen2.5:14b` for balance of quality and speed.

### Nomic Embedding

Option 1: Via Ollama
```bash
ollama pull nomic-embed-text
```

Option 2: Direct with sentence-transformers (faster for batch)
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5')
```

### BGE Reranker

Run directly with PyTorch:
```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')
```

### VRAM Allocation (24GB Total)

| Component | VRAM | Notes |
|-----------|------|-------|
| Qwen 2.5 14b | ~14GB | Loaded once, kept warm |
| BGE Reranker | ~3GB | Loaded on demand or kept warm |
| Nomic Embedding | ~1GB | Can share with reranker |
| Overhead | ~2GB | CUDA, framework |
| **Total** | ~20GB | Fits within 24GB with headroom |

## Consequences

### Positive
- **Single GPU** — all local inference fits on one RTX 3090
- **Ollama simplicity** — easy model management and API
- **Low latency** — local inference, no network calls during blackouts
- **Bilingual support** — Qwen 2.5 and BGE-m3 handle Persian well

### Negative
- **Model limits** — can't run largest models (70b+) without quantization
- **Concurrent load** — heavy concurrent requests may queue
- **Power/heat** — GPU runs continuously under load

### Neutral
- May need to tune Ollama's `num_gpu` and `num_ctx` parameters
- Consider quantized models (GGUF) for efficiency

## Implementation Notes

- Run Ollama as systemd service for auto-restart
- Monitor GPU memory with `nvidia-smi`
- Consider GPU temperature monitoring and throttling
- Implement request queuing if concurrent load is high
- Test with expected load to validate VRAM allocation