Research/chunking/docs/adr/0011-inference-infrastructure.md

ADR 0011: Inference Infrastructure

Status

Accepted

Context

The system needs local inference for:

• LLM generation: Qwen 2.5 (bilingual Persian/English)
• Local embeddings: Nomic-embed-text
• Local reranking: BGE Reranker

Hardware available: NVIDIA RTX 3090 (24GB VRAM)

Decision

Local Inference Stack

Component	Tool	Model	VRAM Usage
LLM	Ollama	Qwen 2.5	~12-16GB (depends on quantization)
Embeddings	Ollama or direct	Nomic-embed-text	~1-2GB
Reranker	Python + PyTorch	BGE Reranker	~2-4GB

Ollama Configuration

# Pull Qwen 2.5
ollama pull qwen2.5:14b  # or 7b for lower VRAM

# API endpoint
OLLAMA_HOST=0.0.0.0:11434 ollama serve

Qwen 2.5 model selection:

• qwen2.5:7b — fits comfortably, fast inference
• qwen2.5:14b — better quality, still fits in 24GB
• qwen2.5:32b — would require quantization or exceed VRAM

Recommendation: Start with qwen2.5:14b for balance of quality and speed.

Nomic Embedding

Option 1: Via Ollama

ollama pull nomic-embed-text

Option 2: Direct with sentence-transformers (faster for batch)

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5')

BGE Reranker

Run directly with PyTorch:

from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')

VRAM Allocation (24GB Total)

Component	VRAM	Notes
Qwen 2.5 14b	~14GB	Loaded once, kept warm
BGE Reranker	~3GB	Loaded on demand or kept warm
Nomic Embedding	~1GB	Can share with reranker
Overhead	~2GB	CUDA, framework
Total	~20GB	Fits within 24GB with headroom

Consequences

Positive

• Single GPU — all local inference fits on one RTX 3090
• Ollama simplicity — easy model management and API
• Low latency — local inference, no network calls during blackouts
• Bilingual support — Qwen 2.5 and BGE-m3 handle Persian well

Negative

• Model limits — can't run largest models (70b+) without quantization
• Concurrent load — heavy concurrent requests may queue
• Power/heat — GPU runs continuously under load

Neutral

• May need to tune Ollama's num_gpu and num_ctx parameters
• Consider quantized models (GGUF) for efficiency

Implementation Notes

• Run Ollama as systemd service for auto-restart
• Monitor GPU memory with nvidia-smi
• Consider GPU temperature monitoring and throttling
• Implement request queuing if concurrent load is high
• Test with expected load to validate VRAM allocation