Insurance RAG System — Architecture & Design

Architecture decision records (ADRs) and domain model for migrating an insurance company's RAG chatbot from ChromaDB to Qdrant.

Problem

The current system uses ~30 ChromaDB collections, one per insurance domain (car, health, fire, life, travel...). A collection classifier routes each query to exactly one collection before retrieval — but insurance domains are semantically similar, and misclassification produces irrelevant results with no recovery path. The classifier has no measurable accuracy, and its blind spots are unknown.

The goal is to redesign the retrieval architecture so the system can handle PDFs, Word docs, CSVs, Excel files, and text — not just CSV Q&A pairs — while remaining fully operational during internet blackouts of up to 90 days.

Constraints

Constraint	Implication
Internet blackouts (up to 90 days)	All components must have local fallbacks. Cloud APIs are optional enhancements, not dependencies.
Bilingual content (Persian + English)	Embedding models, rerankers, and LLMs must handle both languages.
Enterprise-scale, on-premises	No reliance on managed cloud services. Full data sovereignty.

Architecture at a Glance

User Query
  → Embed (OpenAI online / Nomic offline)
  → Hybrid Search (dense + BM25 sparse) in Qdrant
  → Rerank (Cohere online / BGE offline)
  → Top-N chunks + metadata
  → LLM (Qwen 2.5 local / cloud)
  → Answer + citations

Core decisions:

Area	Choice	Why
Collection strategy	Single collection + metadata filtering	Eliminates the classification failure mode entirely
Chunking	Hybrid: semantic for unstructured, row-by-row for tabular	Markdown-first pipeline, coherent chunks
Embeddings	Dual: OpenAI `text-embedding-3-large` + Nomic `nomic-embed-text`	Seamless online/offline switching at query time
Reranking	Dual: Cohere Rerank + BGE Reranker v2-m3	Cross-encoder precision, multilingual support
Vector DB	Qdrant (Docker, on-prem)	Filtered search, named vectors, sparse support
Search	Dense + sparse (BM25) with RRF fusion	Exact term matching + semantic similarity
Inference	Ollama + Qwen 2.5 on RTX 3090	Local bilingual LLM, ~20GB VRAM fits one GPU
Observability	Langfuse (self-hosted)	Full request tracing, cost tracking, quality scoring

Project Structure

chunking/
├── README.md                  ← You are here
├── CONTEXT.md                 ← Domain model, glossary, constraints
├── raw/
│   └── 001.md                 ← Original research request
└── docs/
    ├── README.md              ← ADR index with architecture overview
    └── adr/
        ├── 0001-single-collection-with-metadata.md
        ├── 0002-chunking-strategy.md
        ├── 0003-dual-embedding-strategy.md
        ├── 0004-reranking-strategy.md
        ├── 0005-retrieval-evaluation-framework.md
        ├── 0006-qdrant-deployment.md
        ├── 0007-hybrid-search.md
        ├── 0008-metadata-schema.md
        ├── 0009-ingestion-pipeline.md
        ├── 0010-query-pipeline.md
        ├── 0011-inference-infrastructure.md
        └── 0012-observability-auth.md

How to Read This

Start with CONTEXT.md — understand the domain, the glossary, and why the current architecture fails.
Browse docs/README.md — get the full architecture overview and a table of all 12 decisions.
Read individual ADRs as needed — each follows the Context / Decision / Consequences format with alternatives considered.

Tech Stack

Component	Tool	Deployment
Vector database	Qdrant	Docker, on-prem
Primary embeddings	OpenAI `text-embedding-3-large`	Cloud API
Offline embeddings	Nomic `nomic-embed-text`	Ollama / sentence-transformers
Primary reranker	Cohere Rerank	Cloud API
Offline reranker	BGE Reranker v2-m3	Local PyTorch
LLM (offline)	Qwen 2.5 (14B)	Ollama on RTX 3090
Hybrid search	BM25 sparse + dense vectors	Qdrant native
Observability	Langfuse	Self-hosted Docker
Ingestion	FastAPI + Celery + Redis	On-prem
Chunking	Markdown-first pipeline	Unstructured / pandoc

License

Internal research document. Not for distribution.

4.4 KiB Raw Blame History