93 lines
4.4 KiB
Markdown
93 lines
4.4 KiB
Markdown
# Insurance RAG System — Architecture & Design
|
|
|
|
Architecture decision records (ADRs) and domain model for migrating an insurance company's RAG chatbot from ChromaDB to Qdrant.
|
|
|
|
## Problem
|
|
|
|
The current system uses ~30 ChromaDB collections, one per insurance domain (car, health, fire, life, travel...). A collection classifier routes each query to exactly one collection before retrieval — but insurance domains are semantically similar, and misclassification produces irrelevant results with no recovery path. The classifier has no measurable accuracy, and its blind spots are unknown.
|
|
|
|
The goal is to redesign the retrieval architecture so the system can handle PDFs, Word docs, CSVs, Excel files, and text — not just CSV Q&A pairs — while remaining fully operational during internet blackouts of up to 90 days.
|
|
|
|
## Constraints
|
|
|
|
| Constraint | Implication |
|
|
|------------|-------------|
|
|
| Internet blackouts (up to 90 days) | All components must have local fallbacks. Cloud APIs are optional enhancements, not dependencies. |
|
|
| Bilingual content (Persian + English) | Embedding models, rerankers, and LLMs must handle both languages. |
|
|
| Enterprise-scale, on-premises | No reliance on managed cloud services. Full data sovereignty. |
|
|
|
|
## Architecture at a Glance
|
|
|
|
```
|
|
User Query
|
|
→ Embed (OpenAI online / Nomic offline)
|
|
→ Hybrid Search (dense + BM25 sparse) in Qdrant
|
|
→ Rerank (Cohere online / BGE offline)
|
|
→ Top-N chunks + metadata
|
|
→ LLM (Qwen 2.5 local / cloud)
|
|
→ Answer + citations
|
|
```
|
|
|
|
**Core decisions:**
|
|
|
|
| Area | Choice | Why |
|
|
|------|--------|-----|
|
|
| Collection strategy | Single collection + metadata filtering | Eliminates the classification failure mode entirely |
|
|
| Chunking | Hybrid: semantic for unstructured, row-by-row for tabular | Markdown-first pipeline, coherent chunks |
|
|
| Embeddings | Dual: OpenAI `text-embedding-3-large` + Nomic `nomic-embed-text` | Seamless online/offline switching at query time |
|
|
| Reranking | Dual: Cohere Rerank + BGE Reranker v2-m3 | Cross-encoder precision, multilingual support |
|
|
| Vector DB | Qdrant (Docker, on-prem) | Filtered search, named vectors, sparse support |
|
|
| Search | Dense + sparse (BM25) with RRF fusion | Exact term matching + semantic similarity |
|
|
| Inference | Ollama + Qwen 2.5 on RTX 3090 | Local bilingual LLM, ~20GB VRAM fits one GPU |
|
|
| Observability | Langfuse (self-hosted) | Full request tracing, cost tracking, quality scoring |
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
chunking/
|
|
├── README.md ← You are here
|
|
├── CONTEXT.md ← Domain model, glossary, constraints
|
|
├── raw/
|
|
│ └── 001.md ← Original research request
|
|
└── docs/
|
|
├── README.md ← ADR index with architecture overview
|
|
└── adr/
|
|
├── 0001-single-collection-with-metadata.md
|
|
├── 0002-chunking-strategy.md
|
|
├── 0003-dual-embedding-strategy.md
|
|
├── 0004-reranking-strategy.md
|
|
├── 0005-retrieval-evaluation-framework.md
|
|
├── 0006-qdrant-deployment.md
|
|
├── 0007-hybrid-search.md
|
|
├── 0008-metadata-schema.md
|
|
├── 0009-ingestion-pipeline.md
|
|
├── 0010-query-pipeline.md
|
|
├── 0011-inference-infrastructure.md
|
|
└── 0012-observability-auth.md
|
|
```
|
|
|
|
## How to Read This
|
|
|
|
1. **Start with `CONTEXT.md`** — understand the domain, the glossary, and why the current architecture fails.
|
|
2. **Browse `docs/README.md`** — get the full architecture overview and a table of all 12 decisions.
|
|
3. **Read individual ADRs** as needed — each follows the Context / Decision / Consequences format with alternatives considered.
|
|
|
|
## Tech Stack
|
|
|
|
| Component | Tool | Deployment |
|
|
|-----------|------|------------|
|
|
| Vector database | Qdrant | Docker, on-prem |
|
|
| Primary embeddings | OpenAI `text-embedding-3-large` | Cloud API |
|
|
| Offline embeddings | Nomic `nomic-embed-text` | Ollama / sentence-transformers |
|
|
| Primary reranker | Cohere Rerank | Cloud API |
|
|
| Offline reranker | BGE Reranker v2-m3 | Local PyTorch |
|
|
| LLM (offline) | Qwen 2.5 (14B) | Ollama on RTX 3090 |
|
|
| Hybrid search | BM25 sparse + dense vectors | Qdrant native |
|
|
| Observability | Langfuse | Self-hosted Docker |
|
|
| Ingestion | FastAPI + Celery + Redis | On-prem |
|
|
| Chunking | Markdown-first pipeline | Unstructured / pandoc |
|
|
|
|
## License
|
|
|
|
Internal research document. Not for distribution.
|