mba/Research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(rag): add domain model and original research input

Browse Source
This commit is contained in:
Mahdi Bazrafshan
2026-07-21 10:10:06 +03:30
parent 6c9cebaaa0
commit 642aec93f2
15 changed files with 1147 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
chunking/CONTEXT.md Normal file
View File

@@ -0,0 +1,54 @@
# Domain Model: Insurance RAG System
## Glossary
### Collection
A logical partition of documents in the vector database. In the current ChromaDB architecture, documents are spread across ~30 collections, each representing an insurance domain or document category. **Pain point:** The system must classify incoming queries to a single collection before retrieval; misclassification leads to irrelevant results.
### Collection Classifier
The first stage of the retrieval pipeline. Responsible for routing an incoming question to exactly one collection. Current accuracy is unknown but degrades on ambiguous queries and edge cases where insurance domains overlap (e.g., a car accident claim question could belong to "car insurance" or "claims processing").
**Current implementation:** Hybrid keyword-first cascade:
1. Keyword match found → boost similarity score to 98% → route to that collection
2. No keyword match → rely on base embedding similarity + LLM assistance
3. When signals conflict → embedding similarity score dominates
**Blind spots:**
- No metrics on keyword coverage (% of queries with identifiable domain keywords)
- No metrics on classification accuracy
- Similarity score can be misleading for semantically similar domains
### Insurance Domain
A category of insurance business: car, health, fire, life, travel, etc. Domains are semantically similar — many questions could plausibly belong to multiple domains, making classification non-trivial.
### Retrieval Pipeline
Two-stage process:
1. **Collection Classifier** → routes query to one collection
2. **Vector Search** → retrieves relevant chunks within that collection
Current bottleneck: Stage 1 errors are unrecoverable.
## Constraints
### Infrastructure Instability
Iran's internet connectivity is subject to government-imposed blackouts lasting up to 90 days. The system must operate fully offline with zero external API dependencies during these periods.
**Implication:** All components (embedding, LLM, vector DB) must have local fallbacks. Cloud APIs are optional enhancements, not dependencies.
### Bilingual Content
Documents and queries are in Persian (Farsi) and English. Embedding models and LLMs must handle both languages effectively.
## Architecture Decisions
1. **Single collection with metadata filtering** (ADR 0001) — eliminates collection classification problem
2. **Hybrid chunking** (ADR 0002) — semantic for unstructured, row-by-row for tabular, markdown-first
3. **Dual embeddings** (ADR 0003) — OpenAI + Nomic for online/offline resilience
4. **Dual reranking** (ADR 0004) — Cohere + BGE for precision
5. **Evaluation framework** (ADR 0005) — test set + metrics for quality measurement
6. **Qdrant Docker deployment** (ADR 0006) — fully on-prem vector database
7. **Hybrid search** (ADR 0007) — dense + sparse (BM25) for exact term matching
8. **Metadata schema** (ADR 0008) — domain, document_type, language, source, plus tabular extensions
9. **API-based continuous ingestion** (ADR 0009) — real-time document processing
10. **Full-context query pipeline** (ADR 0010) — chunks + metadata + history, citations for admin layer
11. **Ollama inference on RTX 3090** (ADR 0011) — Qwen 2.5 + BGE + Nomic
12. **Langfuse observability + API key auth** (ADR 0012) — full tracing and access control

49
chunking/docs/README.md Normal file
View File

@@ -0,0 +1,49 @@
# Insurance RAG System Documentation
This directory contains architecture decision records (ADRs) for the Insurance RAG System migration from ChromaDB to Qdrant.
## Quick Links
| ADR | Title | Status |
|-----|-------|--------|
| [0001](adr/0001-single-collection-with-metadata.md) | Single Collection with Metadata Filtering | Accepted |
| [0002](adr/0002-chunking-strategy.md) | Hybrid Chunking Strategy | Accepted |
| [0003](adr/0003-dual-embedding-strategy.md) | Dual Embedding Model Strategy | Accepted |
| [0004](adr/0004-reranking-strategy.md) | Dual Reranking Strategy | Accepted |
| [0005](adr/0005-retrieval-evaluation-framework.md) | Retrieval Evaluation Framework | Accepted |
| [0006](adr/0006-qdrant-deployment.md) | Qdrant Docker Deployment | Accepted |
| [0007](adr/0007-hybrid-search.md) | Hybrid Search (Dense + Sparse) | Accepted |
| [0008](adr/0008-metadata-schema.md) | Metadata Schema | Accepted |
| [0009](adr/0009-ingestion-pipeline.md) | API-Based Continuous Ingestion | Accepted |
| [0010](adr/0010-query-pipeline.md) | Full-Context Query Pipeline | Accepted |
| [0011](adr/0011-inference-infrastructure.md) | Ollama Inference on RTX 3090 | Accepted |
| [0012](adr/0012-observability-auth.md) | Langfuse Observability + API Key Auth | Accepted |
## Architecture Overview
### Problem Context
The current system uses ~30 collections in ChromaDB, requiring query classification before retrieval. Misclassification leads to irrelevant results with no recovery path. Insurance domains are semantically similar, making classification non-trivial.
### Key Constraints
1. **Infrastructure Instability** — Iran's internet blackouts (up to 90 days) require fully offline capability
2. **Bilingual Content** — Persian (Farsi) and English documents and queries
### Core Decisions
| Decision Point | Choice | Rationale |
|----------------|--------|-----------|
| Collection Strategy | Single collection + metadata filtering | Eliminates classification failure mode |
| Chunking | Hybrid: semantic for unstructured, row-by-row for tabular | Markdown-first pipeline |
| Embeddings | Dual: OpenAI (online) + Nomic (offline) | Resilience during blackouts |
| Reranking | Dual: Cohere (online) + BGE (offline) | Precision across domains |
| Vector DB | Qdrant Docker | Fully on-prem, excellent filtered search |
| Search | Hybrid: dense + sparse (BM25) | Exact term matching + semantic similarity |
| Inference | Ollama + Qwen 2.5 on RTX 3090 | Local bilingual LLM |
| Observability | Langfuse | Full tracing and access control |
## Related Documents
- [Domain Model (CONTEXT.md)](../CONTEXT.md) — Glossary, constraints, and architecture decisions summary
- [Original Grilling Input](../raw/001.md) — Research request that drove these decisions

52
chunking/docs/adr/0001-single-collection-with-metadata.md Normal file
View File

@@ -0,0 +1,52 @@
# ADR 0001: Single Collection with Metadata Filtering
## Status
Accepted
## Context
The current architecture uses ~30 collections in ChromaDB, each representing an insurance domain. The system must classify incoming queries to a single collection before retrieval. Misclassification leads to irrelevant results with no recovery path.
The current hybrid classifier (keyword-first cascade with similarity + LLM) has:
- No measurable accuracy metrics
- No visibility into keyword coverage
- Degraded performance on ambiguous queries
- Similarity score domination when signals conflict
Insurance domains are semantically similar — many questions could plausibly belong to multiple domains (e.g., "claim denial appeal" could be car, health, fire, etc.).
## Decision
Migrate to a **single collection** in Qdrant with rich metadata on each chunk:
- `domain`: car | health | fire | life | travel | ...
- `document_type`: policy | claim | faq | terms | ...
- `language`: fa | en
- `source_file`: filename
- `chunk_index`: position in source document
Queries will use metadata filtering to narrow search space when domain is known, or search the entire collection when ambiguous.
## Consequences
### Positive
- **Eliminates classification failure mode** — no single point of failure at the routing stage
- **Graceful degradation** — ambiguous queries search broader, still retrieve relevant results
- **Simpler pipeline** — one collection to manage, one index to optimize
- **Qdrant excels at filtered search** — metadata filtering happens before vector search, maintaining performance
- **Easier to add new domains** — just add documents with new domain metadata, no collection creation
### Negative
- **Larger index to search** — all domains in one collection
- **Requires disciplined metadata** — incorrect or missing metadata will cause retrieval failures
- **Potential for cross-domain noise** — irrelevant domains could appear in results if filtering isn't precise
### Neutral
- Reranking becomes more important to sort results across domains
- Embedding quality becomes more critical — must distinguish semantically similar domains
## Alternatives Considered
1. **Keep 30 collections + improve classifier** — adds metrics but retains single point of failure
2. **Hybrid: few collections (3-5) + metadata** — reduces but doesn't eliminate classification problem
3. **Multi-collection search with reranking** — more robust but slower and more complex

62
chunking/docs/adr/0002-chunking-strategy.md Normal file
View File

@@ -0,0 +1,62 @@
# ADR 0002: Chunking Strategy
## Status
Accepted
## Context
The current system chunks CSV files row-by-row, treating each Q&A pair as a single chunk. The migration expands support to PDF, Word, Excel, and text files with predominantly unstructured content (FAQs, guidance, mixed text).
Chunk quality directly impacts retrieval precision. Arbitrary splits (mid-sentence, mid-thought) produce incoherent chunks that degrade both embedding quality and answer generation.
## Decision
Adopt a **hybrid chunking strategy** with markdown as the intermediate format:
### 1. Markdown-First Pipeline
All documents are converted to markdown before chunking. This provides:
- Consistent processing regardless of source format
- Preserved structure (headers, lists, tables)
- Easier debugging and inspection
### 2. Type-Specific Chunking
| Source Type | Chunking Method | Rationale |
|-------------|-----------------|-----------|
| CSV | Row-by-row | Already natural Q&A pairs |
| Excel | Row-by-row | Preserves tabular context per record |
| PDF | Convert to markdown → semantic chunking | Unstructured prose needs coherent boundaries |
| Word (.docx) | Convert to markdown → semantic chunking | Same as PDF |
| Text files | Convert to markdown → semantic chunking | Same as PDF |
| Structured documents (policies with headers) | Markdown section-aware chunking | Headers define natural boundaries |
### 3. Semantic Chunking Parameters
- **Unit:** Sentences (not tokens)
- **Grouping:** Consecutive sentences grouped until semantic similarity drops below threshold
- **Overlap:** 1-2 sentences at boundaries to preserve context
- **Max chunk size:** Determined by embedding model limits (typically 512-8192 tokens depending on model)
## Consequences
### Positive
- **Coherent chunks** — complete thoughts, no mid-sentence splits
- **Consistent pipeline** — one chunking logic for all unstructured content
- **Debuggable** — markdown is human-readable, easy to inspect chunk quality
- **Flexible** — can adjust semantic threshold per document type if needed
### Negative
- **Processing overhead** — conversion to markdown adds a step
- **Variable chunk sizes** — harder to predict storage and latency
- **Semantic chunking requires embedding model** — adds computation during ingestion
### Neutral
- Need to select semantic similarity threshold (tuning required)
- Need to handle edge cases: very long sentences, tables in markdown, code blocks
## Implementation Notes
- Use `unstructured` or `pandoc` for document → markdown conversion
- Use LlamaIndex's `SemanticSplitter` or custom implementation with embedding model
- Consider `MarkdownElementNodeParser` for section-aware chunking when headers are present
- Store original source file path and chunk index in metadata for citation

66
chunking/docs/adr/0003-dual-embedding-strategy.md Normal file
View File

@@ -0,0 +1,66 @@
# ADR 0003: Dual Embedding Model Strategy
## Status
Accepted
## Context
The system operates in an environment with intermittent internet connectivity, including government-imposed blackouts lasting up to 90 days. The retrieval system must function fully offline with zero external API dependencies during these periods.
Documents and queries are bilingual (Persian/Farsi and English). Embedding models must handle both languages effectively.
## Decision
Store **two embeddings per chunk** in Qdrant using named vectors:
| Model | Use Case | Type |
|-------|----------|------|
| `text-embedding-3-large` (OpenAI) | Primary retrieval when internet available | Cloud |
| `nomic-embed-text` | Fallback during blackouts, fully local | On-prem |
### Qdrant Configuration
```python
# Named vectors in Qdrant collection
collection_config = {
"vectors": {
"openai": {"size": 3072, "distance": "Cosine"},
"nomic": {"size": 768, "distance": "Cosine"}
}
}
```
### Query-Time Behavior
- **Online:** Query with OpenAI embeddings, retrieve using `openai` vector
- **Offline:** Query with nomic embeddings, retrieve using `nomic` vector
- **A/B testing:** Can compare retrieval quality between models in real-time
## Consequences
### Positive
- **Resilience** — system works identically online and offline
- **No re-indexing** — switch models at query time, not ingestion time
- **A/B testing** — can measure retrieval quality difference between models
- **Graceful degradation** — fallback is automatic when cloud APIs unavailable
### Negative
- **2x storage** — two vectors per chunk
- **2x ingestion time** — must embed with both models during indexing
- **Complexity** — query logic must select correct vector at runtime
### Neutral
- Requires monitoring to detect when cloud APIs become unavailable
- May need to tune retrieval parameters (top_k, score threshold) per model
## Generation Layer
During offline periods, answer generation uses **Qwen 2.5** running locally. This provides:
- Bilingual support (Persian/English)
- Fully on-prem inference
- No external API dependencies
## Alternatives Considered
1. **Single model (nomic only)** — simpler but loses quality of `text-embedding-3-large` when online
2. **Single model (multilingual-e5)** — good bilingual support but not tested in production
3. **Re-embed on switch** — would require re-indexing entire corpus during blackout (impractical)

59
chunking/docs/adr/0004-reranking-strategy.md Normal file
View File

@@ -0,0 +1,59 @@
# ADR 0004: Reranking Strategy
## Status
Accepted
## Context
The current system has no reranking step. With a single collection spanning all insurance domains, initial retrieval may return chunks from multiple domains with varying relevance. A reranker can reorder results by relevance, pushing the most domain-appropriate chunks to the top.
Reranking is critical for retrieval precision when:
- Query is ambiguous (could match multiple domains)
- Initial embedding search returns noisy results
- Cross-domain similarity causes false positives
## Decision
Add a **dual reranking layer** matching the dual embedding strategy:
| Model | Use Case | Type |
|-------|----------|------|
| Cohere Rerank | Primary when internet available | Cloud API |
| BGE Reranker (`BAAI/bge-reranker-base` or `bge-reranker-v2-m3`) | Fallback during blackouts | On-prem |
### Retrieval Pipeline (Updated)
1. **Query embedding** — OpenAI or nomic depending on connectivity
2. **Initial retrieval** — Top-K chunks (K=20-50) from Qdrant
3. **Reranking** — Score and reorder chunks by relevance
4. **Top-N selection** — Return top N (N=5-10) for generation
### Why BGE Reranker for Local
- `bge-reranker-v2-m3` supports multilingual (Persian/English)
- Cross-encoder architecture provides high accuracy
- Runs on CPU if needed (slower) or GPU for production speed
## Consequences
### Positive
- **Higher precision** — reranker separates relevant from superficially similar
- **Domain disambiguation** — pushes correct domain to top even with noisy initial retrieval
- **Consistent architecture** — same dual-model pattern as embeddings and generation
- **Graceful degradation** — local reranker works during blackouts
### Negative
- **Added latency** — reranking adds 50-200ms depending on model and hardware
- **Additional complexity** — one more model to manage and monitor
- **GPU recommended** — BGE reranker is slow on CPU for large candidate sets
### Neutral
- Need to tune K (initial retrieval size) and N (final output size)
- May need different thresholds for online vs offline reranking quality
## Implementation Notes
- Use Cohere's `/rerank` API when available
- Use `FlagEmbedding` or `sentence-transformers` for BGE locally
- Consider `bge-reranker-v2-m3` for best multilingual support
- Start with K=20 initial candidates, N=5 final results; tune based on metrics

86
chunking/docs/adr/0005-retrieval-evaluation-framework.md Normal file
View File

@@ -0,0 +1,86 @@
# ADR 0005: Retrieval Evaluation Framework
## Status
Accepted
## Context
The current system has no metrics for retrieval quality. The previous classification accuracy was unknown, making improvement impossible. The new architecture needs measurement from day one to:
1. Establish baseline performance
2. Compare embedding models (OpenAI vs nomic)
3. Compare reranking impact
4. Detect regressions after changes
5. Tune hyperparameters (K, N, thresholds)
## Decision
Implement a **multi-source test set** combining:
### 1. Historical Query Labeling
- Sample real queries from production logs
- Manually label with correct chunk IDs and domain
- Focus on edge cases and ambiguous queries
- Target: ~50-100 labeled queries
### 2. Synthetic Q&A Generation
- Use LLM to generate questions from each chunk
- Question + source chunk = labeled test pair
- Ensures coverage across all domains and document types
- Target: ~100-200 synthetic pairs
### 3. LLM Auto-Labeling with Human Review
- Use LLM to judge relevance of retrieved chunks
- Flag low-confidence or disputed labels for human review
- Accelerates labeling while maintaining quality
- Target: ~50-100 auto-labeled with review
### Metrics to Track
| Metric | Definition | Target |
|--------|------------|--------|
| Precision@5 | Fraction of top 5 results that are relevant | > 0.80 |
| Precision@10 | Fraction of top 10 results that are relevant | > 0.70 |
| MRR | Mean Reciprocal Rank of first relevant result | > 0.85 |
| Recall@20 | Fraction of relevant docs found in top 20 | > 0.90 |
| Latency p50 | Median retrieval time | < 500ms |
| Latency p95 | 95th percentile retrieval time | < 1000ms |
| Domain Accuracy | Correct domain in top 5 results | > 0.95 |
### Evaluation Pipeline
```
Test Query → Embed → Retrieve (K=20) → Rerank → Top N → Compare to Labels → Log Metrics
```
### Comparison Modes
1. **Online vs Offline** — Compare OpenAI + Cohere vs Nomic + BGE
2. **With vs Without Reranking** — Measure reranking impact
3. **Before vs After Changes** — Detect regressions
## Consequences
### Positive
- **Measurable improvement** — quantifiable baseline and progress
- **Model comparison** — data-driven decision on embedding/reranking choices
- **Regression detection** — catch quality degradation early
- **Hyperparameter tuning** — optimize K, N, thresholds with data
### Negative
- **Upfront investment** — labeling requires human effort
- **Maintenance** — test set needs updates as corpus grows
- **Storage** — need to store labels and evaluation results
### Neutral
- Synthetic questions may not match real query distribution
- LLM labeling may have bias — human review essential for edge cases
## Implementation Notes
- Store test set in JSON/Parquet: `{query, relevant_chunk_ids, domain, source}`
- Build evaluation script that runs retrieval and computes metrics
- Log results to a simple database or file for trend analysis
- Run evaluation on every significant change
- Consider integrating with LlamaIndex's evaluation module

85
chunking/docs/adr/0006-qdrant-deployment.md Normal file
View File

@@ -0,0 +1,85 @@
# ADR 0006: Qdrant Deployment
## Status
Accepted
## Context
The system is migrating from ChromaDB to Qdrant. Deployment must work fully on-premises with no external dependencies, given the risk of prolonged internet blackouts (up to 90 days).
## Decision
Deploy **Qdrant as a Docker container** on-premises.
### Configuration
```yaml
# docker-compose.yml
services:
qdrant:
image: qdrant/qdrant:latest
ports:
- "6333:6333" # REST API
- "6334:6334" # gRPC API
volumes:
- ./qdrant_storage:/qdrant/storage
environment:
- QDRANT__LOG_LEVEL=INFO
restart: unless-stopped
```
### Why Docker
- **Isolation** — consistent environment, no dependency conflicts
- **Portability** — easy to move between servers
- **Simplicity** — single container, no orchestration complexity
- **Persistence** — volume mount ensures data survives container restarts
- **No external dependencies** — fully on-prem
### Collection Configuration
Single collection with named vectors:
```python
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
client = QdrantClient(host="localhost", port=6333)
client.create_collection(
collection_name="insurance_docs",
vectors_config={
"openai": VectorParams(size=3072, distance=Distance.COSINE),
"nomic": VectorParams(size=768, distance=Distance.COSINE),
}
)
```
## Consequences
### Positive
- **Fully on-prem** — works during internet blackouts
- **Simple operations** — single container to manage
- **Fast access** — no network latency to external services
- **Data sovereignty** — all data stays in your infrastructure
### Negative
- **Manual backups** — must implement backup strategy
- **Single point of failure** — no built-in replication (consider Qdrant cluster if HA needed)
- **Hardware dependent** — performance tied to host resources
### Neutral
- Storage requirements depend on corpus size and vector dimensions
- May need to tune Qdrant's HNSW parameters for optimal recall/speed trade-off
- Consider resource limits in Docker for predictable performance
## Backup Strategy
- Regular snapshots: `qdrant-cli snapshot create`
- Volume backup: backup `./qdrant_storage` directory
- Test restore procedure before production deployment
## Alternatives Considered
1. **Qdrant Cloud** — requires internet, not viable
2. **Kubernetes deployment** — adds complexity, not needed for single-node setup
3. **Standalone binary** — Docker provides better isolation and process management

100
chunking/docs/adr/0007-hybrid-search.md Normal file
View File

@@ -0,0 +1,100 @@
# ADR 0007: Hybrid Search (Dense + Sparse)
## Status
Accepted
## Context
Insurance queries often contain specific terms that require exact matching:
- Policy numbers and IDs
- Coverage names (e.g., "comprehensive", "third-party")
- Legal phrases and terminology
- Product codes and identifiers
Pure dense vector search excels at semantic similarity but can dilute exact term matches. A query for "policy AC-12345" should match documents containing that exact string, not just semantically similar policy-related content.
## Decision
Implement **hybrid search** combining dense and sparse vectors:
### Dense Vectors (Semantic)
- `text-embedding-3-large` (online) or `nomic-embed-text` (offline)
- Captures semantic meaning and intent
### Sparse Vectors (Lexical)
- **BM25** — classic term frequency scoring, fast, no model needed
- Alternative: **SPLADE** — learned sparse embeddings with term expansion
### Qdrant Configuration
```python
from qdrant_client.models import VectorParams, SparseVectorParams
client.create_collection(
collection_name="insurance_docs",
vectors_config={
"openai": VectorParams(size=3072, distance=Distance.COSINE),
"nomic": VectorParams(size=768, distance=Distance.COSINE),
},
sparse_vectors_config={
"bm25": SparseVectorParams(
modifier=models.Modifier.IDF, # Use IDF weighting
)
}
)
```
### Query Fusion
Use **Reciprocal Rank Fusion (RRF)** to combine dense and sparse results:
```python
def hybrid_search(query, top_k=20):
# Dense search
dense_results = client.search(
collection_name="insurance_docs",
query_vector=("openai", dense_embedding),
limit=top_k
)
# Sparse search
sparse_results = client.search(
collection_name="insurance_docs",
query_vector=("bm25", sparse_vector),
limit=top_k
)
# RRF fusion
return reciprocal_rank_fusion(dense_results, sparse_results)
```
### Why BM25 over SPLADE
- No model required — simpler deployment
- Faster indexing and query
- Works offline without additional dependencies
- SPLADE could be added later if term expansion proves valuable
## Consequences
### Positive
- **Exact term matching** — policy numbers, IDs, specific terminology
- **Semantic + lexical** — best of both worlds
- **Improved precision** — especially for queries with specific terms
- **No additional model** — BM25 is built into Qdrant
### Negative
- **Added complexity** — must maintain and query two vector types
- **Fusion tuning** — RRF parameters may need adjustment
- **Storage overhead** — sparse vectors add storage (typically smaller than dense)
### Neutral
- Need to implement RRF or similar fusion logic
- May need different fusion weights for different query types
## Implementation Notes
- Use Qdrant's built-in BM25 sparse indexing
- Implement RRF with tunable `k` parameter (default k=60)
- Log dense vs sparse contributions to final ranking for debugging
- Consider query routing: queries with numbers/IDs → boost sparse weight

81
chunking/docs/adr/0008-metadata-schema.md Normal file
View File

@@ -0,0 +1,81 @@
# ADR 0008: Chunk Metadata Schema
## Status
Accepted
## Context
Each chunk in Qdrant needs metadata for filtering, citation, and debugging. The schema must balance flexibility with consistency, and handle both document-based (PDF, Word) and tabular (CSV, Excel) sources.
## Decision
### Core Metadata Schema
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `domain` | string | Yes | Insurance domain: `car`, `health`, `fire`, `life`, `travel`, etc. |
| `document_type` | string | Yes | Document category: `policy`, `faq`, `claim`, `terms`, `guidance`, etc. |
| `language` | string | Yes | Language code: `fa` (Persian), `en` (English) |
| `source_file` | string | Yes | Original filename with extension |
| `chunk_index` | int | Yes | Zero-based position in source document |
| `content_hash` | string | Yes | SHA-256 hash of chunk content |
| `created_at` | datetime | Yes | Ingestion timestamp |
### Tabular Data Extensions
For CSV and Excel sources, additional fields:
| Field | Type | Description |
|-------|------|-------------|
| `row_number` | int | Row index in original file |
| `column_names` | list[str] | Column headers for context |
| `original_format` | string | Source format: `csv`, `xlsx`, `xls` |
Example payload for a CSV row:
```json
{
"domain": "car",
"document_type": "faq",
"language": "fa",
"source_file": "car_insurance_faq.csv",
"chunk_index": 0,
"content_hash": "a1b2c3...",
"created_at": "2025-01-15T10:30:00Z",
"row_number": 1,
"column_names": ["question", "answer", "category"],
"original_format": "csv"
}
```
### Document Metadata (Optional, Future)
Reserved fields for future use:
- `policy_version`: string
- `effective_date`: date
- `department`: string
- `region`: string
## Consequences
### Positive
- **Consistent filtering** — all chunks have core fields for retrieval filtering
- **Citation support** — source file + chunk index enables precise citations
- **Deduplication** — content hash identifies duplicate content
- **Tabular context** — column names preserved for row-based chunks
### Negative
- **Manual tagging** — domain and document_type require human input or heuristics at ingestion
- **Storage overhead** — metadata stored with each chunk
### Neutral
- Schema is minimal but extensible
- Column names may vary across files — need normalization strategy
## Implementation Notes
- Define metadata validator at ingestion time
- Auto-detect language using `langdetect` or similar
- Derive domain from file path or manual mapping
- Compute content_hash before insertion
- Store `created_at` in ISO 8601 format for consistency

109
chunking/docs/adr/0009-ingestion-pipeline.md Normal file
View File

@@ -0,0 +1,109 @@
# ADR 0009: Ingestion Pipeline
## Status
Accepted
## Context
Documents arrive via API endpoints. The system must process them continuously as they're received, making new content immediately searchable. The pipeline handles multiple file formats (PDF, Word, CSV, Excel, TXT) and produces chunks with dual embeddings and sparse vectors.
## Decision
### Architecture
```
API Endpoint (/ingest)
↓
File Validator (type, size, format)
↓
Document Converter → Markdown
↓
Chunker (semantic / row-based by type)
↓
Metadata Enricher (domain, language, source, etc.)
↓
Dual Embedder (OpenAI + Nomic)
↓
Sparse Vectorizer (BM25)
↓
Qdrant Indexer
↓
Response (chunk IDs, status)
```
### Components
**1. API Endpoint**
- REST API receiving multipart file uploads
- Accepts: PDF, Word (.docx), CSV, Excel (.xlsx, .xls), TXT
- Returns: ingestion ID, chunk count, status
**2. File Validator**
- Check file type by extension and magic bytes
- Enforce size limits (e.g., 50MB max)
- Reject unsupported formats
**3. Document Converter**
- PDF → Markdown: `pandoc` or `pymupdf`
- Word → Markdown: `pandoc` or `python-docx`
- CSV/Excel → preserve as structured data, convert to markdown representation
- TXT → minimal processing, detect encoding
**4. Chunker**
- Unstructured (PDF, Word, TXT): Semantic chunking with sentence boundaries
- Tabular (CSV, Excel): Row-by-row chunking
- Preserve column names as metadata for tabular sources
**5. Metadata Enricher**
- Auto-detect language (`langdetect`)
- Domain from API parameter or file path
- Generate content hash (SHA-256)
- Add timestamps
**6. Dual Embedder**
- Always generate both OpenAI and Nomic embeddings
- Queue parallel API calls to OpenAI
- Run Nomic locally (on-prem)
- Handle OpenAI failures gracefully (continue with Nomic only)
**7. Sparse Vectorizer**
- Use Qdrant's built-in BM25 sparse indexing
- Tokenize text, generate sparse vector
**8. Qdrant Indexer**
- Upsert chunks with all vectors and metadata
- Return chunk IDs for confirmation
### Continuous Processing
- API endpoint queues documents for processing
- Background workers pick up documents and process asynchronously
- Status endpoint to check ingestion progress
- Webhook or callback when ingestion completes (optional)
## Consequences
### Positive
- **Immediate availability** — new docs searchable within seconds
- **Scalable** — background workers can scale horizontally
- **Resilient** — OpenAI failures don't block ingestion (fallback to Nomic)
- **Observable** — status tracking for each ingestion job
### Negative
- **Complexity** — async processing requires job queue and workers
- **Resource usage** — continuous embedding requires compute/GPU availability
- **Error handling** — partial failures need clear status reporting
### Neutral
- Need to implement job queue (Redis, Celery, or similar)
- Consider rate limiting to prevent overload
- May need to batch small files for efficiency
## Implementation Notes
- Use FastAPI for ingestion endpoint
- Use Celery + Redis for job queue
- Store ingestion logs and status in database
- Implement retry logic for transient failures
- Consider deduplication: check content_hash before inserting

132
chunking/docs/adr/0010-query-pipeline.md Normal file
View File

@@ -0,0 +1,132 @@
# ADR 0010: Query Pipeline
## Status
Accepted
## Context
The query pipeline transforms user questions into answers using the RAG architecture. It must handle:
- Dual embedding models (online/offline)
- Hybrid search (dense + sparse)
- Dual reranking (online/offline)
- Bilingual generation (Persian/English)
- Citation and traceability for admin monitoring
## Decision
### Architecture
```
User Query
↓
Query Embedder (OpenAI or Nomic)
↓
Hybrid Search (dense + sparse vectors)
↓
Reranker (Cohere or BGE)
↓
Top-N Chunks (N=5-10)
↓
Context Builder (chunks + metadata + history)
↓
LLM Generator (Qwen 2.5 or cloud LLM)
↓
Answer + Citations + Metadata
```
### Context for LLM
The LLM receives:
1. **Retrieved chunks** — text content of top-N results
2. **Metadata** — domain, source_file, document_type for each chunk
3. **Conversation history** — previous turns for multi-turn context
Prompt template:
```
You are an insurance assistant. Answer the user's question based on the provided context.
Context:
[Domain: {domain}] [Source: {source_file}]
{chunk_text}
...
Conversation history:
{history}
Question: {query}
Answer the question using only the provided context. Cite sources when relevant.
```
### Admin Layer Metadata
API response includes:
```json
{
"answer": "Your car insurance policy covers...",
"citations": [
{"source_file": "car_faq.pdf", "chunk_index": 5, "domain": "car"}
],
"retrieval_metadata": {
"embedding_model": "nomic",
"reranker": "bge",
"latency_ms": 450,
"total_chunks_retrieved": 20,
"chunks_after_rerank": 5
},
"confidence": "high"
}
```
### Query-Time Model Selection
```python
def get_query_pipeline():
if is_internet_available():
return {
"embedder": "openai",
"reranker": "cohere",
"llm": "cloud" # or Qwen if preferred
}
else:
return {
"embedder": "nomic",
"reranker": "bge",
"llm": "qwen"
}
```
### Citations in Response
LLM is instructed to cite sources:
- "According to `car_faq.pdf`, your policy covers..."
- "As stated in the `terms_and_conditions.pdf` document..."
Citations use `source_file` metadata, enabling admins to trace answers to exact documents.
## Consequences
### Positive
- **Full context** — LLM has all information needed for accurate answers
- **Traceability** — admin layer can audit which files informed each answer
- **Multi-turn support** — conversation history enables follow-up questions
- **Model flexibility** — automatic fallback during offline periods
### Negative
- **Token usage** — full context increases generation cost
- **Latency** — more context means longer LLM inference time
- **Context window limits** — may need to truncate for very long conversations
### Neutral
- Need to implement conversation history storage
- May need to tune how much history to include
## Implementation Notes
- Use LlamaIndex or LangChain for pipeline orchestration
- Store conversation history in Redis or database
- Implement rate limiting on query endpoint
- Log all retrieval metadata for analysis
- Consider streaming responses for better UX

97
chunking/docs/adr/0011-inference-infrastructure.md Normal file
View File

@@ -0,0 +1,97 @@
# ADR 0011: Inference Infrastructure
## Status
Accepted
## Context
The system needs local inference for:
- **LLM generation:** Qwen 2.5 (bilingual Persian/English)
- **Local embeddings:** Nomic-embed-text
- **Local reranking:** BGE Reranker
Hardware available: NVIDIA RTX 3090 (24GB VRAM)
## Decision
### Local Inference Stack
| Component | Tool | Model | VRAM Usage |
|-----------|------|-------|------------|
| LLM | Ollama | Qwen 2.5 | ~12-16GB (depends on quantization) |
| Embeddings | Ollama or direct | Nomic-embed-text | ~1-2GB |
| Reranker | Python + PyTorch | BGE Reranker | ~2-4GB |
### Ollama Configuration
```bash
# Pull Qwen 2.5
ollama pull qwen2.5:14b # or 7b for lower VRAM
# API endpoint
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
**Qwen 2.5 model selection:**
- `qwen2.5:7b` — fits comfortably, fast inference
- `qwen2.5:14b` — better quality, still fits in 24GB
- `qwen2.5:32b` — would require quantization or exceed VRAM
Recommendation: Start with `qwen2.5:14b` for balance of quality and speed.
### Nomic Embedding
Option 1: Via Ollama
```bash
ollama pull nomic-embed-text
```
Option 2: Direct with sentence-transformers (faster for batch)
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5')
```
### BGE Reranker
Run directly with PyTorch:
```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')
```
### VRAM Allocation (24GB Total)
| Component | VRAM | Notes |
|-----------|------|-------|
| Qwen 2.5 14b | ~14GB | Loaded once, kept warm |
| BGE Reranker | ~3GB | Loaded on demand or kept warm |
| Nomic Embedding | ~1GB | Can share with reranker |
| Overhead | ~2GB | CUDA, framework |
| **Total** | ~20GB | Fits within 24GB with headroom |
## Consequences
### Positive
- **Single GPU** — all local inference fits on one RTX 3090
- **Ollama simplicity** — easy model management and API
- **Low latency** — local inference, no network calls during blackouts
- **Bilingual support** — Qwen 2.5 and BGE-m3 handle Persian well
### Negative
- **Model limits** — can't run largest models (70b+) without quantization
- **Concurrent load** — heavy concurrent requests may queue
- **Power/heat** — GPU runs continuously under load
### Neutral
- May need to tune Ollama's `num_gpu` and `num_ctx` parameters
- Consider quantized models (GGUF) for efficiency
## Implementation Notes
- Run Ollama as systemd service for auto-restart
- Monitor GPU memory with `nvidia-smi`
- Consider GPU temperature monitoring and throttling
- Implement request queuing if concurrent load is high
- Test with expected load to validate VRAM allocation

114
chunking/docs/adr/0012-observability-auth.md Normal file
View File

@@ -0,0 +1,114 @@
# ADR 0012: Observability and Authentication
## Status
Accepted
## Context
The system requires:
- Full observability for retrieval and generation pipeline (tracing, metrics, debugging)
- Authentication for API access control
- Frontend handled separately — API backend only
## Decision
### Observability: Langfuse
**Langfuse** provides comprehensive LLM observability:
- Trace every request end-to-end
- Log retrieval steps (embedding, search, reranking)
- Log generation steps (prompt, context, completion)
- Track costs and latency
- Debug failed requests
- Evaluate quality with scores
**Integration:**
```python
from langfuse import Langfuse
from langfuse.decorators import observe
langfuse = Langfuse(
public_key="...",
secret_key="...",
host="http://localhost:3000" # Self-hosted
)
@observe
def query_pipeline(question: str):
# Traces entire pipeline automatically
...
```
**Deployment:** Self-hosted Langfuse via Docker (matches Qdrant deployment pattern).
### Tracked Metrics
| Metric | Level | Description |
|--------|-------|-------------|
| Query latency | Request | End-to-end response time |
| Embedding latency | Step | Time to embed query |
| Retrieval latency | Step | Vector search time |
| Reranking latency | Step | Rerank processing time |
| Generation latency | Step | LLM inference time |
| Token usage | Request | Input + output tokens |
| Model used | Request | Which embedding/reranker/LLM |
| Retrieval count | Request | Chunks retrieved before/after reranking |
| User feedback | Request | Thumbs up/down on answers |
### Authentication: API Key
Simple API key authentication for the ingestion and query endpoints:
```python
from fastapi import Security, HTTPException
from fastapi.security import APIKeyHeader
api_key_header = APIKeyHeader(name="X-API-Key")
async def verify_api_key(api_key: str = Security(api_key_header)):
if api_key not in VALID_API_KEYS:
raise HTTPException(status_code=401, detail="Invalid API key")
return api_key
```
**API Key Management:**
- Store hashed keys in database
- Generate keys via admin endpoint
- Revoke keys when needed
- Rate limit per key (optional)
### API Structure
```
POST /ingest - Upload documents (API key required)
GET /status/{job_id} - Check ingestion status (API key required)
POST /query - Ask question (API key required)
GET /health - Health check (no auth)
```
## Consequences
### Positive
- **Full traceability** — every request traced from query to answer
- **Debugging** — identify bottlenecks and failures quickly
- **Cost tracking** — monitor token usage and model costs
- **Simple auth** — API keys are easy to manage and rotate
- **Self-hosted** — Langfuse runs on-prem, no external dependency
### Negative
- **Storage overhead** — traces consume storage
- **Performance impact** — tracing adds minimal latency
- **Key management** — need secure key storage and rotation process
### Neutral
- Need to set trace retention policy (e.g., 30 days)
- May need to sample traces under high load
## Implementation Notes
- Deploy Langfuse with Docker Compose alongside Qdrant
- Use Langfuse's Python SDK decorators for automatic tracing
- Integrate user feedback collection (thumbs up/down)
- Set up dashboards for monitoring key metrics
- Implement API key rotation schedule

1
chunking/raw/001.md Normal file
View File

@@ -0,0 +1 @@
I have a rag application that now is just working with q&a csv. the app is now working with LlamaIndex and chromadb. we dont use reranking and our chunking strategy is per row in csv. we are insurance company and the rag app is responsible to answer insurance company that is very similar to each other in eahc insrance domain like car, health, fire and so on. so I emphasize very strongly on retrieval step so the app can find the most relevant and correct answer. your main task is focus on different chunking and embedding, reranking strategies. we dont need super reasoning. our most concern is about retrieval. our previous app has about 30 collections and our most problem is finding the right collection. I want to change this architecture and migrate from chromadb to qdrant and use all file types like pdf, .docs, csv, excel, .txt and so on. the research I want is: Research the key architecture decisions for our new AI chatbot (chunking, embeddings, reranking, RAG, vector DB, etc.) and prepare a concise decision report. For each topic, provide at least three options prioritized by recommendation, including a title, ranking, rationale, pros, cons, and a final recommendation based on an enterprise-scale, bilingual, on-premises deployment.