Mahdi Bazrafshan
754da323ff
Why: - Need documentation for team onboarding and reference - Need technical details for strategy implementations - Need API reference for developers Changes: - README.md: Documentation index and quick start guide - api-reference.md: All 10 endpoints with examples - architecture.md: System structure and design decisions - configuration.md: All environment variables and parameters - data-flow.md: How data moves through the system - evaluation-metrics.md: How scoring works with weights - strategy-technical-details.md: Deep dive into each strategy's implementation
184 lines
7.3 KiB
Markdown
184 lines
7.3 KiB
Markdown
# Architecture Overview
|
||
|
||
System structure and design decisions for the RAG Chunking Benchmarker.
|
||
|
||
---
|
||
|
||
## System Overview
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ RAG Chunking Benchmarker │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
│ │ Documents │ │ Chunking │ │ Benchmarking│ │
|
||
│ │ Module │ │ Module │ │ Module │ │
|
||
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
||
│ │ │ │ │
|
||
│ ▼ ▼ ▼ │
|
||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||
│ │ Core Layer │ │
|
||
│ │ (config, models, exceptions, dependencies) │ │
|
||
│ └─────────────────────────────────────────────────────────┘ │
|
||
│ │ │ │ │
|
||
│ ▼ ▼ ▼ │
|
||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
│ │ SQLite │ │ Qdrant │ │ OpenAI │ │
|
||
│ │ (metadata) │ │ (vectors) │ │ (LLM/Emb) │ │
|
||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Module Structure
|
||
|
||
```
|
||
src/
|
||
├── core/ # Foundation layer
|
||
│ ├── config.py # Settings from .env
|
||
│ ├── models.py # Shared Pydantic models
|
||
│ ├── exceptions.py # Custom exceptions
|
||
│ └── dependencies.py # Client singletons
|
||
│
|
||
├── storage/ # Data persistence
|
||
│ ├── sqlite.py # Structured data (docs, queries, experiments)
|
||
│ └── qdrant.py # Vector embeddings
|
||
│
|
||
├── documents/ # Document handling
|
||
│ ├── parser.py # .docx/.doc → DocumentTree + markdown
|
||
│ ├── service.py # Upload, process, delete orchestration
|
||
│ ├── routes.py # REST API endpoints
|
||
│ └── models.py # Request/response schemas
|
||
│
|
||
├── chunking/ # Chunking strategies
|
||
│ ├── base.py # Abstract base class + utilities
|
||
│ ├── embedding.py # OpenAI embedding service
|
||
│ ├── service.py # Orchestration (chunk → embed → store)
|
||
│ └── strategies/ # Strategy implementations
|
||
│ ├── fixed_size.py
|
||
│ ├── recursive.py
|
||
│ ├── semantic.py
|
||
│ ├── contextual_retrieval.py
|
||
│ └── semantic_parent_child.py
|
||
│
|
||
├── benchmarking/ # Evaluation system
|
||
│ ├── evaluation.py # LLM-as-Judge scoring
|
||
│ ├── benchmark_service.py # Orchestration
|
||
│ ├── query_service.py # Single query pipeline
|
||
│ ├── routes.py # REST API endpoints
|
||
│ ├── models.py # Request/response schemas
|
||
│ └── report.py # HTML report generation
|
||
│
|
||
└── main.py # FastAPI app factory
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
### Document Processing
|
||
|
||
```
|
||
Upload .docx → Parser → DocumentTree + Markdown → SQLite
|
||
↓
|
||
Process → Strategy.chunk() → Chunks → Embed → Qdrant
|
||
```
|
||
|
||
### Query Pipeline
|
||
|
||
```
|
||
Question → Embed → Qdrant Search → Top-K Chunks → LLM → Answer
|
||
```
|
||
|
||
### Benchmark Pipeline
|
||
|
||
```
|
||
Questions JSON → For each question × strategy:
|
||
├── Query Pipeline
|
||
└── LLM-as-Judge Evaluation
|
||
↓
|
||
Aggregate Metrics → SQLite → HTML Report
|
||
```
|
||
|
||
---
|
||
|
||
## Key Design Decisions
|
||
|
||
| Decision | Rationale |
|
||
|----------|-----------|
|
||
| **One collection per strategy** | Enables independent comparison |
|
||
| **SQLite for metadata** | Simple, no setup, sufficient scale |
|
||
| **Qdrant for vectors** | Purpose-built for vector search |
|
||
| **OpenAI for embeddings** | Industry standard, consistent results |
|
||
| **LLM-as-Judge for evaluation** | Automated, consistent scoring |
|
||
| **Per-strategy failure isolation** | One strategy failure doesn't break others |
|
||
| **Text stored in Qdrant payload** | Enables retrieval without extra DB lookups |
|
||
|
||
---
|
||
|
||
## Technology Stack
|
||
|
||
| Component | Technology | Version |
|
||
|-----------|------------|---------|
|
||
| Framework | FastAPI | 0.100+ |
|
||
| ORM/DB | SQLite | built-in |
|
||
| Vector DB | Qdrant | 1.18+ |
|
||
| Embeddings | OpenAI text-embedding-3-small | - |
|
||
| LLM | OpenAI gpt-4o-mini | - |
|
||
| Tokenizer | tiktoken (cl100k_base) | - |
|
||
| PDF/DOCX | python-docx, LibreOffice | - |
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All configuration via environment variables (`.env`):
|
||
|
||
```env
|
||
# Database
|
||
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
|
||
QDRANT_URL=http://localhost:6333
|
||
QDRANT_API_KEY=
|
||
|
||
# OpenAI
|
||
OPENAI_API_KEY=sk-...
|
||
|
||
# Chunking
|
||
CHUNK_SIZE=512
|
||
CHUNK_OVERLAP=50
|
||
SEMANTIC_THRESHOLD=0.5
|
||
SEMANTIC_MIN_CHUNK_SIZE=5
|
||
|
||
# Models
|
||
EMBEDDING_MODEL=text-embedding-3-small
|
||
LLM_MODEL=gpt-4o-mini
|
||
```
|
||
|
||
---
|
||
|
||
## Error Handling
|
||
|
||
| Exception | Handler | HTTP Status |
|
||
|-----------|---------|-------------|
|
||
| ChunkingError | chunking_exception_handler | 400 |
|
||
| BenchmarkError | benchmark_exception_handler | 400 |
|
||
| QueryError | benchmark_exception_handler | 400 |
|
||
| DocumentProcessingError | chunking_exception_handler | 400 |
|
||
| QdrantError | chunking_exception_handler | 400 |
|
||
| EmbeddingError | chunking_exception_handler | 400 |
|
||
|
||
---
|
||
|
||
## Scaling Considerations
|
||
|
||
| Component | Current | Production Recommendation |
|
||
|-----------|---------|---------------------------|
|
||
| SQLite | Single file | PostgreSQL for concurrent access |
|
||
| Qdrant | Single instance | Qdrant Cloud or cluster |
|
||
| OpenAI | Direct API | Connection pooling, rate limiting |
|
||
| Workers | Single process | Celery for async processing |
|
||
| Caching | None | Redis for repeated queries |
|