docs: add comprehensive project documentation
Why: - Need documentation for team onboarding and reference - Need technical details for strategy implementations - Need API reference for developers Changes: - README.md: Documentation index and quick start guide - api-reference.md: All 10 endpoints with examples - architecture.md: System structure and design decisions - configuration.md: All environment variables and parameters - data-flow.md: How data moves through the system - evaluation-metrics.md: How scoring works with weights - strategy-technical-details.md: Deep dive into each strategy's implementation
This commit is contained in:
183
docs/architecture.md
Normal file
183
docs/architecture.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Architecture Overview
|
||||
|
||||
System structure and design decisions for the RAG Chunking Benchmarker.
|
||||
|
||||
---
|
||||
|
||||
## System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ RAG Chunking Benchmarker │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Documents │ │ Chunking │ │ Benchmarking│ │
|
||||
│ │ Module │ │ Module │ │ Module │ │
|
||||
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Layer │ │
|
||||
│ │ (config, models, exceptions, dependencies) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ SQLite │ │ Qdrant │ │ OpenAI │ │
|
||||
│ │ (metadata) │ │ (vectors) │ │ (LLM/Emb) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── core/ # Foundation layer
|
||||
│ ├── config.py # Settings from .env
|
||||
│ ├── models.py # Shared Pydantic models
|
||||
│ ├── exceptions.py # Custom exceptions
|
||||
│ └── dependencies.py # Client singletons
|
||||
│
|
||||
├── storage/ # Data persistence
|
||||
│ ├── sqlite.py # Structured data (docs, queries, experiments)
|
||||
│ └── qdrant.py # Vector embeddings
|
||||
│
|
||||
├── documents/ # Document handling
|
||||
│ ├── parser.py # .docx/.doc → DocumentTree + markdown
|
||||
│ ├── service.py # Upload, process, delete orchestration
|
||||
│ ├── routes.py # REST API endpoints
|
||||
│ └── models.py # Request/response schemas
|
||||
│
|
||||
├── chunking/ # Chunking strategies
|
||||
│ ├── base.py # Abstract base class + utilities
|
||||
│ ├── embedding.py # OpenAI embedding service
|
||||
│ ├── service.py # Orchestration (chunk → embed → store)
|
||||
│ └── strategies/ # Strategy implementations
|
||||
│ ├── fixed_size.py
|
||||
│ ├── recursive.py
|
||||
│ ├── semantic.py
|
||||
│ ├── contextual_retrieval.py
|
||||
│ └── semantic_parent_child.py
|
||||
│
|
||||
├── benchmarking/ # Evaluation system
|
||||
│ ├── evaluation.py # LLM-as-Judge scoring
|
||||
│ ├── benchmark_service.py # Orchestration
|
||||
│ ├── query_service.py # Single query pipeline
|
||||
│ ├── routes.py # REST API endpoints
|
||||
│ ├── models.py # Request/response schemas
|
||||
│ └── report.py # HTML report generation
|
||||
│
|
||||
└── main.py # FastAPI app factory
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Document Processing
|
||||
|
||||
```
|
||||
Upload .docx → Parser → DocumentTree + Markdown → SQLite
|
||||
↓
|
||||
Process → Strategy.chunk() → Chunks → Embed → Qdrant
|
||||
```
|
||||
|
||||
### Query Pipeline
|
||||
|
||||
```
|
||||
Question → Embed → Qdrant Search → Top-K Chunks → LLM → Answer
|
||||
```
|
||||
|
||||
### Benchmark Pipeline
|
||||
|
||||
```
|
||||
Questions JSON → For each question × strategy:
|
||||
├── Query Pipeline
|
||||
└── LLM-as-Judge Evaluation
|
||||
↓
|
||||
Aggregate Metrics → SQLite → HTML Report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| **One collection per strategy** | Enables independent comparison |
|
||||
| **SQLite for metadata** | Simple, no setup, sufficient scale |
|
||||
| **Qdrant for vectors** | Purpose-built for vector search |
|
||||
| **OpenAI for embeddings** | Industry standard, consistent results |
|
||||
| **LLM-as-Judge for evaluation** | Automated, consistent scoring |
|
||||
| **Per-strategy failure isolation** | One strategy failure doesn't break others |
|
||||
| **Text stored in Qdrant payload** | Enables retrieval without extra DB lookups |
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
|
||||
| Component | Technology | Version |
|
||||
|-----------|------------|---------|
|
||||
| Framework | FastAPI | 0.100+ |
|
||||
| ORM/DB | SQLite | built-in |
|
||||
| Vector DB | Qdrant | 1.18+ |
|
||||
| Embeddings | OpenAI text-embedding-3-small | - |
|
||||
| LLM | OpenAI gpt-4o-mini | - |
|
||||
| Tokenizer | tiktoken (cl100k_base) | - |
|
||||
| PDF/DOCX | python-docx, LibreOffice | - |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration via environment variables (`.env`):
|
||||
|
||||
```env
|
||||
# Database
|
||||
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
|
||||
QDRANT_URL=http://localhost:6333
|
||||
QDRANT_API_KEY=
|
||||
|
||||
# OpenAI
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# Chunking
|
||||
CHUNK_SIZE=512
|
||||
CHUNK_OVERLAP=50
|
||||
SEMANTIC_THRESHOLD=0.5
|
||||
SEMANTIC_MIN_CHUNK_SIZE=5
|
||||
|
||||
# Models
|
||||
EMBEDDING_MODEL=text-embedding-3-small
|
||||
LLM_MODEL=gpt-4o-mini
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Exception | Handler | HTTP Status |
|
||||
|-----------|---------|-------------|
|
||||
| ChunkingError | chunking_exception_handler | 400 |
|
||||
| BenchmarkError | benchmark_exception_handler | 400 |
|
||||
| QueryError | benchmark_exception_handler | 400 |
|
||||
| DocumentProcessingError | chunking_exception_handler | 400 |
|
||||
| QdrantError | chunking_exception_handler | 400 |
|
||||
| EmbeddingError | chunking_exception_handler | 400 |
|
||||
|
||||
---
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
| Component | Current | Production Recommendation |
|
||||
|-----------|---------|---------------------------|
|
||||
| SQLite | Single file | PostgreSQL for concurrent access |
|
||||
| Qdrant | Single instance | Qdrant Cloud or cluster |
|
||||
| OpenAI | Direct API | Connection pooling, rate limiting |
|
||||
| Workers | Single process | Celery for async processing |
|
||||
| Caching | None | Redis for repeated queries |
|
||||
Reference in New Issue
Block a user