diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..f514de4 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,103 @@ +# Documentation Index + +Complete documentation for the RAG Chunking Benchmarker. + +--- + +## Quick Start + +1. **New to the project?** Start with [Architecture Overview](architecture.md) +2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md) +3. **Need to use the API?** Check [API Reference](api-reference.md) +4. **Configuring the system?** See [Configuration Guide](configuration.md) +5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md) +6. **Curious about data flow?** See [Data Flow](data-flow.md) + +--- + +## Documentation Files + +| File | Purpose | Audience | +|------|---------|----------| +| [architecture.md](architecture.md) | System structure and design | New team members | +| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers | +| [api-reference.md](api-reference.md) | All endpoints documented | Developers | +| [configuration.md](configuration.md) | Settings and environment variables | DevOps | +| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists | +| [data-flow.md](data-flow.md) | How data moves through the system | Engineers | +| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone | +| [phases.md](phases.md) | Implementation phases | Project managers | +| [tasks.md](tasks.md) | Task tracking | Developers | + +--- + +## Architecture at a Glance + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ RAG Chunking Benchmarker │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Documents → Parser → Chunking → Embedding → Qdrant │ +│ │ +│ Questions → Query Pipeline → LLM → Answers │ +│ │ +│ Answers → Evaluation → Metrics → Reports │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Key Concepts + +### Chunking Strategies + +| Strategy | Splitting | Best For | +|----------|-----------|----------| +| fixed_size | Token count | Baseline comparison | +| recursive | Structural boundaries | Structured documents | +| semantic | Similarity thresholds | Topic shifts | +| contextual_retrieval | LLM-enriched tokens | Retrieval quality | +| semantic_parent_child | Paragraph clusters | Context needed | + +### Evaluation Metrics + +| Metric | What It Measures | +|--------|------------------| +| Context Relevance | Did we find the right information? | +| Answer Similarity | Did we produce the right answer? | +| Faithfulness | Is the answer trustworthy? | +| Hallucination | Did we invent information? | + +--- + +## API Endpoints + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/documents` | GET/POST | Manage documents | +| `/documents/{id}/process` | POST | Run chunking strategies | +| `/queries` | POST | Ask questions | +| `/benchmarks` | POST | Run comparisons | +| `/benchmarks/{id}/report` | GET | View results | +| `/experiments` | GET | List experiments | + +--- + +## Configuration + +All settings via `.env` file. See [Configuration Guide](configuration.md) for details. + +Key settings: +- `OPENAI_API_KEY` - Required for embeddings and LLM +- `CHUNK_SIZE` - Target tokens per chunk (default: 512) +- `SEMANTIC_THRESHOLD` - Similarity threshold (default: 0.5) + +--- + +## Further Reading + +- [Phases](phases.md) - Implementation roadmap +- [Tasks](tasks.md) - Task tracking +- [ADRs](adr/) - Architectural Decision Records diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 0000000..a0265c6 --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,309 @@ +# API Reference + +Complete documentation of all REST API endpoints. + +--- + +## Base URL + +``` +http://localhost:8000 +``` + +--- + +## Endpoints + +### Documents + +#### `GET /documents` + +List all uploaded documents. + +**Response:** +```json +{ + "items": [ + { + "id": "doc-abc123", + "filename": "insurance.docx", + "paragraph_count": 245, + "chunk_counts": { + "recursive": 218, + "fixed_size": 117 + }, + "created_at": "2026-07-25T10:00:00" + } + ], + "total": 1, + "offset": 0, + "limit": 50 +} +``` + +--- + +#### `POST /documents` + +Upload a document (.docx or .doc). + +**Request:** +- Content-Type: `multipart/form-data` +- Body: `file` (binary) + +**Response (201):** +```json +{ + "id": "doc-abc123", + "filename": "insurance.docx", + "paragraph_count": 245, + "chunk_counts": {}, + "created_at": "2026-07-25T10:00:00" +} +``` + +--- + +#### `DELETE /documents/{doc_id}` + +Delete a document and its vectors. + +**Response:** +```json +{ + "deleted": true, + "document_id": "doc-abc123" +} +``` + +--- + +#### `POST /documents/{doc_id}/process` + +Run chunking strategies on a document. + +**Request:** +```json +{ + "strategies": [ + "recursive", + "fixed_size", + "semantic", + "contextual_retrieval", + "semantic_parent_child" + ] +} +``` + +**Response:** +```json +{ + "document_id": "doc-abc123", + "strategies_completed": [ + { + "strategy": "recursive", + "status": "completed", + "chunks_produced": 218 + } + ], + "strategies_failed": [] +} +``` + +--- + +### Strategies + +#### `GET /strategies` + +List all available chunking strategies. + +**Response:** +```json +{ + "strategies": [ + { + "name": "recursive", + "description": "Cascade splitting: headers > double newline > ..." + }, + { + "name": "fixed_size", + "description": "Fixed-size token splitting with overlap" + } + ] +} +``` + +--- + +### Queries + +#### `POST /queries` + +Ask a question against a document using a specific strategy. + +**Request:** +```json +{ + "document_id": "doc-abc123", + "strategy": "recursive", + "question": "What are the main topics?", + "top_k": 5 +} +``` + +**Response (201):** +```json +{ + "query_id": "q-xyz789", + "document_id": "doc-abc123", + "strategy": "recursive", + "question": "What are the main topics?", + "answer": "The document covers insurance regulations...", + "retrieved_chunks": [ + { + "chunk_id": "recursive_doc_000045", + "score": 0.892, + "text": " chunk content...", + "parent_id": null + } + ], + "latency_breakdown": { + "embed_seconds": 0.15, + "search_seconds": 0.02, + "answer_seconds": 1.23, + "total_seconds": 1.40 + }, + "token_usage": { + "prompt_tokens": 1250, + "completion_tokens": 150, + "total_tokens": 1400 + }, + "created_at": "2026-07-25T12:00:00" +} +``` + +--- + +#### `GET /queries/{query_id}` + +Retrieve a past query. + +**Response:** Same as POST /queries response. + +--- + +### Benchmarks + +#### `POST /benchmarks` + +Run a benchmark comparing multiple strategies. + +**Request:** +```json +{ + "document_id": "doc-abc123", + "strategies": ["recursive", "fixed_size", "semantic"], + "questions_file": "files/questions.json", + "top_k": 5, + "dry_run": false +} +``` + +**Response (201):** +```json +{ + "experiment_id": "exp-abc123", + "document_id": "doc-abc123", + "strategies_used": ["recursive", "fixed_size", "semantic"], + "questions_count": 21, + "aggregate_metrics": { + "recursive": { + "avg_context_relevance": 8.5, + "avg_answer_similarity": 7.8, + "avg_faithfulness": 9.2, + "hallucination_rate": 0.05, + "total_questions": 21, + "failed_questions": 0 + } + }, + "best_strategy": "recursive", + "total_latency_seconds": 120.5, + "estimated_cost_usd": 0.22, + "created_at": "2026-07-25T12:00:00" +} +``` + +--- + +#### `GET /benchmarks/{experiment_id}` + +Retrieve experiment results. + +**Response:** Same as POST /benchmarks response. + +--- + +#### `GET /benchmarks/{experiment_id}/report` + +Generate HTML report. + +**Query Parameters:** +- `view`: `managerial` (default) or `technical` + +**Response:** HTML page + +--- + +#### `GET /experiments` + +List all experiments. + +**Response:** +```json +{ + "items": [ + { + "id": "exp-abc123", + "document_id": "doc-abc123", + "strategies_used": ["recursive", "fixed_size"], + "created_at": "2026-07-25T12:00:00" + } + ], + "total": 1, + "offset": 0, + "limit": 50 +} +``` + +--- + +## Error Responses + +All errors return: +```json +{ + "detail": "Error message", + "type": "ErrorClassName" +} +``` + +| Status Code | Error Type | Description | +|-------------|------------|-------------| +| 400 | ChunkingError | Invalid request | +| 400 | QueryError | Query failed | +| 400 | BenchmarkError | Benchmark failed | +| 404 | QueryError | Resource not found | +| 422 | ValidationError | Invalid request body | + +--- + +## Rate Limits + +None configured. For production, consider adding rate limiting. + +--- + +## Authentication + +None configured. For production, add API key or OAuth2 authentication. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..c4a149a --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,183 @@ +# Architecture Overview + +System structure and design decisions for the RAG Chunking Benchmarker. + +--- + +## System Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ RAG Chunking Benchmarker │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Documents │ │ Chunking │ │ Benchmarking│ │ +│ │ Module │ │ Module │ │ Module │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Core Layer │ │ +│ │ (config, models, exceptions, dependencies) │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ SQLite │ │ Qdrant │ │ OpenAI │ │ +│ │ (metadata) │ │ (vectors) │ │ (LLM/Emb) │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Module Structure + +``` +src/ +├── core/ # Foundation layer +│ ├── config.py # Settings from .env +│ ├── models.py # Shared Pydantic models +│ ├── exceptions.py # Custom exceptions +│ └── dependencies.py # Client singletons +│ +├── storage/ # Data persistence +│ ├── sqlite.py # Structured data (docs, queries, experiments) +│ └── qdrant.py # Vector embeddings +│ +├── documents/ # Document handling +│ ├── parser.py # .docx/.doc → DocumentTree + markdown +│ ├── service.py # Upload, process, delete orchestration +│ ├── routes.py # REST API endpoints +│ └── models.py # Request/response schemas +│ +├── chunking/ # Chunking strategies +│ ├── base.py # Abstract base class + utilities +│ ├── embedding.py # OpenAI embedding service +│ ├── service.py # Orchestration (chunk → embed → store) +│ └── strategies/ # Strategy implementations +│ ├── fixed_size.py +│ ├── recursive.py +│ ├── semantic.py +│ ├── contextual_retrieval.py +│ └── semantic_parent_child.py +│ +├── benchmarking/ # Evaluation system +│ ├── evaluation.py # LLM-as-Judge scoring +│ ├── benchmark_service.py # Orchestration +│ ├── query_service.py # Single query pipeline +│ ├── routes.py # REST API endpoints +│ ├── models.py # Request/response schemas +│ └── report.py # HTML report generation +│ +└── main.py # FastAPI app factory +``` + +--- + +## Data Flow + +### Document Processing + +``` +Upload .docx → Parser → DocumentTree + Markdown → SQLite + ↓ +Process → Strategy.chunk() → Chunks → Embed → Qdrant +``` + +### Query Pipeline + +``` +Question → Embed → Qdrant Search → Top-K Chunks → LLM → Answer +``` + +### Benchmark Pipeline + +``` +Questions JSON → For each question × strategy: + ├── Query Pipeline + └── LLM-as-Judge Evaluation + ↓ + Aggregate Metrics → SQLite → HTML Report +``` + +--- + +## Key Design Decisions + +| Decision | Rationale | +|----------|-----------| +| **One collection per strategy** | Enables independent comparison | +| **SQLite for metadata** | Simple, no setup, sufficient scale | +| **Qdrant for vectors** | Purpose-built for vector search | +| **OpenAI for embeddings** | Industry standard, consistent results | +| **LLM-as-Judge for evaluation** | Automated, consistent scoring | +| **Per-strategy failure isolation** | One strategy failure doesn't break others | +| **Text stored in Qdrant payload** | Enables retrieval without extra DB lookups | + +--- + +## Technology Stack + +| Component | Technology | Version | +|-----------|------------|---------| +| Framework | FastAPI | 0.100+ | +| ORM/DB | SQLite | built-in | +| Vector DB | Qdrant | 1.18+ | +| Embeddings | OpenAI text-embedding-3-small | - | +| LLM | OpenAI gpt-4o-mini | - | +| Tokenizer | tiktoken (cl100k_base) | - | +| PDF/DOCX | python-docx, LibreOffice | - | + +--- + +## Configuration + +All configuration via environment variables (`.env`): + +```env +# Database +DATABASE_URL=sqlite:///./data/chunking_benchmark.db +QDRANT_URL=http://localhost:6333 +QDRANT_API_KEY= + +# OpenAI +OPENAI_API_KEY=sk-... + +# Chunking +CHUNK_SIZE=512 +CHUNK_OVERLAP=50 +SEMANTIC_THRESHOLD=0.5 +SEMANTIC_MIN_CHUNK_SIZE=5 + +# Models +EMBEDDING_MODEL=text-embedding-3-small +LLM_MODEL=gpt-4o-mini +``` + +--- + +## Error Handling + +| Exception | Handler | HTTP Status | +|-----------|---------|-------------| +| ChunkingError | chunking_exception_handler | 400 | +| BenchmarkError | benchmark_exception_handler | 400 | +| QueryError | benchmark_exception_handler | 400 | +| DocumentProcessingError | chunking_exception_handler | 400 | +| QdrantError | chunking_exception_handler | 400 | +| EmbeddingError | chunking_exception_handler | 400 | + +--- + +## Scaling Considerations + +| Component | Current | Production Recommendation | +|-----------|---------|---------------------------| +| SQLite | Single file | PostgreSQL for concurrent access | +| Qdrant | Single instance | Qdrant Cloud or cluster | +| OpenAI | Direct API | Connection pooling, rate limiting | +| Workers | Single process | Celery for async processing | +| Caching | None | Redis for repeated queries | diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..3e7f66a --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,164 @@ +# Configuration Guide + +All settings and environment variables for the RAG Chunking Benchmarker. + +--- + +## Environment Variables + +Create a `.env` file in the project root: + +```env +# ── Database ────────────────────────────────────────────────────── + +# SQLite database path +DATABASE_URL=sqlite:///./data/chunking_benchmark.db + +# Qdrant vector database +QDRANT_URL=http://localhost:6333 +QDRANT_API_KEY= + +# ── OpenAI ──────────────────────────────────────────────────────── + +# API key for embeddings and LLM +OPENAI_API_KEY=sk-your-key-here + +# ── Chunking Parameters ─────────────────────────────────────────── + +# Fixed-size strategy +CHUNK_SIZE=512 +CHUNK_OVERLAP=50 + +# Semantic strategy +SEMANTIC_THRESHOLD=0.5 +SEMANTIC_MIN_CHUNK_SIZE=5 + +# ── Models ──────────────────────────────────────────────────────── + +# Embedding model (used for all strategies) +EMBEDDING_MODEL=text-embedding-3-small + +# LLM model (used for contextual retrieval and evaluation) +LLM_MODEL=gpt-4o-mini +``` + +--- + +## Parameter Details + +### Chunking Parameters + +| Parameter | Type | Default | Range | Description | +|-----------|------|---------|-------|-------------| +| `CHUNK_SIZE` | int | 512 | 100-2000 | Target tokens per chunk | +| `CHUNK_OVERLAP` | int | 50 | 0-200 | Token overlap between chunks | +| `SEMANTIC_THRESHOLD` | float | 0.5 | 0.0-1.0 | Cosine similarity threshold for chunk boundaries | +| `SEMANTIC_MIN_CHUNK_SIZE` | int | 5 | 1-20 | Minimum sentences per semantic chunk | + +### Model Parameters + +| Parameter | Type | Default | Options | Description | +|-----------|------|---------|---------|-------------| +| `EMBEDDING_MODEL` | str | text-embedding-3-small | text-embedding-3-small, text-embedding-3-large | OpenAI embedding model | +| `LLM_MODEL` | str | gpt-4o-mini | gpt-4o-mini, gpt-4o | Model for contextual retrieval and evaluation | + +--- + +## Default Values in Code + +```python +# src/core/config.py +class Settings(BaseSettings): + # Database + database_url: str = "sqlite:///./data/chunking_benchmark.db" + qdrant_url: str = "http://localhost:6333" + qdrant_api_key: str = "" + + # OpenAI + openai_api_key: str = "" + + # Chunking + chunk_size: int = 512 + chunk_overlap: int = 50 + semantic_threshold: float = 0.5 + semantic_min_chunk_size: int = 5 + + # Models + embedding_model: str = "text-embedding-3-small" + llm_model: str = "gpt-4o-mini" +``` + +--- + +## Effect of Parameters + +### chunk_size + +| Value | Chunks | Quality | Speed | +|-------|--------|---------|-------| +| 256 | More | Higher context loss | Faster | +| 512 | Balanced | Balanced | Balanced | +| 1024 | Fewer | Lower context loss | Slower | + +### chunk_overlap + +| Value | Context Preservation | Redundancy | +|-------|---------------------|------------| +| 0 | None | None | +| 50 | Moderate | Low | +| 100 | High | High | + +### semantic_threshold + +| Value | Chunk Size | Topic Sensitivity | +|-------|------------|-------------------| +| 0.3 | Larger | Less sensitive | +| 0.5 | Balanced | Balanced | +| 0.7 | Smaller | More sensitive | + +--- + +## Environment-Specific Configs + +### Development + +```env +DATABASE_URL=sqlite:///./data/dev.db +QDRANT_URL=http://localhost:6333 +LOG_LEVEL=DEBUG +``` + +### Production + +```env +DATABASE_URL=postgresql://user:pass@localhost/dbname +QDRANT_URL=http://qdrant:6333 +QDRANT_API_KEY=your-api-key +LOG_LEVEL=WARNING +``` + +### Testing + +```env +DATABASE_URL=sqlite:///./data/test.db +QDRANT_URL=http://localhost:6333 +CHUNK_SIZE=100 +``` + +--- + +## Validation + +Settings are validated on startup. Invalid values will cause: + +``` +pydantic.ValidationError: 1 validation error for Settings +``` + +Common errors: + +| Error | Cause | Fix | +|-------|-------|-----| +| Missing OPENAI_API_KEY | No API key | Add to .env | +| Invalid QDRANT_URL | Wrong URL format | Check URL | +| CHUNK_SIZE < 1 | Too small | Increase value | diff --git a/docs/data-flow.md b/docs/data-flow.md new file mode 100644 index 0000000..aee646a --- /dev/null +++ b/docs/data-flow.md @@ -0,0 +1,385 @@ +# Data Flow + +How data moves through the RAG Chunking Benchmarker system. + +--- + +## Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ DATA FLOW │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ INGESTION │ +│ ───────── │ +│ .docx/.doc → Parser → DocumentTree + Markdown → SQLite │ +│ │ +│ PROCESSING │ +│ ────────── │ +│ Document → Strategy.chunk() → Chunks → Embed → Qdrant │ +│ │ +│ QUERYING │ +│ ───────── │ +│ Question → Embed → Qdrant Search → Chunks → LLM → Answer │ +│ │ +│ BENCHMARKING │ +│ ─────────── │ +│ Questions → (Query × Strategies) → Evaluation → Report │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 1. Document Ingestion + +### Flow + +``` +User uploads .docx/.doc + ↓ + FastAPI receives file + ↓ + Parser extracts text blocks + ↓ + Builds DocumentTree (hierarchy) + ↓ + Generates markdown (flat text) + ↓ + Stores in SQLite +``` + +### Data Structures + +**Input:** Binary file (.docx/.doc) + +**Output:** +```python +ParseResult( + tree=DocumentTree, # Hierarchical structure + markdown=str, # Flat text for chunking + plain_text=str, # Raw text + paragraph_count=int # Number of text blocks +) +``` + +### Storage + +**SQLite:** +```sql +INSERT INTO documents (id, filename, parsed_text, document_tree, created_at) +VALUES (?, ?, ?, ?, ?) +``` + +--- + +## 2. Document Processing + +### Flow + +``` +User selects strategies + ↓ +For each strategy: + ↓ + Load DocumentTree + markdown from SQLite + ↓ + Strategy.chunk() → list[Chunk] + ↓ + Embed chunks via OpenAI + ↓ + Store in Qdrant collection + ↓ + Update chunk_counts in SQLite +``` + +### Data Structures + +**Input:** +```python +{ + "document_id": "doc-abc123", + "strategies": ["recursive", "fixed_size", "semantic"] +} +``` + +**Processing:** +```python +# For each strategy +chunks = strategy.chunk( + doc_name="insurance.docx", + tree=document_tree, + markdown=markdown_text +) + +embeddings = embed_texts([chunk.text for chunk in chunks]) + +qdr.upsert_chunks(chunks, embeddings) +``` + +**Output:** +```python +{ + "strategies_completed": [ + {"strategy": "recursive", "chunks_produced": 218} + ] +} +``` + +### Storage + +**Qdrant:** +```python +{ + "id": "uuid5(chunk_id)", + "vector": [0.023, -0.156, ...], # 1536 dimensions + "payload": { + "document_name": "insurance.docx", + "chunk_id": "recursive_doc_000045", + "strategy_name": "recursive", + "text": "chunk content...", + "token_count": 146, + "character_count": 205 + } +} +``` + +--- + +## 3. Query Pipeline + +### Flow + +``` +User asks question + ↓ +Embed question via OpenAI + ↓ +Search Qdrant for similar chunks + ↓ +Retrieve top-K chunks + ↓ +Build context from chunks + ↓ +Generate answer via LLM + ↓ +Store query result in SQLite +``` + +### Data Structures + +**Input:** +```python +{ + "document_id": "doc-abc123", + "strategy": "recursive", + "question": "What are the main topics?", + "top_k": 5 +} +``` + +**Processing:** +```python +# 1. Embed question +question_embedding = embed_single(question) + +# 2. Vector search +hits = qdr.search( + strategy="recursive", + query_vector=question_embedding, + top_k=5 +) + +# 3. Build context +context = "\n".join([hit.text for hit in hits]) + +# 4. Generate answer +answer = llm.generate( + system="Answer based on context...", + user=f"Context: {context}\nQuestion: {question}" +) +``` + +**Output:** +```python +{ + "answer": "The document covers insurance regulations...", + "retrieved_chunks": [...], + "latency_breakdown": {...}, + "token_usage": {...} +} +``` + +### Storage + +**SQLite:** +```sql +INSERT INTO queries (id, document_id, strategy_name, question, answer, + retrieved_chunks, latency_breakdown, token_usage) +VALUES (?, ?, ?, ?, ?, ?, ?, ?) +``` + +--- + +## 4. Benchmark Pipeline + +### Flow + +``` +Load questions from JSON + ↓ +For each question × strategy: + ↓ + Run query pipeline + ↓ + Evaluate with LLM-as-Judge + ↓ + Store per-question results + ↓ +Aggregate metrics per strategy + ↓ +Store experiment in SQLite + ↓ +Generate HTML report +``` + +### Data Structures + +**Input:** +```python +{ + "document_id": "doc-abc123", + "strategies": ["recursive", "fixed_size"], + "questions_file": "files/questions.json" +} +``` + +**Processing:** +```python +# For each question +for question in questions: + for strategy in strategies: + # Query + result = run_query(question, strategy) + + # Evaluate + scores = evaluate_single( + question=question, + context=result.retrieved_chunks, + expected=question.expected_answer, + generated=result.answer + ) + + # Store + per_question_results.append({...}) +``` + +**Output:** +```python +{ + "experiment_id": "exp-abc123", + "aggregate_metrics": { + "recursive": { + "avg_context_relevance": 8.5, + "avg_answer_similarity": 7.8, + "avg_faithfulness": 9.2, + "hallucination_rate": 0.05 + } + }, + "best_strategy": "recursive" +} +``` + +--- + +## 5. Report Generation + +### Flow + +``` +Load experiment from SQLite + ↓ +Calculate rankings + ↓ +Generate HTML (managerial or technical view) + ↓ +Return HTML response +``` + +### Views + +| View | Focus | Content | +|------|-------|---------| +| Managerial | Decision-making | KPIs, winner, recommendations | +| Technical | Observability | Detailed scores, token usage, per-question | + +--- + +## Data Persistence + +### SQLite Tables + +| Table | Purpose | Key Fields | +|-------|---------|------------| +| documents | Uploaded documents | id, filename, parsed_text, document_tree | +| queries | Query results | id, document_id, question, answer | +| experiments | Benchmark results | id, document_id, aggregate_metrics | + +### Qdrant Collections + +| Collection | Purpose | Fields | +|------------|---------|--------| +| recursive_collection | Recursive strategy vectors | chunk_id, text, scores | +| fixed_size_collection | Fixed-size strategy vectors | chunk_id, text, scores | +| semantic_collection | Semantic strategy vectors | chunk_id, text, scores | +| contextual_retrieval_collection | Contextual strategy vectors | chunk_id, text, scores | +| semantic_parent_child_collection | Parent-child strategy vectors | chunk_id, text, parent_id | + +--- + +## Data Transformation + +### Document → Chunks + +``` +Input: Full document text (10,000 tokens) +Output: 50-300 chunks (100-500 tokens each) + +Transformation: +- Tokenization +- Boundary detection +- Metadata enrichment +``` + +### Chunks → Vectors + +``` +Input: Chunk text +Output: 1536-dimensional vector + +Transformation: +- Text → Tokens → Embedding API → Vector +``` + +### Question → Answer + +``` +Input: Question string +Output: Answer string + metadata + +Transformation: +- Question → Embedding → Vector Search → Chunks → LLM → Answer +``` + +--- + +## Performance Characteristics + +| Operation | Time | Tokens | Cost | +|-----------|------|--------|------| +| Document upload | ~2s | 0 | Free | +| Process (1 strategy) | ~5s | N/A | ~$0.001 (embeddings) | +| Process (5 strategies) | ~25s | N/A | ~$0.005 | +| Query | ~2s | ~1500 | ~$0.001 | +| Benchmark (21 questions) | ~120s | ~200,000 | ~$0.05 | diff --git a/docs/evaluation-metrics.md b/docs/evaluation-metrics.md new file mode 100644 index 0000000..a505ba9 --- /dev/null +++ b/docs/evaluation-metrics.md @@ -0,0 +1,222 @@ +# Evaluation Metrics + +How scoring works in the benchmarking system. + +--- + +## Overview + +The system uses **LLM-as-Judge** evaluation to automatically score answer quality across 4 metrics. + +--- + +## Metrics + +### 1. Context Relevance (1-10) + +**Question:** How relevant are the retrieved chunks to answering the question? + +| Score | Meaning | +|-------|---------| +| 9-10 | Chunks directly answer the question | +| 7-8 | Chunks are highly relevant | +| 5-6 | Chunks are somewhat relevant | +| 3-4 | Chunks are partially relevant | +| 1-2 | Chunks are not relevant | + +**What it measures:** Did the retrieval system find the right information? + +--- + +### 2. Answer Similarity (1-10) + +**Question:** How similar is the generated answer to the expected answer? + +| Score | Meaning | +|-------|---------| +| 9-10 | Nearly identical to expected | +| 7-8 | Covers most key points | +| 5-6 | Covers some key points | +| 3-4 | Partially correct | +| 1-2 | Completely different | + +**What it measures:** Did the system produce the right answer? + +--- + +### 3. Faithfulness (1-10) + +**Question:** Is the generated answer grounded in the retrieved context? + +| Score | Meaning | +|-------|---------| +| 9-10 | Entirely based on context | +| 7-8 | Mostly based on context | +| 5-6 | Partially based on context | +| 3-4 | Some external knowledge used | +| 1-2 | Mostly external knowledge | + +**What it measures:** Is the answer trustworthy? + +--- + +### 4. Hallucination (boolean) + +**Question:** Did the LLM invent information not in the context? + +| Value | Meaning | +|-------|---------| +| false | Answer is grounded in context | +| true | Answer contains invented information | + +**What it measures:** Is the answer fabricated? + +--- + +## Scoring Process + +### Step 1: Prepare Evaluation Context + +``` +Question: {user_question} +Context: {retrieved_chunks} +Expected Answer: {golden_answer} +Generated Answer: {system_answer} +``` + +### Step 2: Send to LLM-as-Judge + +```python +prompt = f""" +Evaluate this RAG system output: + +Question: {question} +Context: {context} +Expected: {expected} +Generated: {generated} + +Return JSON: +{{ + "context_relevance": <1-10>, + "answer_similarity": <1-10>, + "faithfulness": <1-10>, + "hallucination": , + "reasoning": "" +}} +""" +``` + +### Step 3: Parse Response + +```python +scores = json.loads(llm_response) +# Validate ranges +for metric in ["context_relevance", "answer_similarity", "faithfulness"]: + scores[metric] = max(1, min(10, scores[metric])) +``` + +--- + +## Overall Score Calculation + +Each strategy gets an overall score weighted by importance: + +```python +overall = ( + context_relevance * 0.3 + # 30% weight + answer_similarity * 0.4 + # 40% weight + faithfulness * 0.3 # 30% weight +) * (1 - hallucination_rate) # Hallucination penalty +``` + +### Why These Weights? + +| Metric | Weight | Rationale | +|--------|--------|-----------| +| Answer Similarity | 40% | Most important - did we get the right answer? | +| Context Relevance | 30% | Found the right information | +| Faithfulness | 30% | Answer is trustworthy | +| Hallucination | Penalty | Fabricated info is unacceptable | + +--- + +## Aggregate Metrics + +Per strategy, we calculate: + +| Metric | Calculation | +|--------|-------------| +| avg_context_relevance | mean of all context_relevance scores | +| avg_answer_similarity | mean of all answer_similarity scores | +| avg_faithfulness | mean of all faithfulness scores | +| hallucination_rate | count(hallucination=True) / total_questions | +| total_questions | number of questions evaluated | +| failed_questions | questions that errored | + +--- + +## Example Output + +```json +{ + "recursive": { + "avg_context_relevance": 8.5, + "avg_answer_similarity": 7.8, + "avg_faithfulness": 9.2, + "hallucination_rate": 0.05, + "total_questions": 21, + "failed_questions": 0 + } +} +``` + +--- + +## Interpretation Guide + +### Good Scores + +| Metric | Target | Meaning | +|--------|--------|---------| +| Context Relevance | ≥ 8 | Retrieval is accurate | +| Answer Similarity | ≥ 8 | Answers match expectations | +| Faithfulness | ≥ 9 | Answers are trustworthy | +| Hallucination Rate | ≤ 0.1 | Low fabrication rate | + +### Warning Signs + +| Metric | Warning | Meaning | +|--------|---------|---------| +| Context Relevance | < 6 | Retrieval needs improvement | +| Answer Similarity | < 6 | Answers are off-target | +| Faithfulness | < 7 | Model is adding external knowledge | +| Hallucination Rate | > 0.2 | High fabrication rate | + +--- + +## Cost Considerations + +Each evaluation uses: + +| Component | Cost | +|-----------|------| +| LLM call (gpt-4o-mini) | ~$0.001 per evaluation | +| Input tokens | ~500 per evaluation | +| Output tokens | ~100 per evaluation | + +**Total for 21 questions × 5 strategies:** +- 105 evaluations × $0.001 = **$0.105** + +--- + +## Configuration + +Evaluation parameters in `src/benchmarking/evaluation.py`: + +```python +_EVALUATION_SYSTEM_PROMPT = "You are an expert evaluator..." +_EVALUATION_USER_PROMPT = "Evaluate this RAG system output..." +model = settings.llm_model # gpt-4o-mini +temperature = 0.0 # Deterministic +max_tokens = 500 # Response limit +``` diff --git a/docs/strategy-technical-details.md b/docs/strategy-technical-details.md new file mode 100644 index 0000000..2034fea --- /dev/null +++ b/docs/strategy-technical-details.md @@ -0,0 +1,345 @@ +# Strategy Technical Details + +Deep technical explanation of each chunking strategy's implementation. + +--- + +## 1. Fixed-Size Chunking + +**File:** `src/chunking/strategies/fixed_size.py` + +### How It Works + +Splits text into fixed-size token chunks with overlap. + +### Splitting Unit + +**Token-level** using `tiktoken` (cl100k_base encoding). + +### Algorithm + +``` +1. Encode full markdown → token array +2. Take first chunk_size tokens → chunk 1 +3. Slide forward by (chunk_size - overlap) tokens +4. Repeat until end of text +``` + +### Parameters + +| Parameter | Config Key | Default | Description | +|-----------|------------|---------|-------------| +| Chunk size | `chunk_size` | 512 | Tokens per chunk | +| Overlap | `chunk_overlap` | 50 | Token overlap between chunks | + +### Code Logic + +```python +tokens = _encoder.encode(text) +start = 0 +while start < len(tokens): + end = min(start + chunk_size, len(tokens)) + chunk_tokens = tokens[start:end] + chunks.append(_encoder.decode(chunk_tokens)) + start = end - overlap +``` + +### Characteristics + +- **Splitting boundary:** Arbitrary (mid-sentence possible) +- **Structure awareness:** None +- **Speed:** Fastest (no API calls) +- **Best for:** Baseline comparison, simple documents + +--- + +## 2. Recursive Chunking + +**File:** `src/chunking/strategies/recursive.py` + +### How It Works + +Cascade splitting using a separator hierarchy. Tries meaningful boundaries first, falls back to less meaningful ones. + +### Splitting Unit + +**Character-level** with regex patterns. + +### Separator Cascade (Priority Order) + +| Priority | Separator | Regex Pattern | Description | +|----------|-----------|---------------|-------------| +| 1 | Header | `^(#{1,6})\s+` | Markdown headers | +| 2 | Double newline | `\n\n` | Paragraph breaks | +| 3 | Single newline | `\n` | Line breaks | +| 4 | Sentence | `[.!?]\s+` | Sentence endings | +| 5 | Space | `\s+` | Word boundaries | + +### Algorithm + +``` +1. Try splitting by highest-priority separator +2. If parts still too big → recurse with next separator +3. Merge small parts back up to target size +4. If single part exceeds target → hard split by character count +``` + +### Code Logic + +```python +for sep_name, pattern in _SEPARATORS: + parts = pattern.split(text) + if len(parts) <= 1: + continue # this separator didn't split + + # Merge parts back up to target_size + current = "" + for part in parts: + candidate = (current + " " + part).strip() + if len(candidate) <= target_size: + current = candidate + else: + chunks.append(current) + current = part + + return chunks # Return on first successful split +``` + +### Characteristics + +- **Splitting boundary:** Semantic (headers, paragraphs, sentences) +- **Structure awareness:** High (respects markdown structure) +- **Speed:** Fast (no API calls) +- **Best for:** Structured documents with clear hierarchy + +--- + +## 3. Semantic Chunking + +**File:** `src/chunking/strategies/semantic.py` + +### How It Works + +Groups sentences by semantic similarity. When similarity drops, starts a new chunk. + +### Splitting Unit + +**Sentence-level** with similarity-based boundaries. + +### Algorithm + +``` +1. Split markdown into sentences +2. Embed each sentence via OpenAI (text-embedding-3-small) +3. Compute cosine similarity between adjacent sentences +4. When similarity < SEMANTIC_THRESHOLD → chunk boundary +5. Enforce SEMANTIC_MIN_CHUNK_SIZE (minimum sentences per chunk) +``` + +### Parameters + +| Parameter | Config Key | Default | Description | +|-----------|------------|---------|-------------| +| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for boundary | +| Min size | `semantic_min_chunk_size` | 5 | Minimum sentences per chunk | + +### Code Logic + +```python +sentences = split_sentences(markdown) +embeddings = embed_texts(sentences) # OpenAI API call + +current_group = [sentences[0]] +for i in range(1, len(sentences)): + sim = cosine_similarity(embeddings[i-1], embeddings[i]) + + if sim < threshold and len(current_group) >= min_size: + # Topic shift → new chunk + chunks.append(" ".join(current_group)) + current_group = [sentences[i]] + else: + current_group.append(sentences[i]) +``` + +### Characteristics + +- **Splitting boundary:** Semantic (topic changes) +- **Structure awareness:** Medium (sentence-level) +- **Speed:** Medium (requires embeddings) +- **Best for:** Documents with topic shifts, no clear structure + +--- + +## 4. Contextual Retrieval + +**File:** `src/chunking/strategies/contextual_retrieval.py` + +### How It Works + +Based on Anthropic's 2024 research. Prepends a short context summary to each chunk before embedding. + +### Splitting Unit + +**Token-level** (same as fixed-size). + +### Algorithm + +``` +1. Split text using fixed-size token splitting +2. For each chunk: + a. Find surrounding text (500 chars before/after) + b. Send to LLM with context prompt + c. LLM generates 1-2 sentence context prefix + d. Enriched chunk = context + original text +3. Embed enriched chunks (not original) +``` + +### LLM Prompt + +``` +System: You are a document analysis assistant. Given a section of text +from a document, write a short context prefix (1-2 sentences) that +would help someone find this section later via search. + +User: +Preceding text: {preceding_500_chars} +This section: {chunk_text} +Following text: {following_500_chars} +``` + +### Code Logic + +```python +for chunk_text in raw_chunks: + # Find surrounding context + preceding = full_text[pos-500:pos] + following = full_text[pos+len:pos+len+500] + + # Generate context prefix + context = llm.generate( + system=CONTEXT_PROMPT, + user=f"Preceding: {preceding}\nSection: {chunk_text}\nFollowing: {following}" + ) + + # Enrich chunk + enriched = f"{context}\n\n{chunk_text}" + chunks.append(enriched) +``` + +### Characteristics + +- **Splitting boundary:** Token-based (like fixed-size) +- **Structure awareness:** None (relies on LLM for context) +- **Speed:** Slowest (1 LLM call per chunk) +- **Best for:** Improving retrieval quality, unstructured documents + +--- + +## 5. Semantic Parent-Child + +**File:** `src/chunking/strategies/semantic_parent_child.py` + +### How It Works + +Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child. + +### Splitting Unit + +**Paragraph-level** with semantic clustering. + +### Algorithm + +``` +1. Split markdown into paragraphs +2. Embed each paragraph +3. Cluster consecutive paragraphs by similarity +4. Each cluster = parent chunk (full cluster text) +5. Each paragraph = child chunk (linked to parent) +``` + +### Query-Time Behavior + +``` +1. Vector search finds matching child paragraph +2. Use parent_id to fetch parent cluster +3. Return both child + parent to LLM +``` + +### Parameters + +| Parameter | Config Key | Default | Description | +|-----------|------------|---------|-------------| +| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for clustering | + +### Code Logic + +```python +paragraphs = split_paragraphs(markdown) +embeddings = embed_texts(paragraphs) + +# Cluster paragraphs +clusters = [[0]] +for i in range(1, len(paragraphs)): + sim = cosine_similarity(embeddings[i-1], embeddings[i]) + if sim >= threshold: + clusters[-1].append(i) # Same cluster + else: + clusters.append([i]) # New cluster + +# Create parent-child chunks +for cluster in clusters: + parent_text = "\n\n".join(paragraphs[i] for i in cluster) + parent_id = make_chunk_id(...) + + for para_idx in cluster: + chunks.append(build_chunk( + text=paragraphs[para_idx], + parent_id=parent_id + )) +``` + +### Characteristics + +- **Splitting boundary:** Semantic (paragraph clusters) +- **Structure awareness:** Medium (paragraph-level) +- **Speed:** Medium (requires embeddings) +- **Best for:** Documents needing context, flat structure + +--- + +## Comparison Table + +| Strategy | Split Unit | Boundary Logic | API Calls | Speed | Best For | +|----------|------------|----------------|-----------|-------|----------| +| fixed_size | Token | Arbitrary | 0 | Fast | Baseline | +| recursive | Character | Structural | 0 | Fast | Structured docs | +| semantic | Sentence | Similarity | 1 (embeddings) | Medium | Topic shifts | +| contextual_retrieval | Token | LLM-generated | N (1 per chunk) | Slow | Retrieval quality | +| semantic_parent_child | Paragraph | Similarity | 1 (embeddings) | Medium | Context needed | + +--- + +## Configuration Reference + +All strategy parameters are in `src/core/config.py`: + +```python +class Settings(BaseSettings): + chunk_size: int = 512 + chunk_overlap: int = 50 + semantic_threshold: float = 0.5 + semantic_min_chunk_size: int = 5 + embedding_model: str = "text-embedding-3-small" + llm_model: str = "gpt-4o-mini" +``` + +--- + +## ADR References + +| ADR | Strategy | Decision | +|-----|----------|----------| +| ADR 0012 | semantic | Sentence-level with min chunk size | +| ADR 0015 | recursive | Direct API, no LangChain/LlamaIndex | +| ADR 0011 | contextual_retrieval | Embed enriched text, not original | +| ADR 0003 | all | Per-strategy failure isolation |