mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add comprehensive project documentation

Browse Source 
Why:
- Need documentation for team onboarding and reference
- Need technical details for strategy implementations
- Need API reference for developers

Changes:
- README.md: Documentation index and quick start guide
- api-reference.md: All 10 endpoints with examples
- architecture.md: System structure and design decisions
- configuration.md: All environment variables and parameters
- data-flow.md: How data moves through the system
- evaluation-metrics.md: How scoring works with weights
- strategy-technical-details.md: Deep dive into each strategy's implementation
This commit is contained in:
Mahdi Bazrafshan
2026-07-27 14:15:28 +03:30
parent 4e76203e27
commit 754da323ff
7 changed files with 1711 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
docs/README.md Normal file
View File

@@ -0,0 +1,103 @@
# Documentation Index
Complete documentation for the RAG Chunking Benchmarker.
---
## Quick Start
1. **New to the project?** Start with [Architecture Overview](architecture.md)
2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
3. **Need to use the API?** Check [API Reference](api-reference.md)
4. **Configuring the system?** See [Configuration Guide](configuration.md)
5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
6. **Curious about data flow?** See [Data Flow](data-flow.md)
---
## Documentation Files
| File | Purpose | Audience |
|------|---------|----------|
| [architecture.md](architecture.md) | System structure and design | New team members |
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
| [api-reference.md](api-reference.md) | All endpoints documented | Developers |
| [configuration.md](configuration.md) | Settings and environment variables | DevOps |
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
| [phases.md](phases.md) | Implementation phases | Project managers |
| [tasks.md](tasks.md) | Task tracking | Developers |
---
## Architecture at a Glance
```
┌─────────────────────────────────────────────────────────────────┐
│ RAG Chunking Benchmarker │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Documents → Parser → Chunking → Embedding → Qdrant │
│ │
│ Questions → Query Pipeline → LLM → Answers │
│ │
│ Answers → Evaluation → Metrics → Reports │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## Key Concepts
### Chunking Strategies
| Strategy | Splitting | Best For |
|----------|-----------|----------|
| fixed_size | Token count | Baseline comparison |
| recursive | Structural boundaries | Structured documents |
| semantic | Similarity thresholds | Topic shifts |
| contextual_retrieval | LLM-enriched tokens | Retrieval quality |
| semantic_parent_child | Paragraph clusters | Context needed |
### Evaluation Metrics
| Metric | What It Measures |
|--------|------------------|
| Context Relevance | Did we find the right information? |
| Answer Similarity | Did we produce the right answer? |
| Faithfulness | Is the answer trustworthy? |
| Hallucination | Did we invent information? |
---
## API Endpoints
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/documents` | GET/POST | Manage documents |
| `/documents/{id}/process` | POST | Run chunking strategies |
| `/queries` | POST | Ask questions |
| `/benchmarks` | POST | Run comparisons |
| `/benchmarks/{id}/report` | GET | View results |
| `/experiments` | GET | List experiments |
---
## Configuration
All settings via `.env` file. See [Configuration Guide](configuration.md) for details.
Key settings:
- `OPENAI_API_KEY` - Required for embeddings and LLM
- `CHUNK_SIZE` - Target tokens per chunk (default: 512)
- `SEMANTIC_THRESHOLD` - Similarity threshold (default: 0.5)
---
## Further Reading
- [Phases](phases.md) - Implementation roadmap
- [Tasks](tasks.md) - Task tracking
- [ADRs](adr/) - Architectural Decision Records

309
docs/api-reference.md Normal file
View File

@@ -0,0 +1,309 @@
# API Reference
Complete documentation of all REST API endpoints.
---
## Base URL
```
http://localhost:8000
```
---
## Endpoints
### Documents
#### `GET /documents`
List all uploaded documents.
**Response:**
```json
{
"items": [
{
"id": "doc-abc123",
"filename": "insurance.docx",
"paragraph_count": 245,
"chunk_counts": {
"recursive": 218,
"fixed_size": 117
},
"created_at": "2026-07-25T10:00:00"
}
],
"total": 1,
"offset": 0,
"limit": 50
}
```
---
#### `POST /documents`
Upload a document (.docx or .doc).
**Request:**
- Content-Type: `multipart/form-data`
- Body: `file` (binary)
**Response (201):**
```json
{
"id": "doc-abc123",
"filename": "insurance.docx",
"paragraph_count": 245,
"chunk_counts": {},
"created_at": "2026-07-25T10:00:00"
}
```
---
#### `DELETE /documents/{doc_id}`
Delete a document and its vectors.
**Response:**
```json
{
"deleted": true,
"document_id": "doc-abc123"
}
```
---
#### `POST /documents/{doc_id}/process`
Run chunking strategies on a document.
**Request:**
```json
{
"strategies": [
"recursive",
"fixed_size",
"semantic",
"contextual_retrieval",
"semantic_parent_child"
]
}
```
**Response:**
```json
{
"document_id": "doc-abc123",
"strategies_completed": [
{
"strategy": "recursive",
"status": "completed",
"chunks_produced": 218
}
],
"strategies_failed": []
}
```
---
### Strategies
#### `GET /strategies`
List all available chunking strategies.
**Response:**
```json
{
"strategies": [
{
"name": "recursive",
"description": "Cascade splitting: headers > double newline > ..."
},
{
"name": "fixed_size",
"description": "Fixed-size token splitting with overlap"
}
]
}
```
---
### Queries
#### `POST /queries`
Ask a question against a document using a specific strategy.
**Request:**
```json
{
"document_id": "doc-abc123",
"strategy": "recursive",
"question": "What are the main topics?",
"top_k": 5
}
```
**Response (201):**
```json
{
"query_id": "q-xyz789",
"document_id": "doc-abc123",
"strategy": "recursive",
"question": "What are the main topics?",
"answer": "The document covers insurance regulations...",
"retrieved_chunks": [
{
"chunk_id": "recursive_doc_000045",
"score": 0.892,
"text": " chunk content...",
"parent_id": null
}
],
"latency_breakdown": {
"embed_seconds": 0.15,
"search_seconds": 0.02,
"answer_seconds": 1.23,
"total_seconds": 1.40
},
"token_usage": {
"prompt_tokens": 1250,
"completion_tokens": 150,
"total_tokens": 1400
},
"created_at": "2026-07-25T12:00:00"
}
```
---
#### `GET /queries/{query_id}`
Retrieve a past query.
**Response:** Same as POST /queries response.
---
### Benchmarks
#### `POST /benchmarks`
Run a benchmark comparing multiple strategies.
**Request:**
```json
{
"document_id": "doc-abc123",
"strategies": ["recursive", "fixed_size", "semantic"],
"questions_file": "files/questions.json",
"top_k": 5,
"dry_run": false
}
```
**Response (201):**
```json
{
"experiment_id": "exp-abc123",
"document_id": "doc-abc123",
"strategies_used": ["recursive", "fixed_size", "semantic"],
"questions_count": 21,
"aggregate_metrics": {
"recursive": {
"avg_context_relevance": 8.5,
"avg_answer_similarity": 7.8,
"avg_faithfulness": 9.2,
"hallucination_rate": 0.05,
"total_questions": 21,
"failed_questions": 0
}
},
"best_strategy": "recursive",
"total_latency_seconds": 120.5,
"estimated_cost_usd": 0.22,
"created_at": "2026-07-25T12:00:00"
}
```
---
#### `GET /benchmarks/{experiment_id}`
Retrieve experiment results.
**Response:** Same as POST /benchmarks response.
---
#### `GET /benchmarks/{experiment_id}/report`
Generate HTML report.
**Query Parameters:**
- `view`: `managerial` (default) or `technical`
**Response:** HTML page
---
#### `GET /experiments`
List all experiments.
**Response:**
```json
{
"items": [
{
"id": "exp-abc123",
"document_id": "doc-abc123",
"strategies_used": ["recursive", "fixed_size"],
"created_at": "2026-07-25T12:00:00"
}
],
"total": 1,
"offset": 0,
"limit": 50
}
```
---
## Error Responses
All errors return:
```json
{
"detail": "Error message",
"type": "ErrorClassName"
}
```
| Status Code | Error Type | Description |
|-------------|------------|-------------|
| 400 | ChunkingError | Invalid request |
| 400 | QueryError | Query failed |
| 400 | BenchmarkError | Benchmark failed |
| 404 | QueryError | Resource not found |
| 422 | ValidationError | Invalid request body |
---
## Rate Limits
None configured. For production, consider adding rate limiting.
---
## Authentication
None configured. For production, add API key or OAuth2 authentication.

183
docs/architecture.md Normal file
View File

@@ -0,0 +1,183 @@
# Architecture Overview
System structure and design decisions for the RAG Chunking Benchmarker.
---
## System Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ RAG Chunking Benchmarker │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Documents │ │ Chunking │ │ Benchmarking│ │
│ │ Module │ │ Module │ │ Module │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Core Layer │ │
│ │ (config, models, exceptions, dependencies) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ SQLite │ │ Qdrant │ │ OpenAI │ │
│ │ (metadata) │ │ (vectors) │ │ (LLM/Emb) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## Module Structure
```
src/
├── core/ # Foundation layer
│ ├── config.py # Settings from .env
│ ├── models.py # Shared Pydantic models
│ ├── exceptions.py # Custom exceptions
│ └── dependencies.py # Client singletons
│
├── storage/ # Data persistence
│ ├── sqlite.py # Structured data (docs, queries, experiments)
│ └── qdrant.py # Vector embeddings
│
├── documents/ # Document handling
│ ├── parser.py # .docx/.doc → DocumentTree + markdown
│ ├── service.py # Upload, process, delete orchestration
│ ├── routes.py # REST API endpoints
│ └── models.py # Request/response schemas
│
├── chunking/ # Chunking strategies
│ ├── base.py # Abstract base class + utilities
│ ├── embedding.py # OpenAI embedding service
│ ├── service.py # Orchestration (chunk → embed → store)
│ └── strategies/ # Strategy implementations
│ ├── fixed_size.py
│ ├── recursive.py
│ ├── semantic.py
│ ├── contextual_retrieval.py
│ └── semantic_parent_child.py
│
├── benchmarking/ # Evaluation system
│ ├── evaluation.py # LLM-as-Judge scoring
│ ├── benchmark_service.py # Orchestration
│ ├── query_service.py # Single query pipeline
│ ├── routes.py # REST API endpoints
│ ├── models.py # Request/response schemas
│ └── report.py # HTML report generation
│
└── main.py # FastAPI app factory
```
---
## Data Flow
### Document Processing
```
Upload .docx → Parser → DocumentTree + Markdown → SQLite
↓
Process → Strategy.chunk() → Chunks → Embed → Qdrant
```
### Query Pipeline
```
Question → Embed → Qdrant Search → Top-K Chunks → LLM → Answer
```
### Benchmark Pipeline
```
Questions JSON → For each question × strategy:
├── Query Pipeline
└── LLM-as-Judge Evaluation
↓
Aggregate Metrics → SQLite → HTML Report
```
---
## Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| **One collection per strategy** | Enables independent comparison |
| **SQLite for metadata** | Simple, no setup, sufficient scale |
| **Qdrant for vectors** | Purpose-built for vector search |
| **OpenAI for embeddings** | Industry standard, consistent results |
| **LLM-as-Judge for evaluation** | Automated, consistent scoring |
| **Per-strategy failure isolation** | One strategy failure doesn't break others |
| **Text stored in Qdrant payload** | Enables retrieval without extra DB lookups |
---
## Technology Stack
| Component | Technology | Version |
|-----------|------------|---------|
| Framework | FastAPI | 0.100+ |
| ORM/DB | SQLite | built-in |
| Vector DB | Qdrant | 1.18+ |
| Embeddings | OpenAI text-embedding-3-small | - |
| LLM | OpenAI gpt-4o-mini | - |
| Tokenizer | tiktoken (cl100k_base) | - |
| PDF/DOCX | python-docx, LibreOffice | - |
---
## Configuration
All configuration via environment variables (`.env`):
```env
# Database
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
# OpenAI
OPENAI_API_KEY=sk-...
# Chunking
CHUNK_SIZE=512
CHUNK_OVERLAP=50
SEMANTIC_THRESHOLD=0.5
SEMANTIC_MIN_CHUNK_SIZE=5
# Models
EMBEDDING_MODEL=text-embedding-3-small
LLM_MODEL=gpt-4o-mini
```
---
## Error Handling
| Exception | Handler | HTTP Status |
|-----------|---------|-------------|
| ChunkingError | chunking_exception_handler | 400 |
| BenchmarkError | benchmark_exception_handler | 400 |
| QueryError | benchmark_exception_handler | 400 |
| DocumentProcessingError | chunking_exception_handler | 400 |
| QdrantError | chunking_exception_handler | 400 |
| EmbeddingError | chunking_exception_handler | 400 |
---
## Scaling Considerations
| Component | Current | Production Recommendation |
|-----------|---------|---------------------------|
| SQLite | Single file | PostgreSQL for concurrent access |
| Qdrant | Single instance | Qdrant Cloud or cluster |
| OpenAI | Direct API | Connection pooling, rate limiting |
| Workers | Single process | Celery for async processing |
| Caching | None | Redis for repeated queries |

164
docs/configuration.md Normal file
View File

@@ -0,0 +1,164 @@
# Configuration Guide
All settings and environment variables for the RAG Chunking Benchmarker.
---
## Environment Variables
Create a `.env` file in the project root:
```env
# ── Database ──────────────────────────────────────────────────────
# SQLite database path
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
# Qdrant vector database
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
# ── OpenAI ────────────────────────────────────────────────────────
# API key for embeddings and LLM
OPENAI_API_KEY=sk-your-key-here
# ── Chunking Parameters ───────────────────────────────────────────
# Fixed-size strategy
CHUNK_SIZE=512
CHUNK_OVERLAP=50
# Semantic strategy
SEMANTIC_THRESHOLD=0.5
SEMANTIC_MIN_CHUNK_SIZE=5
# ── Models ────────────────────────────────────────────────────────
# Embedding model (used for all strategies)
EMBEDDING_MODEL=text-embedding-3-small
# LLM model (used for contextual retrieval and evaluation)
LLM_MODEL=gpt-4o-mini
```
---
## Parameter Details
### Chunking Parameters
| Parameter | Type | Default | Range | Description |
|-----------|------|---------|-------|-------------|
| `CHUNK_SIZE` | int | 512 | 100-2000 | Target tokens per chunk |
| `CHUNK_OVERLAP` | int | 50 | 0-200 | Token overlap between chunks |
| `SEMANTIC_THRESHOLD` | float | 0.5 | 0.0-1.0 | Cosine similarity threshold for chunk boundaries |
| `SEMANTIC_MIN_CHUNK_SIZE` | int | 5 | 1-20 | Minimum sentences per semantic chunk |
### Model Parameters
| Parameter | Type | Default | Options | Description |
|-----------|------|---------|---------|-------------|
| `EMBEDDING_MODEL` | str | text-embedding-3-small | text-embedding-3-small, text-embedding-3-large | OpenAI embedding model |
| `LLM_MODEL` | str | gpt-4o-mini | gpt-4o-mini, gpt-4o | Model for contextual retrieval and evaluation |
---
## Default Values in Code
```python
# src/core/config.py
class Settings(BaseSettings):
# Database
database_url: str = "sqlite:///./data/chunking_benchmark.db"
qdrant_url: str = "http://localhost:6333"
qdrant_api_key: str = ""
# OpenAI
openai_api_key: str = ""
# Chunking
chunk_size: int = 512
chunk_overlap: int = 50
semantic_threshold: float = 0.5
semantic_min_chunk_size: int = 5
# Models
embedding_model: str = "text-embedding-3-small"
llm_model: str = "gpt-4o-mini"
```
---
## Effect of Parameters
### chunk_size
| Value | Chunks | Quality | Speed |
|-------|--------|---------|-------|
| 256 | More | Higher context loss | Faster |
| 512 | Balanced | Balanced | Balanced |
| 1024 | Fewer | Lower context loss | Slower |
### chunk_overlap
| Value | Context Preservation | Redundancy |
|-------|---------------------|------------|
| 0 | None | None |
| 50 | Moderate | Low |
| 100 | High | High |
### semantic_threshold
| Value | Chunk Size | Topic Sensitivity |
|-------|------------|-------------------|
| 0.3 | Larger | Less sensitive |
| 0.5 | Balanced | Balanced |
| 0.7 | Smaller | More sensitive |
---
## Environment-Specific Configs
### Development
```env
DATABASE_URL=sqlite:///./data/dev.db
QDRANT_URL=http://localhost:6333
LOG_LEVEL=DEBUG
```
### Production
```env
DATABASE_URL=postgresql://user:pass@localhost/dbname
QDRANT_URL=http://qdrant:6333
QDRANT_API_KEY=your-api-key
LOG_LEVEL=WARNING
```
### Testing
```env
DATABASE_URL=sqlite:///./data/test.db
QDRANT_URL=http://localhost:6333
CHUNK_SIZE=100
```
---
## Validation
Settings are validated on startup. Invalid values will cause:
```
pydantic.ValidationError: 1 validation error for Settings
```
Common errors:
| Error | Cause | Fix |
|-------|-------|-----|
| Missing OPENAI_API_KEY | No API key | Add to .env |
| Invalid QDRANT_URL | Wrong URL format | Check URL |
| CHUNK_SIZE < 1 | Too small | Increase value |

385
docs/data-flow.md Normal file
View File

@@ -0,0 +1,385 @@
# Data Flow
How data moves through the RAG Chunking Benchmarker system.
---
## Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ DATA FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ INGESTION │
│ ───────── │
│ .docx/.doc → Parser → DocumentTree + Markdown → SQLite │
│ │
│ PROCESSING │
│ ────────── │
│ Document → Strategy.chunk() → Chunks → Embed → Qdrant │
│ │
│ QUERYING │
│ ───────── │
│ Question → Embed → Qdrant Search → Chunks → LLM → Answer │
│ │
│ BENCHMARKING │
│ ─────────── │
│ Questions → (Query × Strategies) → Evaluation → Report │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## 1. Document Ingestion
### Flow
```
User uploads .docx/.doc
↓
FastAPI receives file
↓
Parser extracts text blocks
↓
Builds DocumentTree (hierarchy)
↓
Generates markdown (flat text)
↓
Stores in SQLite
```
### Data Structures
**Input:** Binary file (.docx/.doc)
**Output:**
```python
ParseResult(
tree=DocumentTree, # Hierarchical structure
markdown=str, # Flat text for chunking
plain_text=str, # Raw text
paragraph_count=int # Number of text blocks
)
```
### Storage
**SQLite:**
```sql
INSERT INTO documents (id, filename, parsed_text, document_tree, created_at)
VALUES (?, ?, ?, ?, ?)
```
---
## 2. Document Processing
### Flow
```
User selects strategies
↓
For each strategy:
↓
Load DocumentTree + markdown from SQLite
↓
Strategy.chunk() → list[Chunk]
↓
Embed chunks via OpenAI
↓
Store in Qdrant collection
↓
Update chunk_counts in SQLite
```
### Data Structures
**Input:**
```python
{
"document_id": "doc-abc123",
"strategies": ["recursive", "fixed_size", "semantic"]
}
```
**Processing:**
```python
# For each strategy
chunks = strategy.chunk(
doc_name="insurance.docx",
tree=document_tree,
markdown=markdown_text
)
embeddings = embed_texts([chunk.text for chunk in chunks])
qdr.upsert_chunks(chunks, embeddings)
```
**Output:**
```python
{
"strategies_completed": [
{"strategy": "recursive", "chunks_produced": 218}
]
}
```
### Storage
**Qdrant:**
```python
{
"id": "uuid5(chunk_id)",
"vector": [0.023, -0.156, ...], # 1536 dimensions
"payload": {
"document_name": "insurance.docx",
"chunk_id": "recursive_doc_000045",
"strategy_name": "recursive",
"text": "chunk content...",
"token_count": 146,
"character_count": 205
}
}
```
---
## 3. Query Pipeline
### Flow
```
User asks question
↓
Embed question via OpenAI
↓
Search Qdrant for similar chunks
↓
Retrieve top-K chunks
↓
Build context from chunks
↓
Generate answer via LLM
↓
Store query result in SQLite
```
### Data Structures
**Input:**
```python
{
"document_id": "doc-abc123",
"strategy": "recursive",
"question": "What are the main topics?",
"top_k": 5
}
```
**Processing:**
```python
# 1. Embed question
question_embedding = embed_single(question)
# 2. Vector search
hits = qdr.search(
strategy="recursive",
query_vector=question_embedding,
top_k=5
)
# 3. Build context
context = "\n".join([hit.text for hit in hits])
# 4. Generate answer
answer = llm.generate(
system="Answer based on context...",
user=f"Context: {context}\nQuestion: {question}"
)
```
**Output:**
```python
{
"answer": "The document covers insurance regulations...",
"retrieved_chunks": [...],
"latency_breakdown": {...},
"token_usage": {...}
}
```
### Storage
**SQLite:**
```sql
INSERT INTO queries (id, document_id, strategy_name, question, answer,
retrieved_chunks, latency_breakdown, token_usage)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
```
---
## 4. Benchmark Pipeline
### Flow
```
Load questions from JSON
↓
For each question × strategy:
↓
Run query pipeline
↓
Evaluate with LLM-as-Judge
↓
Store per-question results
↓
Aggregate metrics per strategy
↓
Store experiment in SQLite
↓
Generate HTML report
```
### Data Structures
**Input:**
```python
{
"document_id": "doc-abc123",
"strategies": ["recursive", "fixed_size"],
"questions_file": "files/questions.json"
}
```
**Processing:**
```python
# For each question
for question in questions:
for strategy in strategies:
# Query
result = run_query(question, strategy)
# Evaluate
scores = evaluate_single(
question=question,
context=result.retrieved_chunks,
expected=question.expected_answer,
generated=result.answer
)
# Store
per_question_results.append({...})
```
**Output:**
```python
{
"experiment_id": "exp-abc123",
"aggregate_metrics": {
"recursive": {
"avg_context_relevance": 8.5,
"avg_answer_similarity": 7.8,
"avg_faithfulness": 9.2,
"hallucination_rate": 0.05
}
},
"best_strategy": "recursive"
}
```
---
## 5. Report Generation
### Flow
```
Load experiment from SQLite
↓
Calculate rankings
↓
Generate HTML (managerial or technical view)
↓
Return HTML response
```
### Views
| View | Focus | Content |
|------|-------|---------|
| Managerial | Decision-making | KPIs, winner, recommendations |
| Technical | Observability | Detailed scores, token usage, per-question |
---
## Data Persistence
### SQLite Tables
| Table | Purpose | Key Fields |
|-------|---------|------------|
| documents | Uploaded documents | id, filename, parsed_text, document_tree |
| queries | Query results | id, document_id, question, answer |
| experiments | Benchmark results | id, document_id, aggregate_metrics |
### Qdrant Collections
| Collection | Purpose | Fields |
|------------|---------|--------|
| recursive_collection | Recursive strategy vectors | chunk_id, text, scores |
| fixed_size_collection | Fixed-size strategy vectors | chunk_id, text, scores |
| semantic_collection | Semantic strategy vectors | chunk_id, text, scores |
| contextual_retrieval_collection | Contextual strategy vectors | chunk_id, text, scores |
| semantic_parent_child_collection | Parent-child strategy vectors | chunk_id, text, parent_id |
---
## Data Transformation
### Document → Chunks
```
Input: Full document text (10,000 tokens)
Output: 50-300 chunks (100-500 tokens each)
Transformation:
- Tokenization
- Boundary detection
- Metadata enrichment
```
### Chunks → Vectors
```
Input: Chunk text
Output: 1536-dimensional vector
Transformation:
- Text → Tokens → Embedding API → Vector
```
### Question → Answer
```
Input: Question string
Output: Answer string + metadata
Transformation:
- Question → Embedding → Vector Search → Chunks → LLM → Answer
```
---
## Performance Characteristics
| Operation | Time | Tokens | Cost |
|-----------|------|--------|------|
| Document upload | ~2s | 0 | Free |
| Process (1 strategy) | ~5s | N/A | ~$0.001 (embeddings) |
| Process (5 strategies) | ~25s | N/A | ~$0.005 |
| Query | ~2s | ~1500 | ~$0.001 |
| Benchmark (21 questions) | ~120s | ~200,000 | ~$0.05 |

222
docs/evaluation-metrics.md Normal file
View File

@@ -0,0 +1,222 @@
# Evaluation Metrics
How scoring works in the benchmarking system.
---
## Overview
The system uses **LLM-as-Judge** evaluation to automatically score answer quality across 4 metrics.
---
## Metrics
### 1. Context Relevance (1-10)
**Question:** How relevant are the retrieved chunks to answering the question?
| Score | Meaning |
|-------|---------|
| 9-10 | Chunks directly answer the question |
| 7-8 | Chunks are highly relevant |
| 5-6 | Chunks are somewhat relevant |
| 3-4 | Chunks are partially relevant |
| 1-2 | Chunks are not relevant |
**What it measures:** Did the retrieval system find the right information?
---
### 2. Answer Similarity (1-10)
**Question:** How similar is the generated answer to the expected answer?
| Score | Meaning |
|-------|---------|
| 9-10 | Nearly identical to expected |
| 7-8 | Covers most key points |
| 5-6 | Covers some key points |
| 3-4 | Partially correct |
| 1-2 | Completely different |
**What it measures:** Did the system produce the right answer?
---
### 3. Faithfulness (1-10)
**Question:** Is the generated answer grounded in the retrieved context?
| Score | Meaning |
|-------|---------|
| 9-10 | Entirely based on context |
| 7-8 | Mostly based on context |
| 5-6 | Partially based on context |
| 3-4 | Some external knowledge used |
| 1-2 | Mostly external knowledge |
**What it measures:** Is the answer trustworthy?
---
### 4. Hallucination (boolean)
**Question:** Did the LLM invent information not in the context?
| Value | Meaning |
|-------|---------|
| false | Answer is grounded in context |
| true | Answer contains invented information |
**What it measures:** Is the answer fabricated?
---
## Scoring Process
### Step 1: Prepare Evaluation Context
```
Question: {user_question}
Context: {retrieved_chunks}
Expected Answer: {golden_answer}
Generated Answer: {system_answer}
```
### Step 2: Send to LLM-as-Judge
```python
prompt = f"""
Evaluate this RAG system output:
Question: {question}
Context: {context}
Expected: {expected}
Generated: {generated}
Return JSON:
{{
"context_relevance": <1-10>,
"answer_similarity": <1-10>,
"faithfulness": <1-10>,
"hallucination": <true/false>,
"reasoning": "<explanation>"
}}
"""
```
### Step 3: Parse Response
```python
scores = json.loads(llm_response)
# Validate ranges
for metric in ["context_relevance", "answer_similarity", "faithfulness"]:
scores[metric] = max(1, min(10, scores[metric]))
```
---
## Overall Score Calculation
Each strategy gets an overall score weighted by importance:
```python
overall = (
context_relevance * 0.3 + # 30% weight
answer_similarity * 0.4 + # 40% weight
faithfulness * 0.3 # 30% weight
) * (1 - hallucination_rate) # Hallucination penalty
```
### Why These Weights?
| Metric | Weight | Rationale |
|--------|--------|-----------|
| Answer Similarity | 40% | Most important - did we get the right answer? |
| Context Relevance | 30% | Found the right information |
| Faithfulness | 30% | Answer is trustworthy |
| Hallucination | Penalty | Fabricated info is unacceptable |
---
## Aggregate Metrics
Per strategy, we calculate:
| Metric | Calculation |
|--------|-------------|
| avg_context_relevance | mean of all context_relevance scores |
| avg_answer_similarity | mean of all answer_similarity scores |
| avg_faithfulness | mean of all faithfulness scores |
| hallucination_rate | count(hallucination=True) / total_questions |
| total_questions | number of questions evaluated |
| failed_questions | questions that errored |
---
## Example Output
```json
{
"recursive": {
"avg_context_relevance": 8.5,
"avg_answer_similarity": 7.8,
"avg_faithfulness": 9.2,
"hallucination_rate": 0.05,
"total_questions": 21,
"failed_questions": 0
}
}
```
---
## Interpretation Guide
### Good Scores
| Metric | Target | Meaning |
|--------|--------|---------|
| Context Relevance | ≥ 8 | Retrieval is accurate |
| Answer Similarity | ≥ 8 | Answers match expectations |
| Faithfulness | ≥ 9 | Answers are trustworthy |
| Hallucination Rate | ≤ 0.1 | Low fabrication rate |
### Warning Signs
| Metric | Warning | Meaning |
|--------|---------|---------|
| Context Relevance | < 6 | Retrieval needs improvement |
| Answer Similarity | < 6 | Answers are off-target |
| Faithfulness | < 7 | Model is adding external knowledge |
| Hallucination Rate | > 0.2 | High fabrication rate |
---
## Cost Considerations
Each evaluation uses:
| Component | Cost |
|-----------|------|
| LLM call (gpt-4o-mini) | ~$0.001 per evaluation |
| Input tokens | ~500 per evaluation |
| Output tokens | ~100 per evaluation |
**Total for 21 questions × 5 strategies:**
- 105 evaluations × $0.001 = **$0.105**
---
## Configuration
Evaluation parameters in `src/benchmarking/evaluation.py`:
```python
_EVALUATION_SYSTEM_PROMPT = "You are an expert evaluator..."
_EVALUATION_USER_PROMPT = "Evaluate this RAG system output..."
model = settings.llm_model # gpt-4o-mini
temperature = 0.0 # Deterministic
max_tokens = 500 # Response limit
```

345
docs/strategy-technical-details.md Normal file
View File

@@ -0,0 +1,345 @@
# Strategy Technical Details
Deep technical explanation of each chunking strategy's implementation.
---
## 1. Fixed-Size Chunking
**File:** `src/chunking/strategies/fixed_size.py`
### How It Works
Splits text into fixed-size token chunks with overlap.
### Splitting Unit
**Token-level** using `tiktoken` (cl100k_base encoding).
### Algorithm
```
1. Encode full markdown → token array
2. Take first chunk_size tokens → chunk 1
3. Slide forward by (chunk_size - overlap) tokens
4. Repeat until end of text
```
### Parameters
| Parameter | Config Key | Default | Description |
|-----------|------------|---------|-------------|
| Chunk size | `chunk_size` | 512 | Tokens per chunk |
| Overlap | `chunk_overlap` | 50 | Token overlap between chunks |
### Code Logic
```python
tokens = _encoder.encode(text)
start = 0
while start < len(tokens):
end = min(start + chunk_size, len(tokens))
chunk_tokens = tokens[start:end]
chunks.append(_encoder.decode(chunk_tokens))
start = end - overlap
```
### Characteristics
- **Splitting boundary:** Arbitrary (mid-sentence possible)
- **Structure awareness:** None
- **Speed:** Fastest (no API calls)
- **Best for:** Baseline comparison, simple documents
---
## 2. Recursive Chunking
**File:** `src/chunking/strategies/recursive.py`
### How It Works
Cascade splitting using a separator hierarchy. Tries meaningful boundaries first, falls back to less meaningful ones.
### Splitting Unit
**Character-level** with regex patterns.
### Separator Cascade (Priority Order)
| Priority | Separator | Regex Pattern | Description |
|----------|-----------|---------------|-------------|
| 1 | Header | `^(#{1,6})\s+` | Markdown headers |
| 2 | Double newline | `\n\n` | Paragraph breaks |
| 3 | Single newline | `\n` | Line breaks |
| 4 | Sentence | `[.!?]\s+` | Sentence endings |
| 5 | Space | `\s+` | Word boundaries |
### Algorithm
```
1. Try splitting by highest-priority separator
2. If parts still too big → recurse with next separator
3. Merge small parts back up to target size
4. If single part exceeds target → hard split by character count
```
### Code Logic
```python
for sep_name, pattern in _SEPARATORS:
parts = pattern.split(text)
if len(parts) <= 1:
continue # this separator didn't split
# Merge parts back up to target_size
current = ""
for part in parts:
candidate = (current + " " + part).strip()
if len(candidate) <= target_size:
current = candidate
else:
chunks.append(current)
current = part
return chunks # Return on first successful split
```
### Characteristics
- **Splitting boundary:** Semantic (headers, paragraphs, sentences)
- **Structure awareness:** High (respects markdown structure)
- **Speed:** Fast (no API calls)
- **Best for:** Structured documents with clear hierarchy
---
## 3. Semantic Chunking
**File:** `src/chunking/strategies/semantic.py`
### How It Works
Groups sentences by semantic similarity. When similarity drops, starts a new chunk.
### Splitting Unit
**Sentence-level** with similarity-based boundaries.
### Algorithm
```
1. Split markdown into sentences
2. Embed each sentence via OpenAI (text-embedding-3-small)
3. Compute cosine similarity between adjacent sentences
4. When similarity < SEMANTIC_THRESHOLD → chunk boundary
5. Enforce SEMANTIC_MIN_CHUNK_SIZE (minimum sentences per chunk)
```
### Parameters
| Parameter | Config Key | Default | Description |
|-----------|------------|---------|-------------|
| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for boundary |
| Min size | `semantic_min_chunk_size` | 5 | Minimum sentences per chunk |
### Code Logic
```python
sentences = split_sentences(markdown)
embeddings = embed_texts(sentences) # OpenAI API call
current_group = [sentences[0]]
for i in range(1, len(sentences)):
sim = cosine_similarity(embeddings[i-1], embeddings[i])
if sim < threshold and len(current_group) >= min_size:
# Topic shift → new chunk
chunks.append(" ".join(current_group))
current_group = [sentences[i]]
else:
current_group.append(sentences[i])
```
### Characteristics
- **Splitting boundary:** Semantic (topic changes)
- **Structure awareness:** Medium (sentence-level)
- **Speed:** Medium (requires embeddings)
- **Best for:** Documents with topic shifts, no clear structure
---
## 4. Contextual Retrieval
**File:** `src/chunking/strategies/contextual_retrieval.py`
### How It Works
Based on Anthropic's 2024 research. Prepends a short context summary to each chunk before embedding.
### Splitting Unit
**Token-level** (same as fixed-size).
### Algorithm
```
1. Split text using fixed-size token splitting
2. For each chunk:
a. Find surrounding text (500 chars before/after)
b. Send to LLM with context prompt
c. LLM generates 1-2 sentence context prefix
d. Enriched chunk = context + original text
3. Embed enriched chunks (not original)
```
### LLM Prompt
```
System: You are a document analysis assistant. Given a section of text
from a document, write a short context prefix (1-2 sentences) that
would help someone find this section later via search.
User:
Preceding text: {preceding_500_chars}
This section: {chunk_text}
Following text: {following_500_chars}
```
### Code Logic
```python
for chunk_text in raw_chunks:
# Find surrounding context
preceding = full_text[pos-500:pos]
following = full_text[pos+len:pos+len+500]
# Generate context prefix
context = llm.generate(
system=CONTEXT_PROMPT,
user=f"Preceding: {preceding}\nSection: {chunk_text}\nFollowing: {following}"
)
# Enrich chunk
enriched = f"{context}\n\n{chunk_text}"
chunks.append(enriched)
```
### Characteristics
- **Splitting boundary:** Token-based (like fixed-size)
- **Structure awareness:** None (relies on LLM for context)
- **Speed:** Slowest (1 LLM call per chunk)
- **Best for:** Improving retrieval quality, unstructured documents
---
## 5. Semantic Parent-Child
**File:** `src/chunking/strategies/semantic_parent_child.py`
### How It Works
Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
### Splitting Unit
**Paragraph-level** with semantic clustering.
### Algorithm
```
1. Split markdown into paragraphs
2. Embed each paragraph
3. Cluster consecutive paragraphs by similarity
4. Each cluster = parent chunk (full cluster text)
5. Each paragraph = child chunk (linked to parent)
```
### Query-Time Behavior
```
1. Vector search finds matching child paragraph
2. Use parent_id to fetch parent cluster
3. Return both child + parent to LLM
```
### Parameters
| Parameter | Config Key | Default | Description |
|-----------|------------|---------|-------------|
| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for clustering |
### Code Logic
```python
paragraphs = split_paragraphs(markdown)
embeddings = embed_texts(paragraphs)
# Cluster paragraphs
clusters = [[0]]
for i in range(1, len(paragraphs)):
sim = cosine_similarity(embeddings[i-1], embeddings[i])
if sim >= threshold:
clusters[-1].append(i) # Same cluster
else:
clusters.append([i]) # New cluster
# Create parent-child chunks
for cluster in clusters:
parent_text = "\n\n".join(paragraphs[i] for i in cluster)
parent_id = make_chunk_id(...)
for para_idx in cluster:
chunks.append(build_chunk(
text=paragraphs[para_idx],
parent_id=parent_id
))
```
### Characteristics
- **Splitting boundary:** Semantic (paragraph clusters)
- **Structure awareness:** Medium (paragraph-level)
- **Speed:** Medium (requires embeddings)
- **Best for:** Documents needing context, flat structure
---
## Comparison Table
| Strategy | Split Unit | Boundary Logic | API Calls | Speed | Best For |
|----------|------------|----------------|-----------|-------|----------|
| fixed_size | Token | Arbitrary | 0 | Fast | Baseline |
| recursive | Character | Structural | 0 | Fast | Structured docs |
| semantic | Sentence | Similarity | 1 (embeddings) | Medium | Topic shifts |
| contextual_retrieval | Token | LLM-generated | N (1 per chunk) | Slow | Retrieval quality |
| semantic_parent_child | Paragraph | Similarity | 1 (embeddings) | Medium | Context needed |
---
## Configuration Reference
All strategy parameters are in `src/core/config.py`:
```python
class Settings(BaseSettings):
chunk_size: int = 512
chunk_overlap: int = 50
semantic_threshold: float = 0.5
semantic_min_chunk_size: int = 5
embedding_model: str = "text-embedding-3-small"
llm_model: str = "gpt-4o-mini"
```
---
## ADR References
| ADR | Strategy | Decision |
|-----|----------|----------|
| ADR 0012 | semantic | Sentence-level with min chunk size |
| ADR 0015 | recursive | Direct API, no LangChain/LlamaIndex |
| ADR 0011 | contextual_retrieval | Embed enriched text, not original |
| ADR 0003 | all | Per-strategy failure isolation |