docs: add comprehensive project documentation
Why: - Need documentation for team onboarding and reference - Need technical details for strategy implementations - Need API reference for developers Changes: - README.md: Documentation index and quick start guide - api-reference.md: All 10 endpoints with examples - architecture.md: System structure and design decisions - configuration.md: All environment variables and parameters - data-flow.md: How data moves through the system - evaluation-metrics.md: How scoring works with weights - strategy-technical-details.md: Deep dive into each strategy's implementation
This commit is contained in:
103
docs/README.md
Normal file
103
docs/README.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Documentation Index
|
||||
|
||||
Complete documentation for the RAG Chunking Benchmarker.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **New to the project?** Start with [Architecture Overview](architecture.md)
|
||||
2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
|
||||
3. **Need to use the API?** Check [API Reference](api-reference.md)
|
||||
4. **Configuring the system?** See [Configuration Guide](configuration.md)
|
||||
5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
|
||||
6. **Curious about data flow?** See [Data Flow](data-flow.md)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Files
|
||||
|
||||
| File | Purpose | Audience |
|
||||
|------|---------|----------|
|
||||
| [architecture.md](architecture.md) | System structure and design | New team members |
|
||||
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
|
||||
| [api-reference.md](api-reference.md) | All endpoints documented | Developers |
|
||||
| [configuration.md](configuration.md) | Settings and environment variables | DevOps |
|
||||
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
|
||||
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
|
||||
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
|
||||
| [phases.md](phases.md) | Implementation phases | Project managers |
|
||||
| [tasks.md](tasks.md) | Task tracking | Developers |
|
||||
|
||||
---
|
||||
|
||||
## Architecture at a Glance
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ RAG Chunking Benchmarker │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Documents → Parser → Chunking → Embedding → Qdrant │
|
||||
│ │
|
||||
│ Questions → Query Pipeline → LLM → Answers │
|
||||
│ │
|
||||
│ Answers → Evaluation → Metrics → Reports │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### Chunking Strategies
|
||||
|
||||
| Strategy | Splitting | Best For |
|
||||
|----------|-----------|----------|
|
||||
| fixed_size | Token count | Baseline comparison |
|
||||
| recursive | Structural boundaries | Structured documents |
|
||||
| semantic | Similarity thresholds | Topic shifts |
|
||||
| contextual_retrieval | LLM-enriched tokens | Retrieval quality |
|
||||
| semantic_parent_child | Paragraph clusters | Context needed |
|
||||
|
||||
### Evaluation Metrics
|
||||
|
||||
| Metric | What It Measures |
|
||||
|--------|------------------|
|
||||
| Context Relevance | Did we find the right information? |
|
||||
| Answer Similarity | Did we produce the right answer? |
|
||||
| Faithfulness | Is the answer trustworthy? |
|
||||
| Hallucination | Did we invent information? |
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/documents` | GET/POST | Manage documents |
|
||||
| `/documents/{id}/process` | POST | Run chunking strategies |
|
||||
| `/queries` | POST | Ask questions |
|
||||
| `/benchmarks` | POST | Run comparisons |
|
||||
| `/benchmarks/{id}/report` | GET | View results |
|
||||
| `/experiments` | GET | List experiments |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All settings via `.env` file. See [Configuration Guide](configuration.md) for details.
|
||||
|
||||
Key settings:
|
||||
- `OPENAI_API_KEY` - Required for embeddings and LLM
|
||||
- `CHUNK_SIZE` - Target tokens per chunk (default: 512)
|
||||
- `SEMANTIC_THRESHOLD` - Similarity threshold (default: 0.5)
|
||||
|
||||
---
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Phases](phases.md) - Implementation roadmap
|
||||
- [Tasks](tasks.md) - Task tracking
|
||||
- [ADRs](adr/) - Architectural Decision Records
|
||||
309
docs/api-reference.md
Normal file
309
docs/api-reference.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# API Reference
|
||||
|
||||
Complete documentation of all REST API endpoints.
|
||||
|
||||
---
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
http://localhost:8000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Documents
|
||||
|
||||
#### `GET /documents`
|
||||
|
||||
List all uploaded documents.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "doc-abc123",
|
||||
"filename": "insurance.docx",
|
||||
"paragraph_count": 245,
|
||||
"chunk_counts": {
|
||||
"recursive": 218,
|
||||
"fixed_size": 117
|
||||
},
|
||||
"created_at": "2026-07-25T10:00:00"
|
||||
}
|
||||
],
|
||||
"total": 1,
|
||||
"offset": 0,
|
||||
"limit": 50
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `POST /documents`
|
||||
|
||||
Upload a document (.docx or .doc).
|
||||
|
||||
**Request:**
|
||||
- Content-Type: `multipart/form-data`
|
||||
- Body: `file` (binary)
|
||||
|
||||
**Response (201):**
|
||||
```json
|
||||
{
|
||||
"id": "doc-abc123",
|
||||
"filename": "insurance.docx",
|
||||
"paragraph_count": 245,
|
||||
"chunk_counts": {},
|
||||
"created_at": "2026-07-25T10:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `DELETE /documents/{doc_id}`
|
||||
|
||||
Delete a document and its vectors.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"deleted": true,
|
||||
"document_id": "doc-abc123"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `POST /documents/{doc_id}/process`
|
||||
|
||||
Run chunking strategies on a document.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"strategies": [
|
||||
"recursive",
|
||||
"fixed_size",
|
||||
"semantic",
|
||||
"contextual_retrieval",
|
||||
"semantic_parent_child"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategies_completed": [
|
||||
{
|
||||
"strategy": "recursive",
|
||||
"status": "completed",
|
||||
"chunks_produced": 218
|
||||
}
|
||||
],
|
||||
"strategies_failed": []
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Strategies
|
||||
|
||||
#### `GET /strategies`
|
||||
|
||||
List all available chunking strategies.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"strategies": [
|
||||
{
|
||||
"name": "recursive",
|
||||
"description": "Cascade splitting: headers > double newline > ..."
|
||||
},
|
||||
{
|
||||
"name": "fixed_size",
|
||||
"description": "Fixed-size token splitting with overlap"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Queries
|
||||
|
||||
#### `POST /queries`
|
||||
|
||||
Ask a question against a document using a specific strategy.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategy": "recursive",
|
||||
"question": "What are the main topics?",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
|
||||
**Response (201):**
|
||||
```json
|
||||
{
|
||||
"query_id": "q-xyz789",
|
||||
"document_id": "doc-abc123",
|
||||
"strategy": "recursive",
|
||||
"question": "What are the main topics?",
|
||||
"answer": "The document covers insurance regulations...",
|
||||
"retrieved_chunks": [
|
||||
{
|
||||
"chunk_id": "recursive_doc_000045",
|
||||
"score": 0.892,
|
||||
"text": " chunk content...",
|
||||
"parent_id": null
|
||||
}
|
||||
],
|
||||
"latency_breakdown": {
|
||||
"embed_seconds": 0.15,
|
||||
"search_seconds": 0.02,
|
||||
"answer_seconds": 1.23,
|
||||
"total_seconds": 1.40
|
||||
},
|
||||
"token_usage": {
|
||||
"prompt_tokens": 1250,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 1400
|
||||
},
|
||||
"created_at": "2026-07-25T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `GET /queries/{query_id}`
|
||||
|
||||
Retrieve a past query.
|
||||
|
||||
**Response:** Same as POST /queries response.
|
||||
|
||||
---
|
||||
|
||||
### Benchmarks
|
||||
|
||||
#### `POST /benchmarks`
|
||||
|
||||
Run a benchmark comparing multiple strategies.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategies": ["recursive", "fixed_size", "semantic"],
|
||||
"questions_file": "files/questions.json",
|
||||
"top_k": 5,
|
||||
"dry_run": false
|
||||
}
|
||||
```
|
||||
|
||||
**Response (201):**
|
||||
```json
|
||||
{
|
||||
"experiment_id": "exp-abc123",
|
||||
"document_id": "doc-abc123",
|
||||
"strategies_used": ["recursive", "fixed_size", "semantic"],
|
||||
"questions_count": 21,
|
||||
"aggregate_metrics": {
|
||||
"recursive": {
|
||||
"avg_context_relevance": 8.5,
|
||||
"avg_answer_similarity": 7.8,
|
||||
"avg_faithfulness": 9.2,
|
||||
"hallucination_rate": 0.05,
|
||||
"total_questions": 21,
|
||||
"failed_questions": 0
|
||||
}
|
||||
},
|
||||
"best_strategy": "recursive",
|
||||
"total_latency_seconds": 120.5,
|
||||
"estimated_cost_usd": 0.22,
|
||||
"created_at": "2026-07-25T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `GET /benchmarks/{experiment_id}`
|
||||
|
||||
Retrieve experiment results.
|
||||
|
||||
**Response:** Same as POST /benchmarks response.
|
||||
|
||||
---
|
||||
|
||||
#### `GET /benchmarks/{experiment_id}/report`
|
||||
|
||||
Generate HTML report.
|
||||
|
||||
**Query Parameters:**
|
||||
- `view`: `managerial` (default) or `technical`
|
||||
|
||||
**Response:** HTML page
|
||||
|
||||
---
|
||||
|
||||
#### `GET /experiments`
|
||||
|
||||
List all experiments.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "exp-abc123",
|
||||
"document_id": "doc-abc123",
|
||||
"strategies_used": ["recursive", "fixed_size"],
|
||||
"created_at": "2026-07-25T12:00:00"
|
||||
}
|
||||
],
|
||||
"total": 1,
|
||||
"offset": 0,
|
||||
"limit": 50
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Responses
|
||||
|
||||
All errors return:
|
||||
```json
|
||||
{
|
||||
"detail": "Error message",
|
||||
"type": "ErrorClassName"
|
||||
}
|
||||
```
|
||||
|
||||
| Status Code | Error Type | Description |
|
||||
|-------------|------------|-------------|
|
||||
| 400 | ChunkingError | Invalid request |
|
||||
| 400 | QueryError | Query failed |
|
||||
| 400 | BenchmarkError | Benchmark failed |
|
||||
| 404 | QueryError | Resource not found |
|
||||
| 422 | ValidationError | Invalid request body |
|
||||
|
||||
---
|
||||
|
||||
## Rate Limits
|
||||
|
||||
None configured. For production, consider adding rate limiting.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
None configured. For production, add API key or OAuth2 authentication.
|
||||
183
docs/architecture.md
Normal file
183
docs/architecture.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Architecture Overview
|
||||
|
||||
System structure and design decisions for the RAG Chunking Benchmarker.
|
||||
|
||||
---
|
||||
|
||||
## System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ RAG Chunking Benchmarker │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Documents │ │ Chunking │ │ Benchmarking│ │
|
||||
│ │ Module │ │ Module │ │ Module │ │
|
||||
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Layer │ │
|
||||
│ │ (config, models, exceptions, dependencies) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ SQLite │ │ Qdrant │ │ OpenAI │ │
|
||||
│ │ (metadata) │ │ (vectors) │ │ (LLM/Emb) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── core/ # Foundation layer
|
||||
│ ├── config.py # Settings from .env
|
||||
│ ├── models.py # Shared Pydantic models
|
||||
│ ├── exceptions.py # Custom exceptions
|
||||
│ └── dependencies.py # Client singletons
|
||||
│
|
||||
├── storage/ # Data persistence
|
||||
│ ├── sqlite.py # Structured data (docs, queries, experiments)
|
||||
│ └── qdrant.py # Vector embeddings
|
||||
│
|
||||
├── documents/ # Document handling
|
||||
│ ├── parser.py # .docx/.doc → DocumentTree + markdown
|
||||
│ ├── service.py # Upload, process, delete orchestration
|
||||
│ ├── routes.py # REST API endpoints
|
||||
│ └── models.py # Request/response schemas
|
||||
│
|
||||
├── chunking/ # Chunking strategies
|
||||
│ ├── base.py # Abstract base class + utilities
|
||||
│ ├── embedding.py # OpenAI embedding service
|
||||
│ ├── service.py # Orchestration (chunk → embed → store)
|
||||
│ └── strategies/ # Strategy implementations
|
||||
│ ├── fixed_size.py
|
||||
│ ├── recursive.py
|
||||
│ ├── semantic.py
|
||||
│ ├── contextual_retrieval.py
|
||||
│ └── semantic_parent_child.py
|
||||
│
|
||||
├── benchmarking/ # Evaluation system
|
||||
│ ├── evaluation.py # LLM-as-Judge scoring
|
||||
│ ├── benchmark_service.py # Orchestration
|
||||
│ ├── query_service.py # Single query pipeline
|
||||
│ ├── routes.py # REST API endpoints
|
||||
│ ├── models.py # Request/response schemas
|
||||
│ └── report.py # HTML report generation
|
||||
│
|
||||
└── main.py # FastAPI app factory
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Document Processing
|
||||
|
||||
```
|
||||
Upload .docx → Parser → DocumentTree + Markdown → SQLite
|
||||
↓
|
||||
Process → Strategy.chunk() → Chunks → Embed → Qdrant
|
||||
```
|
||||
|
||||
### Query Pipeline
|
||||
|
||||
```
|
||||
Question → Embed → Qdrant Search → Top-K Chunks → LLM → Answer
|
||||
```
|
||||
|
||||
### Benchmark Pipeline
|
||||
|
||||
```
|
||||
Questions JSON → For each question × strategy:
|
||||
├── Query Pipeline
|
||||
└── LLM-as-Judge Evaluation
|
||||
↓
|
||||
Aggregate Metrics → SQLite → HTML Report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| **One collection per strategy** | Enables independent comparison |
|
||||
| **SQLite for metadata** | Simple, no setup, sufficient scale |
|
||||
| **Qdrant for vectors** | Purpose-built for vector search |
|
||||
| **OpenAI for embeddings** | Industry standard, consistent results |
|
||||
| **LLM-as-Judge for evaluation** | Automated, consistent scoring |
|
||||
| **Per-strategy failure isolation** | One strategy failure doesn't break others |
|
||||
| **Text stored in Qdrant payload** | Enables retrieval without extra DB lookups |
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
|
||||
| Component | Technology | Version |
|
||||
|-----------|------------|---------|
|
||||
| Framework | FastAPI | 0.100+ |
|
||||
| ORM/DB | SQLite | built-in |
|
||||
| Vector DB | Qdrant | 1.18+ |
|
||||
| Embeddings | OpenAI text-embedding-3-small | - |
|
||||
| LLM | OpenAI gpt-4o-mini | - |
|
||||
| Tokenizer | tiktoken (cl100k_base) | - |
|
||||
| PDF/DOCX | python-docx, LibreOffice | - |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration via environment variables (`.env`):
|
||||
|
||||
```env
|
||||
# Database
|
||||
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
|
||||
QDRANT_URL=http://localhost:6333
|
||||
QDRANT_API_KEY=
|
||||
|
||||
# OpenAI
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# Chunking
|
||||
CHUNK_SIZE=512
|
||||
CHUNK_OVERLAP=50
|
||||
SEMANTIC_THRESHOLD=0.5
|
||||
SEMANTIC_MIN_CHUNK_SIZE=5
|
||||
|
||||
# Models
|
||||
EMBEDDING_MODEL=text-embedding-3-small
|
||||
LLM_MODEL=gpt-4o-mini
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Exception | Handler | HTTP Status |
|
||||
|-----------|---------|-------------|
|
||||
| ChunkingError | chunking_exception_handler | 400 |
|
||||
| BenchmarkError | benchmark_exception_handler | 400 |
|
||||
| QueryError | benchmark_exception_handler | 400 |
|
||||
| DocumentProcessingError | chunking_exception_handler | 400 |
|
||||
| QdrantError | chunking_exception_handler | 400 |
|
||||
| EmbeddingError | chunking_exception_handler | 400 |
|
||||
|
||||
---
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
| Component | Current | Production Recommendation |
|
||||
|-----------|---------|---------------------------|
|
||||
| SQLite | Single file | PostgreSQL for concurrent access |
|
||||
| Qdrant | Single instance | Qdrant Cloud or cluster |
|
||||
| OpenAI | Direct API | Connection pooling, rate limiting |
|
||||
| Workers | Single process | Celery for async processing |
|
||||
| Caching | None | Redis for repeated queries |
|
||||
164
docs/configuration.md
Normal file
164
docs/configuration.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Configuration Guide
|
||||
|
||||
All settings and environment variables for the RAG Chunking Benchmarker.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Create a `.env` file in the project root:
|
||||
|
||||
```env
|
||||
# ── Database ──────────────────────────────────────────────────────
|
||||
|
||||
# SQLite database path
|
||||
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
|
||||
|
||||
# Qdrant vector database
|
||||
QDRANT_URL=http://localhost:6333
|
||||
QDRANT_API_KEY=
|
||||
|
||||
# ── OpenAI ────────────────────────────────────────────────────────
|
||||
|
||||
# API key for embeddings and LLM
|
||||
OPENAI_API_KEY=sk-your-key-here
|
||||
|
||||
# ── Chunking Parameters ───────────────────────────────────────────
|
||||
|
||||
# Fixed-size strategy
|
||||
CHUNK_SIZE=512
|
||||
CHUNK_OVERLAP=50
|
||||
|
||||
# Semantic strategy
|
||||
SEMANTIC_THRESHOLD=0.5
|
||||
SEMANTIC_MIN_CHUNK_SIZE=5
|
||||
|
||||
# ── Models ────────────────────────────────────────────────────────
|
||||
|
||||
# Embedding model (used for all strategies)
|
||||
EMBEDDING_MODEL=text-embedding-3-small
|
||||
|
||||
# LLM model (used for contextual retrieval and evaluation)
|
||||
LLM_MODEL=gpt-4o-mini
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Parameter Details
|
||||
|
||||
### Chunking Parameters
|
||||
|
||||
| Parameter | Type | Default | Range | Description |
|
||||
|-----------|------|---------|-------|-------------|
|
||||
| `CHUNK_SIZE` | int | 512 | 100-2000 | Target tokens per chunk |
|
||||
| `CHUNK_OVERLAP` | int | 50 | 0-200 | Token overlap between chunks |
|
||||
| `SEMANTIC_THRESHOLD` | float | 0.5 | 0.0-1.0 | Cosine similarity threshold for chunk boundaries |
|
||||
| `SEMANTIC_MIN_CHUNK_SIZE` | int | 5 | 1-20 | Minimum sentences per semantic chunk |
|
||||
|
||||
### Model Parameters
|
||||
|
||||
| Parameter | Type | Default | Options | Description |
|
||||
|-----------|------|---------|---------|-------------|
|
||||
| `EMBEDDING_MODEL` | str | text-embedding-3-small | text-embedding-3-small, text-embedding-3-large | OpenAI embedding model |
|
||||
| `LLM_MODEL` | str | gpt-4o-mini | gpt-4o-mini, gpt-4o | Model for contextual retrieval and evaluation |
|
||||
|
||||
---
|
||||
|
||||
## Default Values in Code
|
||||
|
||||
```python
|
||||
# src/core/config.py
|
||||
class Settings(BaseSettings):
|
||||
# Database
|
||||
database_url: str = "sqlite:///./data/chunking_benchmark.db"
|
||||
qdrant_url: str = "http://localhost:6333"
|
||||
qdrant_api_key: str = ""
|
||||
|
||||
# OpenAI
|
||||
openai_api_key: str = ""
|
||||
|
||||
# Chunking
|
||||
chunk_size: int = 512
|
||||
chunk_overlap: int = 50
|
||||
semantic_threshold: float = 0.5
|
||||
semantic_min_chunk_size: int = 5
|
||||
|
||||
# Models
|
||||
embedding_model: str = "text-embedding-3-small"
|
||||
llm_model: str = "gpt-4o-mini"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Effect of Parameters
|
||||
|
||||
### chunk_size
|
||||
|
||||
| Value | Chunks | Quality | Speed |
|
||||
|-------|--------|---------|-------|
|
||||
| 256 | More | Higher context loss | Faster |
|
||||
| 512 | Balanced | Balanced | Balanced |
|
||||
| 1024 | Fewer | Lower context loss | Slower |
|
||||
|
||||
### chunk_overlap
|
||||
|
||||
| Value | Context Preservation | Redundancy |
|
||||
|-------|---------------------|------------|
|
||||
| 0 | None | None |
|
||||
| 50 | Moderate | Low |
|
||||
| 100 | High | High |
|
||||
|
||||
### semantic_threshold
|
||||
|
||||
| Value | Chunk Size | Topic Sensitivity |
|
||||
|-------|------------|-------------------|
|
||||
| 0.3 | Larger | Less sensitive |
|
||||
| 0.5 | Balanced | Balanced |
|
||||
| 0.7 | Smaller | More sensitive |
|
||||
|
||||
---
|
||||
|
||||
## Environment-Specific Configs
|
||||
|
||||
### Development
|
||||
|
||||
```env
|
||||
DATABASE_URL=sqlite:///./data/dev.db
|
||||
QDRANT_URL=http://localhost:6333
|
||||
LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql://user:pass@localhost/dbname
|
||||
QDRANT_URL=http://qdrant:6333
|
||||
QDRANT_API_KEY=your-api-key
|
||||
LOG_LEVEL=WARNING
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```env
|
||||
DATABASE_URL=sqlite:///./data/test.db
|
||||
QDRANT_URL=http://localhost:6333
|
||||
CHUNK_SIZE=100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
Settings are validated on startup. Invalid values will cause:
|
||||
|
||||
```
|
||||
pydantic.ValidationError: 1 validation error for Settings
|
||||
```
|
||||
|
||||
Common errors:
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Missing OPENAI_API_KEY | No API key | Add to .env |
|
||||
| Invalid QDRANT_URL | Wrong URL format | Check URL |
|
||||
| CHUNK_SIZE < 1 | Too small | Increase value |
|
||||
385
docs/data-flow.md
Normal file
385
docs/data-flow.md
Normal file
@@ -0,0 +1,385 @@
|
||||
# Data Flow
|
||||
|
||||
How data moves through the RAG Chunking Benchmarker system.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ DATA FLOW │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ INGESTION │
|
||||
│ ───────── │
|
||||
│ .docx/.doc → Parser → DocumentTree + Markdown → SQLite │
|
||||
│ │
|
||||
│ PROCESSING │
|
||||
│ ────────── │
|
||||
│ Document → Strategy.chunk() → Chunks → Embed → Qdrant │
|
||||
│ │
|
||||
│ QUERYING │
|
||||
│ ───────── │
|
||||
│ Question → Embed → Qdrant Search → Chunks → LLM → Answer │
|
||||
│ │
|
||||
│ BENCHMARKING │
|
||||
│ ─────────── │
|
||||
│ Questions → (Query × Strategies) → Evaluation → Report │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Document Ingestion
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
User uploads .docx/.doc
|
||||
↓
|
||||
FastAPI receives file
|
||||
↓
|
||||
Parser extracts text blocks
|
||||
↓
|
||||
Builds DocumentTree (hierarchy)
|
||||
↓
|
||||
Generates markdown (flat text)
|
||||
↓
|
||||
Stores in SQLite
|
||||
```
|
||||
|
||||
### Data Structures
|
||||
|
||||
**Input:** Binary file (.docx/.doc)
|
||||
|
||||
**Output:**
|
||||
```python
|
||||
ParseResult(
|
||||
tree=DocumentTree, # Hierarchical structure
|
||||
markdown=str, # Flat text for chunking
|
||||
plain_text=str, # Raw text
|
||||
paragraph_count=int # Number of text blocks
|
||||
)
|
||||
```
|
||||
|
||||
### Storage
|
||||
|
||||
**SQLite:**
|
||||
```sql
|
||||
INSERT INTO documents (id, filename, parsed_text, document_tree, created_at)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Document Processing
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
User selects strategies
|
||||
↓
|
||||
For each strategy:
|
||||
↓
|
||||
Load DocumentTree + markdown from SQLite
|
||||
↓
|
||||
Strategy.chunk() → list[Chunk]
|
||||
↓
|
||||
Embed chunks via OpenAI
|
||||
↓
|
||||
Store in Qdrant collection
|
||||
↓
|
||||
Update chunk_counts in SQLite
|
||||
```
|
||||
|
||||
### Data Structures
|
||||
|
||||
**Input:**
|
||||
```python
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategies": ["recursive", "fixed_size", "semantic"]
|
||||
}
|
||||
```
|
||||
|
||||
**Processing:**
|
||||
```python
|
||||
# For each strategy
|
||||
chunks = strategy.chunk(
|
||||
doc_name="insurance.docx",
|
||||
tree=document_tree,
|
||||
markdown=markdown_text
|
||||
)
|
||||
|
||||
embeddings = embed_texts([chunk.text for chunk in chunks])
|
||||
|
||||
qdr.upsert_chunks(chunks, embeddings)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```python
|
||||
{
|
||||
"strategies_completed": [
|
||||
{"strategy": "recursive", "chunks_produced": 218}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Storage
|
||||
|
||||
**Qdrant:**
|
||||
```python
|
||||
{
|
||||
"id": "uuid5(chunk_id)",
|
||||
"vector": [0.023, -0.156, ...], # 1536 dimensions
|
||||
"payload": {
|
||||
"document_name": "insurance.docx",
|
||||
"chunk_id": "recursive_doc_000045",
|
||||
"strategy_name": "recursive",
|
||||
"text": "chunk content...",
|
||||
"token_count": 146,
|
||||
"character_count": 205
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Query Pipeline
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
User asks question
|
||||
↓
|
||||
Embed question via OpenAI
|
||||
↓
|
||||
Search Qdrant for similar chunks
|
||||
↓
|
||||
Retrieve top-K chunks
|
||||
↓
|
||||
Build context from chunks
|
||||
↓
|
||||
Generate answer via LLM
|
||||
↓
|
||||
Store query result in SQLite
|
||||
```
|
||||
|
||||
### Data Structures
|
||||
|
||||
**Input:**
|
||||
```python
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategy": "recursive",
|
||||
"question": "What are the main topics?",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
|
||||
**Processing:**
|
||||
```python
|
||||
# 1. Embed question
|
||||
question_embedding = embed_single(question)
|
||||
|
||||
# 2. Vector search
|
||||
hits = qdr.search(
|
||||
strategy="recursive",
|
||||
query_vector=question_embedding,
|
||||
top_k=5
|
||||
)
|
||||
|
||||
# 3. Build context
|
||||
context = "\n".join([hit.text for hit in hits])
|
||||
|
||||
# 4. Generate answer
|
||||
answer = llm.generate(
|
||||
system="Answer based on context...",
|
||||
user=f"Context: {context}\nQuestion: {question}"
|
||||
)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```python
|
||||
{
|
||||
"answer": "The document covers insurance regulations...",
|
||||
"retrieved_chunks": [...],
|
||||
"latency_breakdown": {...},
|
||||
"token_usage": {...}
|
||||
}
|
||||
```
|
||||
|
||||
### Storage
|
||||
|
||||
**SQLite:**
|
||||
```sql
|
||||
INSERT INTO queries (id, document_id, strategy_name, question, answer,
|
||||
retrieved_chunks, latency_breakdown, token_usage)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Benchmark Pipeline
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
Load questions from JSON
|
||||
↓
|
||||
For each question × strategy:
|
||||
↓
|
||||
Run query pipeline
|
||||
↓
|
||||
Evaluate with LLM-as-Judge
|
||||
↓
|
||||
Store per-question results
|
||||
↓
|
||||
Aggregate metrics per strategy
|
||||
↓
|
||||
Store experiment in SQLite
|
||||
↓
|
||||
Generate HTML report
|
||||
```
|
||||
|
||||
### Data Structures
|
||||
|
||||
**Input:**
|
||||
```python
|
||||
{
|
||||
"document_id": "doc-abc123",
|
||||
"strategies": ["recursive", "fixed_size"],
|
||||
"questions_file": "files/questions.json"
|
||||
}
|
||||
```
|
||||
|
||||
**Processing:**
|
||||
```python
|
||||
# For each question
|
||||
for question in questions:
|
||||
for strategy in strategies:
|
||||
# Query
|
||||
result = run_query(question, strategy)
|
||||
|
||||
# Evaluate
|
||||
scores = evaluate_single(
|
||||
question=question,
|
||||
context=result.retrieved_chunks,
|
||||
expected=question.expected_answer,
|
||||
generated=result.answer
|
||||
)
|
||||
|
||||
# Store
|
||||
per_question_results.append({...})
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```python
|
||||
{
|
||||
"experiment_id": "exp-abc123",
|
||||
"aggregate_metrics": {
|
||||
"recursive": {
|
||||
"avg_context_relevance": 8.5,
|
||||
"avg_answer_similarity": 7.8,
|
||||
"avg_faithfulness": 9.2,
|
||||
"hallucination_rate": 0.05
|
||||
}
|
||||
},
|
||||
"best_strategy": "recursive"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Report Generation
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
Load experiment from SQLite
|
||||
↓
|
||||
Calculate rankings
|
||||
↓
|
||||
Generate HTML (managerial or technical view)
|
||||
↓
|
||||
Return HTML response
|
||||
```
|
||||
|
||||
### Views
|
||||
|
||||
| View | Focus | Content |
|
||||
|------|-------|---------|
|
||||
| Managerial | Decision-making | KPIs, winner, recommendations |
|
||||
| Technical | Observability | Detailed scores, token usage, per-question |
|
||||
|
||||
---
|
||||
|
||||
## Data Persistence
|
||||
|
||||
### SQLite Tables
|
||||
|
||||
| Table | Purpose | Key Fields |
|
||||
|-------|---------|------------|
|
||||
| documents | Uploaded documents | id, filename, parsed_text, document_tree |
|
||||
| queries | Query results | id, document_id, question, answer |
|
||||
| experiments | Benchmark results | id, document_id, aggregate_metrics |
|
||||
|
||||
### Qdrant Collections
|
||||
|
||||
| Collection | Purpose | Fields |
|
||||
|------------|---------|--------|
|
||||
| recursive_collection | Recursive strategy vectors | chunk_id, text, scores |
|
||||
| fixed_size_collection | Fixed-size strategy vectors | chunk_id, text, scores |
|
||||
| semantic_collection | Semantic strategy vectors | chunk_id, text, scores |
|
||||
| contextual_retrieval_collection | Contextual strategy vectors | chunk_id, text, scores |
|
||||
| semantic_parent_child_collection | Parent-child strategy vectors | chunk_id, text, parent_id |
|
||||
|
||||
---
|
||||
|
||||
## Data Transformation
|
||||
|
||||
### Document → Chunks
|
||||
|
||||
```
|
||||
Input: Full document text (10,000 tokens)
|
||||
Output: 50-300 chunks (100-500 tokens each)
|
||||
|
||||
Transformation:
|
||||
- Tokenization
|
||||
- Boundary detection
|
||||
- Metadata enrichment
|
||||
```
|
||||
|
||||
### Chunks → Vectors
|
||||
|
||||
```
|
||||
Input: Chunk text
|
||||
Output: 1536-dimensional vector
|
||||
|
||||
Transformation:
|
||||
- Text → Tokens → Embedding API → Vector
|
||||
```
|
||||
|
||||
### Question → Answer
|
||||
|
||||
```
|
||||
Input: Question string
|
||||
Output: Answer string + metadata
|
||||
|
||||
Transformation:
|
||||
- Question → Embedding → Vector Search → Chunks → LLM → Answer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
| Operation | Time | Tokens | Cost |
|
||||
|-----------|------|--------|------|
|
||||
| Document upload | ~2s | 0 | Free |
|
||||
| Process (1 strategy) | ~5s | N/A | ~$0.001 (embeddings) |
|
||||
| Process (5 strategies) | ~25s | N/A | ~$0.005 |
|
||||
| Query | ~2s | ~1500 | ~$0.001 |
|
||||
| Benchmark (21 questions) | ~120s | ~200,000 | ~$0.05 |
|
||||
222
docs/evaluation-metrics.md
Normal file
222
docs/evaluation-metrics.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Evaluation Metrics
|
||||
|
||||
How scoring works in the benchmarking system.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The system uses **LLM-as-Judge** evaluation to automatically score answer quality across 4 metrics.
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
### 1. Context Relevance (1-10)
|
||||
|
||||
**Question:** How relevant are the retrieved chunks to answering the question?
|
||||
|
||||
| Score | Meaning |
|
||||
|-------|---------|
|
||||
| 9-10 | Chunks directly answer the question |
|
||||
| 7-8 | Chunks are highly relevant |
|
||||
| 5-6 | Chunks are somewhat relevant |
|
||||
| 3-4 | Chunks are partially relevant |
|
||||
| 1-2 | Chunks are not relevant |
|
||||
|
||||
**What it measures:** Did the retrieval system find the right information?
|
||||
|
||||
---
|
||||
|
||||
### 2. Answer Similarity (1-10)
|
||||
|
||||
**Question:** How similar is the generated answer to the expected answer?
|
||||
|
||||
| Score | Meaning |
|
||||
|-------|---------|
|
||||
| 9-10 | Nearly identical to expected |
|
||||
| 7-8 | Covers most key points |
|
||||
| 5-6 | Covers some key points |
|
||||
| 3-4 | Partially correct |
|
||||
| 1-2 | Completely different |
|
||||
|
||||
**What it measures:** Did the system produce the right answer?
|
||||
|
||||
---
|
||||
|
||||
### 3. Faithfulness (1-10)
|
||||
|
||||
**Question:** Is the generated answer grounded in the retrieved context?
|
||||
|
||||
| Score | Meaning |
|
||||
|-------|---------|
|
||||
| 9-10 | Entirely based on context |
|
||||
| 7-8 | Mostly based on context |
|
||||
| 5-6 | Partially based on context |
|
||||
| 3-4 | Some external knowledge used |
|
||||
| 1-2 | Mostly external knowledge |
|
||||
|
||||
**What it measures:** Is the answer trustworthy?
|
||||
|
||||
---
|
||||
|
||||
### 4. Hallucination (boolean)
|
||||
|
||||
**Question:** Did the LLM invent information not in the context?
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| false | Answer is grounded in context |
|
||||
| true | Answer contains invented information |
|
||||
|
||||
**What it measures:** Is the answer fabricated?
|
||||
|
||||
---
|
||||
|
||||
## Scoring Process
|
||||
|
||||
### Step 1: Prepare Evaluation Context
|
||||
|
||||
```
|
||||
Question: {user_question}
|
||||
Context: {retrieved_chunks}
|
||||
Expected Answer: {golden_answer}
|
||||
Generated Answer: {system_answer}
|
||||
```
|
||||
|
||||
### Step 2: Send to LLM-as-Judge
|
||||
|
||||
```python
|
||||
prompt = f"""
|
||||
Evaluate this RAG system output:
|
||||
|
||||
Question: {question}
|
||||
Context: {context}
|
||||
Expected: {expected}
|
||||
Generated: {generated}
|
||||
|
||||
Return JSON:
|
||||
{{
|
||||
"context_relevance": <1-10>,
|
||||
"answer_similarity": <1-10>,
|
||||
"faithfulness": <1-10>,
|
||||
"hallucination": <true/false>,
|
||||
"reasoning": "<explanation>"
|
||||
}}
|
||||
"""
|
||||
```
|
||||
|
||||
### Step 3: Parse Response
|
||||
|
||||
```python
|
||||
scores = json.loads(llm_response)
|
||||
# Validate ranges
|
||||
for metric in ["context_relevance", "answer_similarity", "faithfulness"]:
|
||||
scores[metric] = max(1, min(10, scores[metric]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Overall Score Calculation
|
||||
|
||||
Each strategy gets an overall score weighted by importance:
|
||||
|
||||
```python
|
||||
overall = (
|
||||
context_relevance * 0.3 + # 30% weight
|
||||
answer_similarity * 0.4 + # 40% weight
|
||||
faithfulness * 0.3 # 30% weight
|
||||
) * (1 - hallucination_rate) # Hallucination penalty
|
||||
```
|
||||
|
||||
### Why These Weights?
|
||||
|
||||
| Metric | Weight | Rationale |
|
||||
|--------|--------|-----------|
|
||||
| Answer Similarity | 40% | Most important - did we get the right answer? |
|
||||
| Context Relevance | 30% | Found the right information |
|
||||
| Faithfulness | 30% | Answer is trustworthy |
|
||||
| Hallucination | Penalty | Fabricated info is unacceptable |
|
||||
|
||||
---
|
||||
|
||||
## Aggregate Metrics
|
||||
|
||||
Per strategy, we calculate:
|
||||
|
||||
| Metric | Calculation |
|
||||
|--------|-------------|
|
||||
| avg_context_relevance | mean of all context_relevance scores |
|
||||
| avg_answer_similarity | mean of all answer_similarity scores |
|
||||
| avg_faithfulness | mean of all faithfulness scores |
|
||||
| hallucination_rate | count(hallucination=True) / total_questions |
|
||||
| total_questions | number of questions evaluated |
|
||||
| failed_questions | questions that errored |
|
||||
|
||||
---
|
||||
|
||||
## Example Output
|
||||
|
||||
```json
|
||||
{
|
||||
"recursive": {
|
||||
"avg_context_relevance": 8.5,
|
||||
"avg_answer_similarity": 7.8,
|
||||
"avg_faithfulness": 9.2,
|
||||
"hallucination_rate": 0.05,
|
||||
"total_questions": 21,
|
||||
"failed_questions": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interpretation Guide
|
||||
|
||||
### Good Scores
|
||||
|
||||
| Metric | Target | Meaning |
|
||||
|--------|--------|---------|
|
||||
| Context Relevance | ≥ 8 | Retrieval is accurate |
|
||||
| Answer Similarity | ≥ 8 | Answers match expectations |
|
||||
| Faithfulness | ≥ 9 | Answers are trustworthy |
|
||||
| Hallucination Rate | ≤ 0.1 | Low fabrication rate |
|
||||
|
||||
### Warning Signs
|
||||
|
||||
| Metric | Warning | Meaning |
|
||||
|--------|---------|---------|
|
||||
| Context Relevance | < 6 | Retrieval needs improvement |
|
||||
| Answer Similarity | < 6 | Answers are off-target |
|
||||
| Faithfulness | < 7 | Model is adding external knowledge |
|
||||
| Hallucination Rate | > 0.2 | High fabrication rate |
|
||||
|
||||
---
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
Each evaluation uses:
|
||||
|
||||
| Component | Cost |
|
||||
|-----------|------|
|
||||
| LLM call (gpt-4o-mini) | ~$0.001 per evaluation |
|
||||
| Input tokens | ~500 per evaluation |
|
||||
| Output tokens | ~100 per evaluation |
|
||||
|
||||
**Total for 21 questions × 5 strategies:**
|
||||
- 105 evaluations × $0.001 = **$0.105**
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Evaluation parameters in `src/benchmarking/evaluation.py`:
|
||||
|
||||
```python
|
||||
_EVALUATION_SYSTEM_PROMPT = "You are an expert evaluator..."
|
||||
_EVALUATION_USER_PROMPT = "Evaluate this RAG system output..."
|
||||
model = settings.llm_model # gpt-4o-mini
|
||||
temperature = 0.0 # Deterministic
|
||||
max_tokens = 500 # Response limit
|
||||
```
|
||||
345
docs/strategy-technical-details.md
Normal file
345
docs/strategy-technical-details.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# Strategy Technical Details
|
||||
|
||||
Deep technical explanation of each chunking strategy's implementation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Fixed-Size Chunking
|
||||
|
||||
**File:** `src/chunking/strategies/fixed_size.py`
|
||||
|
||||
### How It Works
|
||||
|
||||
Splits text into fixed-size token chunks with overlap.
|
||||
|
||||
### Splitting Unit
|
||||
|
||||
**Token-level** using `tiktoken` (cl100k_base encoding).
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Encode full markdown → token array
|
||||
2. Take first chunk_size tokens → chunk 1
|
||||
3. Slide forward by (chunk_size - overlap) tokens
|
||||
4. Repeat until end of text
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Config Key | Default | Description |
|
||||
|-----------|------------|---------|-------------|
|
||||
| Chunk size | `chunk_size` | 512 | Tokens per chunk |
|
||||
| Overlap | `chunk_overlap` | 50 | Token overlap between chunks |
|
||||
|
||||
### Code Logic
|
||||
|
||||
```python
|
||||
tokens = _encoder.encode(text)
|
||||
start = 0
|
||||
while start < len(tokens):
|
||||
end = min(start + chunk_size, len(tokens))
|
||||
chunk_tokens = tokens[start:end]
|
||||
chunks.append(_encoder.decode(chunk_tokens))
|
||||
start = end - overlap
|
||||
```
|
||||
|
||||
### Characteristics
|
||||
|
||||
- **Splitting boundary:** Arbitrary (mid-sentence possible)
|
||||
- **Structure awareness:** None
|
||||
- **Speed:** Fastest (no API calls)
|
||||
- **Best for:** Baseline comparison, simple documents
|
||||
|
||||
---
|
||||
|
||||
## 2. Recursive Chunking
|
||||
|
||||
**File:** `src/chunking/strategies/recursive.py`
|
||||
|
||||
### How It Works
|
||||
|
||||
Cascade splitting using a separator hierarchy. Tries meaningful boundaries first, falls back to less meaningful ones.
|
||||
|
||||
### Splitting Unit
|
||||
|
||||
**Character-level** with regex patterns.
|
||||
|
||||
### Separator Cascade (Priority Order)
|
||||
|
||||
| Priority | Separator | Regex Pattern | Description |
|
||||
|----------|-----------|---------------|-------------|
|
||||
| 1 | Header | `^(#{1,6})\s+` | Markdown headers |
|
||||
| 2 | Double newline | `\n\n` | Paragraph breaks |
|
||||
| 3 | Single newline | `\n` | Line breaks |
|
||||
| 4 | Sentence | `[.!?]\s+` | Sentence endings |
|
||||
| 5 | Space | `\s+` | Word boundaries |
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Try splitting by highest-priority separator
|
||||
2. If parts still too big → recurse with next separator
|
||||
3. Merge small parts back up to target size
|
||||
4. If single part exceeds target → hard split by character count
|
||||
```
|
||||
|
||||
### Code Logic
|
||||
|
||||
```python
|
||||
for sep_name, pattern in _SEPARATORS:
|
||||
parts = pattern.split(text)
|
||||
if len(parts) <= 1:
|
||||
continue # this separator didn't split
|
||||
|
||||
# Merge parts back up to target_size
|
||||
current = ""
|
||||
for part in parts:
|
||||
candidate = (current + " " + part).strip()
|
||||
if len(candidate) <= target_size:
|
||||
current = candidate
|
||||
else:
|
||||
chunks.append(current)
|
||||
current = part
|
||||
|
||||
return chunks # Return on first successful split
|
||||
```
|
||||
|
||||
### Characteristics
|
||||
|
||||
- **Splitting boundary:** Semantic (headers, paragraphs, sentences)
|
||||
- **Structure awareness:** High (respects markdown structure)
|
||||
- **Speed:** Fast (no API calls)
|
||||
- **Best for:** Structured documents with clear hierarchy
|
||||
|
||||
---
|
||||
|
||||
## 3. Semantic Chunking
|
||||
|
||||
**File:** `src/chunking/strategies/semantic.py`
|
||||
|
||||
### How It Works
|
||||
|
||||
Groups sentences by semantic similarity. When similarity drops, starts a new chunk.
|
||||
|
||||
### Splitting Unit
|
||||
|
||||
**Sentence-level** with similarity-based boundaries.
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Split markdown into sentences
|
||||
2. Embed each sentence via OpenAI (text-embedding-3-small)
|
||||
3. Compute cosine similarity between adjacent sentences
|
||||
4. When similarity < SEMANTIC_THRESHOLD → chunk boundary
|
||||
5. Enforce SEMANTIC_MIN_CHUNK_SIZE (minimum sentences per chunk)
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Config Key | Default | Description |
|
||||
|-----------|------------|---------|-------------|
|
||||
| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for boundary |
|
||||
| Min size | `semantic_min_chunk_size` | 5 | Minimum sentences per chunk |
|
||||
|
||||
### Code Logic
|
||||
|
||||
```python
|
||||
sentences = split_sentences(markdown)
|
||||
embeddings = embed_texts(sentences) # OpenAI API call
|
||||
|
||||
current_group = [sentences[0]]
|
||||
for i in range(1, len(sentences)):
|
||||
sim = cosine_similarity(embeddings[i-1], embeddings[i])
|
||||
|
||||
if sim < threshold and len(current_group) >= min_size:
|
||||
# Topic shift → new chunk
|
||||
chunks.append(" ".join(current_group))
|
||||
current_group = [sentences[i]]
|
||||
else:
|
||||
current_group.append(sentences[i])
|
||||
```
|
||||
|
||||
### Characteristics
|
||||
|
||||
- **Splitting boundary:** Semantic (topic changes)
|
||||
- **Structure awareness:** Medium (sentence-level)
|
||||
- **Speed:** Medium (requires embeddings)
|
||||
- **Best for:** Documents with topic shifts, no clear structure
|
||||
|
||||
---
|
||||
|
||||
## 4. Contextual Retrieval
|
||||
|
||||
**File:** `src/chunking/strategies/contextual_retrieval.py`
|
||||
|
||||
### How It Works
|
||||
|
||||
Based on Anthropic's 2024 research. Prepends a short context summary to each chunk before embedding.
|
||||
|
||||
### Splitting Unit
|
||||
|
||||
**Token-level** (same as fixed-size).
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Split text using fixed-size token splitting
|
||||
2. For each chunk:
|
||||
a. Find surrounding text (500 chars before/after)
|
||||
b. Send to LLM with context prompt
|
||||
c. LLM generates 1-2 sentence context prefix
|
||||
d. Enriched chunk = context + original text
|
||||
3. Embed enriched chunks (not original)
|
||||
```
|
||||
|
||||
### LLM Prompt
|
||||
|
||||
```
|
||||
System: You are a document analysis assistant. Given a section of text
|
||||
from a document, write a short context prefix (1-2 sentences) that
|
||||
would help someone find this section later via search.
|
||||
|
||||
User:
|
||||
Preceding text: {preceding_500_chars}
|
||||
This section: {chunk_text}
|
||||
Following text: {following_500_chars}
|
||||
```
|
||||
|
||||
### Code Logic
|
||||
|
||||
```python
|
||||
for chunk_text in raw_chunks:
|
||||
# Find surrounding context
|
||||
preceding = full_text[pos-500:pos]
|
||||
following = full_text[pos+len:pos+len+500]
|
||||
|
||||
# Generate context prefix
|
||||
context = llm.generate(
|
||||
system=CONTEXT_PROMPT,
|
||||
user=f"Preceding: {preceding}\nSection: {chunk_text}\nFollowing: {following}"
|
||||
)
|
||||
|
||||
# Enrich chunk
|
||||
enriched = f"{context}\n\n{chunk_text}"
|
||||
chunks.append(enriched)
|
||||
```
|
||||
|
||||
### Characteristics
|
||||
|
||||
- **Splitting boundary:** Token-based (like fixed-size)
|
||||
- **Structure awareness:** None (relies on LLM for context)
|
||||
- **Speed:** Slowest (1 LLM call per chunk)
|
||||
- **Best for:** Improving retrieval quality, unstructured documents
|
||||
|
||||
---
|
||||
|
||||
## 5. Semantic Parent-Child
|
||||
|
||||
**File:** `src/chunking/strategies/semantic_parent_child.py`
|
||||
|
||||
### How It Works
|
||||
|
||||
Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
|
||||
|
||||
### Splitting Unit
|
||||
|
||||
**Paragraph-level** with semantic clustering.
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Split markdown into paragraphs
|
||||
2. Embed each paragraph
|
||||
3. Cluster consecutive paragraphs by similarity
|
||||
4. Each cluster = parent chunk (full cluster text)
|
||||
5. Each paragraph = child chunk (linked to parent)
|
||||
```
|
||||
|
||||
### Query-Time Behavior
|
||||
|
||||
```
|
||||
1. Vector search finds matching child paragraph
|
||||
2. Use parent_id to fetch parent cluster
|
||||
3. Return both child + parent to LLM
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Config Key | Default | Description |
|
||||
|-----------|------------|---------|-------------|
|
||||
| Threshold | `semantic_threshold` | 0.5 | Similarity threshold for clustering |
|
||||
|
||||
### Code Logic
|
||||
|
||||
```python
|
||||
paragraphs = split_paragraphs(markdown)
|
||||
embeddings = embed_texts(paragraphs)
|
||||
|
||||
# Cluster paragraphs
|
||||
clusters = [[0]]
|
||||
for i in range(1, len(paragraphs)):
|
||||
sim = cosine_similarity(embeddings[i-1], embeddings[i])
|
||||
if sim >= threshold:
|
||||
clusters[-1].append(i) # Same cluster
|
||||
else:
|
||||
clusters.append([i]) # New cluster
|
||||
|
||||
# Create parent-child chunks
|
||||
for cluster in clusters:
|
||||
parent_text = "\n\n".join(paragraphs[i] for i in cluster)
|
||||
parent_id = make_chunk_id(...)
|
||||
|
||||
for para_idx in cluster:
|
||||
chunks.append(build_chunk(
|
||||
text=paragraphs[para_idx],
|
||||
parent_id=parent_id
|
||||
))
|
||||
```
|
||||
|
||||
### Characteristics
|
||||
|
||||
- **Splitting boundary:** Semantic (paragraph clusters)
|
||||
- **Structure awareness:** Medium (paragraph-level)
|
||||
- **Speed:** Medium (requires embeddings)
|
||||
- **Best for:** Documents needing context, flat structure
|
||||
|
||||
---
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Strategy | Split Unit | Boundary Logic | API Calls | Speed | Best For |
|
||||
|----------|------------|----------------|-----------|-------|----------|
|
||||
| fixed_size | Token | Arbitrary | 0 | Fast | Baseline |
|
||||
| recursive | Character | Structural | 0 | Fast | Structured docs |
|
||||
| semantic | Sentence | Similarity | 1 (embeddings) | Medium | Topic shifts |
|
||||
| contextual_retrieval | Token | LLM-generated | N (1 per chunk) | Slow | Retrieval quality |
|
||||
| semantic_parent_child | Paragraph | Similarity | 1 (embeddings) | Medium | Context needed |
|
||||
|
||||
---
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
All strategy parameters are in `src/core/config.py`:
|
||||
|
||||
```python
|
||||
class Settings(BaseSettings):
|
||||
chunk_size: int = 512
|
||||
chunk_overlap: int = 50
|
||||
semantic_threshold: float = 0.5
|
||||
semantic_min_chunk_size: int = 5
|
||||
embedding_model: str = "text-embedding-3-small"
|
||||
llm_model: str = "gpt-4o-mini"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ADR References
|
||||
|
||||
| ADR | Strategy | Decision |
|
||||
|-----|----------|----------|
|
||||
| ADR 0012 | semantic | Sentence-level with min chunk size |
|
||||
| ADR 0015 | recursive | Direct API, no LangChain/LlamaIndex |
|
||||
| ADR 0011 | contextual_retrieval | Embed enriched text, not original |
|
||||
| ADR 0003 | all | Per-strategy failure isolation |
|
||||
Reference in New Issue
Block a user