docs: add phases, tasks, and chunking strategies documentation

Why:
- Need implementation plan and task tracking documentation
- Need strategy explanations for reference

Changes:
- phases.md: 5-phase implementation plan with status tracking
- tasks.md: 25 tasks mapped to phases and steps
- chunking_strategies.md: detailed explanations of all 5 strategies
This commit is contained in:
2026-07-26 09:38:08 +03:30
parent 29edbf57c4
commit 292ae17cfa
3 changed files with 315 additions and 0 deletions

144
docs/chunking_strategies.md Normal file
View File

@@ -0,0 +1,144 @@
# Chunking Strategies
This document explains the 5 chunking strategies implemented in the RAG Chunking Strategy Benchmarking Framework.
---
## 1. FIXED_SIZE (Baseline)
**How it works:** Splits text into chunks of N tokens with M token overlap.
### Algorithm
1. Encode full markdown into tokens
2. Take first `chunk_size` tokens as chunk 1
3. Slide forward by `chunk_size - overlap` tokens
4. Repeat until end of text
### Pros & Cons
- **Pros:** Simple, predictable, fast, no dependencies
- **Cons:** Ignores meaning — can split mid-sentence, mid-word, or across topics
### Configuration
Uses `chunk_size` and `chunk_overlap` from settings.
---
## 2. RECURSIVE (Cascade Splitting)
**How it works:** Tries to split on meaningful boundaries first, falling back to less meaningful ones.
### Separator Cascade (in order)
1. Markdown headers (`#`, `##`, `###`)
2. Double newline (`\n\n`) — paragraph breaks
3. Single newline (`\n`) — line breaks
4. Sentence endings (`. ! ?` + space)
5. Space (word-level, last resort)
### Algorithm
1. Try splitting by highest-priority separator
2. If parts are still too big, recurse with next separator
3. Merge small parts back up to target size
### Pros & Cons
- **Pros:** Respects document structure, produces natural chunks
- **Cons:** Still rule-based, no semantic understanding
---
## 3. SEMANTIC (Similarity-Based)
**How it works:** Groups sentences by meaning — when similarity drops, it starts a new chunk.
### Algorithm
1. Split markdown into sentences
2. Embed each sentence via OpenAI (`text-embedding-3-small`)
3. Compute cosine similarity between adjacent sentences
4. When similarity < `semantic_threshold`, create chunk boundary
5. Enforce minimum chunk size (`semantic_min_chunk_size` sentences)
### Pros & Cons
- **Pros:** Respects topic changes, produces coherent chunks
- **Cons:** Requires embeddings at chunk-time (API calls), slower, costs money
### Configuration
- `semantic_threshold` (default 0.5)
- `semantic_min_chunk_size` (default 5)
---
## 4. CONTEXTUAL_RETRIEVAL (LLM-Enriched)
**How it works:** Based on Anthropic's research — prepends a short context summary to each chunk before embedding.
### Algorithm
1. Split text using fixed-size token splitting (same as #1)
2. For each chunk, send surrounding text + chunk to LLM
3. LLM generates a 1-2 sentence context prefix
4. Enriched chunk = context prefix + original text
### Example Output
```
This section discusses insurance claim deadlines for property damage...
[Original chunk text about specific deadlines...]
```
### Pros & Cons
- **Pros:** Improved retrieval by 49% in Anthropic's benchmarks
- **Cons:** Expensive (1 LLM call per chunk), slowest strategy
### Configuration
Uses `llm_model` (gpt-4o-mini) for context generation.
---
## 5. SEMANTIC_PARENT_CHILD (Hierarchical)
**How it works:** Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
### Algorithm
1. Split markdown into paragraphs
2. Embed each paragraph
3. Cluster consecutive paragraphs by similarity (threshold-based)
4. Each cluster = parent chunk (full cluster text)
5. Each paragraph = child chunk (linked to parent)
### Query-time Behavior
- Search finds child paragraph via vector match
- Return full parent cluster as context to LLM
### Pros & Cons
- **Pros:** Rich context, no headings needed, works on flat documents
- **Cons:** More storage (both parent + child vectors), complex retrieval
### Configuration
Uses `semantic_threshold` for clustering.
---
## Summary Table
| Strategy | Split Method | Requires LLM | Requires Embeddings | Speed | Cost |
|----------|--------------|--------------|---------------------|-------|------|
| fixed_size | Token count | No | No | Fast | Free |
| recursive | Separator cascade | No | No | Fast | Free |
| semantic | Similarity threshold | No | Yes (at chunk time) | Medium | Low |
| contextual_retrieval | Fixed-size + LLM context | Yes (per chunk) | No | Slow | High |
| semantic_parent_child | Similarity clustering | No | Yes (at chunk time) | Medium | Low |