docs: add phases, tasks, and chunking strategies documentation
Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies
This commit is contained in:
144
docs/chunking_strategies.md
Normal file
144
docs/chunking_strategies.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Chunking Strategies
|
||||
|
||||
This document explains the 5 chunking strategies implemented in the RAG Chunking Strategy Benchmarking Framework.
|
||||
|
||||
---
|
||||
|
||||
## 1. FIXED_SIZE (Baseline)
|
||||
|
||||
**How it works:** Splits text into chunks of N tokens with M token overlap.
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Encode full markdown into tokens
|
||||
2. Take first `chunk_size` tokens as chunk 1
|
||||
3. Slide forward by `chunk_size - overlap` tokens
|
||||
4. Repeat until end of text
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
- **Pros:** Simple, predictable, fast, no dependencies
|
||||
- **Cons:** Ignores meaning — can split mid-sentence, mid-word, or across topics
|
||||
|
||||
### Configuration
|
||||
|
||||
Uses `chunk_size` and `chunk_overlap` from settings.
|
||||
|
||||
---
|
||||
|
||||
## 2. RECURSIVE (Cascade Splitting)
|
||||
|
||||
**How it works:** Tries to split on meaningful boundaries first, falling back to less meaningful ones.
|
||||
|
||||
### Separator Cascade (in order)
|
||||
|
||||
1. Markdown headers (`#`, `##`, `###`)
|
||||
2. Double newline (`\n\n`) — paragraph breaks
|
||||
3. Single newline (`\n`) — line breaks
|
||||
4. Sentence endings (`. ! ?` + space)
|
||||
5. Space (word-level, last resort)
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Try splitting by highest-priority separator
|
||||
2. If parts are still too big, recurse with next separator
|
||||
3. Merge small parts back up to target size
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
- **Pros:** Respects document structure, produces natural chunks
|
||||
- **Cons:** Still rule-based, no semantic understanding
|
||||
|
||||
---
|
||||
|
||||
## 3. SEMANTIC (Similarity-Based)
|
||||
|
||||
**How it works:** Groups sentences by meaning — when similarity drops, it starts a new chunk.
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Split markdown into sentences
|
||||
2. Embed each sentence via OpenAI (`text-embedding-3-small`)
|
||||
3. Compute cosine similarity between adjacent sentences
|
||||
4. When similarity < `semantic_threshold`, create chunk boundary
|
||||
5. Enforce minimum chunk size (`semantic_min_chunk_size` sentences)
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
- **Pros:** Respects topic changes, produces coherent chunks
|
||||
- **Cons:** Requires embeddings at chunk-time (API calls), slower, costs money
|
||||
|
||||
### Configuration
|
||||
|
||||
- `semantic_threshold` (default 0.5)
|
||||
- `semantic_min_chunk_size` (default 5)
|
||||
|
||||
---
|
||||
|
||||
## 4. CONTEXTUAL_RETRIEVAL (LLM-Enriched)
|
||||
|
||||
**How it works:** Based on Anthropic's research — prepends a short context summary to each chunk before embedding.
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Split text using fixed-size token splitting (same as #1)
|
||||
2. For each chunk, send surrounding text + chunk to LLM
|
||||
3. LLM generates a 1-2 sentence context prefix
|
||||
4. Enriched chunk = context prefix + original text
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
This section discusses insurance claim deadlines for property damage...
|
||||
|
||||
[Original chunk text about specific deadlines...]
|
||||
```
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
- **Pros:** Improved retrieval by 49% in Anthropic's benchmarks
|
||||
- **Cons:** Expensive (1 LLM call per chunk), slowest strategy
|
||||
|
||||
### Configuration
|
||||
|
||||
Uses `llm_model` (gpt-4o-mini) for context generation.
|
||||
|
||||
---
|
||||
|
||||
## 5. SEMANTIC_PARENT_CHILD (Hierarchical)
|
||||
|
||||
**How it works:** Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Split markdown into paragraphs
|
||||
2. Embed each paragraph
|
||||
3. Cluster consecutive paragraphs by similarity (threshold-based)
|
||||
4. Each cluster = parent chunk (full cluster text)
|
||||
5. Each paragraph = child chunk (linked to parent)
|
||||
|
||||
### Query-time Behavior
|
||||
|
||||
- Search finds child paragraph via vector match
|
||||
- Return full parent cluster as context to LLM
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
- **Pros:** Rich context, no headings needed, works on flat documents
|
||||
- **Cons:** More storage (both parent + child vectors), complex retrieval
|
||||
|
||||
### Configuration
|
||||
|
||||
Uses `semantic_threshold` for clustering.
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Strategy | Split Method | Requires LLM | Requires Embeddings | Speed | Cost |
|
||||
|----------|--------------|--------------|---------------------|-------|------|
|
||||
| fixed_size | Token count | No | No | Fast | Free |
|
||||
| recursive | Separator cascade | No | No | Fast | Free |
|
||||
| semantic | Similarity threshold | No | Yes (at chunk time) | Medium | Low |
|
||||
| contextual_retrieval | Fixed-size + LLM context | Yes (per chunk) | No | Slow | High |
|
||||
| semantic_parent_child | Similarity clustering | No | Yes (at chunk time) | Medium | Low |
|
||||
Reference in New Issue
Block a user