Mahdi Bazrafshan
292ae17cfa
Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies
145 lines
4.0 KiB
Markdown
145 lines
4.0 KiB
Markdown
# Chunking Strategies
|
|
|
|
This document explains the 5 chunking strategies implemented in the RAG Chunking Strategy Benchmarking Framework.
|
|
|
|
---
|
|
|
|
## 1. FIXED_SIZE (Baseline)
|
|
|
|
**How it works:** Splits text into chunks of N tokens with M token overlap.
|
|
|
|
### Algorithm
|
|
|
|
1. Encode full markdown into tokens
|
|
2. Take first `chunk_size` tokens as chunk 1
|
|
3. Slide forward by `chunk_size - overlap` tokens
|
|
4. Repeat until end of text
|
|
|
|
### Pros & Cons
|
|
|
|
- **Pros:** Simple, predictable, fast, no dependencies
|
|
- **Cons:** Ignores meaning — can split mid-sentence, mid-word, or across topics
|
|
|
|
### Configuration
|
|
|
|
Uses `chunk_size` and `chunk_overlap` from settings.
|
|
|
|
---
|
|
|
|
## 2. RECURSIVE (Cascade Splitting)
|
|
|
|
**How it works:** Tries to split on meaningful boundaries first, falling back to less meaningful ones.
|
|
|
|
### Separator Cascade (in order)
|
|
|
|
1. Markdown headers (`#`, `##`, `###`)
|
|
2. Double newline (`\n\n`) — paragraph breaks
|
|
3. Single newline (`\n`) — line breaks
|
|
4. Sentence endings (`. ! ?` + space)
|
|
5. Space (word-level, last resort)
|
|
|
|
### Algorithm
|
|
|
|
1. Try splitting by highest-priority separator
|
|
2. If parts are still too big, recurse with next separator
|
|
3. Merge small parts back up to target size
|
|
|
|
### Pros & Cons
|
|
|
|
- **Pros:** Respects document structure, produces natural chunks
|
|
- **Cons:** Still rule-based, no semantic understanding
|
|
|
|
---
|
|
|
|
## 3. SEMANTIC (Similarity-Based)
|
|
|
|
**How it works:** Groups sentences by meaning — when similarity drops, it starts a new chunk.
|
|
|
|
### Algorithm
|
|
|
|
1. Split markdown into sentences
|
|
2. Embed each sentence via OpenAI (`text-embedding-3-small`)
|
|
3. Compute cosine similarity between adjacent sentences
|
|
4. When similarity < `semantic_threshold`, create chunk boundary
|
|
5. Enforce minimum chunk size (`semantic_min_chunk_size` sentences)
|
|
|
|
### Pros & Cons
|
|
|
|
- **Pros:** Respects topic changes, produces coherent chunks
|
|
- **Cons:** Requires embeddings at chunk-time (API calls), slower, costs money
|
|
|
|
### Configuration
|
|
|
|
- `semantic_threshold` (default 0.5)
|
|
- `semantic_min_chunk_size` (default 5)
|
|
|
|
---
|
|
|
|
## 4. CONTEXTUAL_RETRIEVAL (LLM-Enriched)
|
|
|
|
**How it works:** Based on Anthropic's research — prepends a short context summary to each chunk before embedding.
|
|
|
|
### Algorithm
|
|
|
|
1. Split text using fixed-size token splitting (same as #1)
|
|
2. For each chunk, send surrounding text + chunk to LLM
|
|
3. LLM generates a 1-2 sentence context prefix
|
|
4. Enriched chunk = context prefix + original text
|
|
|
|
### Example Output
|
|
|
|
```
|
|
This section discusses insurance claim deadlines for property damage...
|
|
|
|
[Original chunk text about specific deadlines...]
|
|
```
|
|
|
|
### Pros & Cons
|
|
|
|
- **Pros:** Improved retrieval by 49% in Anthropic's benchmarks
|
|
- **Cons:** Expensive (1 LLM call per chunk), slowest strategy
|
|
|
|
### Configuration
|
|
|
|
Uses `llm_model` (gpt-4o-mini) for context generation.
|
|
|
|
---
|
|
|
|
## 5. SEMANTIC_PARENT_CHILD (Hierarchical)
|
|
|
|
**How it works:** Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
|
|
|
|
### Algorithm
|
|
|
|
1. Split markdown into paragraphs
|
|
2. Embed each paragraph
|
|
3. Cluster consecutive paragraphs by similarity (threshold-based)
|
|
4. Each cluster = parent chunk (full cluster text)
|
|
5. Each paragraph = child chunk (linked to parent)
|
|
|
|
### Query-time Behavior
|
|
|
|
- Search finds child paragraph via vector match
|
|
- Return full parent cluster as context to LLM
|
|
|
|
### Pros & Cons
|
|
|
|
- **Pros:** Rich context, no headings needed, works on flat documents
|
|
- **Cons:** More storage (both parent + child vectors), complex retrieval
|
|
|
|
### Configuration
|
|
|
|
Uses `semantic_threshold` for clustering.
|
|
|
|
---
|
|
|
|
## Summary Table
|
|
|
|
| Strategy | Split Method | Requires LLM | Requires Embeddings | Speed | Cost |
|
|
|----------|--------------|--------------|---------------------|-------|------|
|
|
| fixed_size | Token count | No | No | Fast | Free |
|
|
| recursive | Separator cascade | No | No | Fast | Free |
|
|
| semantic | Similarity threshold | No | Yes (at chunk time) | Medium | Low |
|
|
| contextual_retrieval | Fixed-size + LLM context | Yes (per chunk) | No | Slow | High |
|
|
| semantic_parent_child | Similarity clustering | No | Yes (at chunk time) | Medium | Low |
|