63 lines
2.7 KiB
Markdown
63 lines
2.7 KiB
Markdown
# ADR 0002: Chunking Strategy
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
The current system chunks CSV files row-by-row, treating each Q&A pair as a single chunk. The migration expands support to PDF, Word, Excel, and text files with predominantly unstructured content (FAQs, guidance, mixed text).
|
|
|
|
Chunk quality directly impacts retrieval precision. Arbitrary splits (mid-sentence, mid-thought) produce incoherent chunks that degrade both embedding quality and answer generation.
|
|
|
|
## Decision
|
|
|
|
Adopt a **hybrid chunking strategy** with markdown as the intermediate format:
|
|
|
|
### 1. Markdown-First Pipeline
|
|
All documents are converted to markdown before chunking. This provides:
|
|
- Consistent processing regardless of source format
|
|
- Preserved structure (headers, lists, tables)
|
|
- Easier debugging and inspection
|
|
|
|
### 2. Type-Specific Chunking
|
|
|
|
| Source Type | Chunking Method | Rationale |
|
|
|-------------|-----------------|-----------|
|
|
| CSV | Row-by-row | Already natural Q&A pairs |
|
|
| Excel | Row-by-row | Preserves tabular context per record |
|
|
| PDF | Convert to markdown → semantic chunking | Unstructured prose needs coherent boundaries |
|
|
| Word (.docx) | Convert to markdown → semantic chunking | Same as PDF |
|
|
| Text files | Convert to markdown → semantic chunking | Same as PDF |
|
|
| Structured documents (policies with headers) | Markdown section-aware chunking | Headers define natural boundaries |
|
|
|
|
### 3. Semantic Chunking Parameters
|
|
- **Unit:** Sentences (not tokens)
|
|
- **Grouping:** Consecutive sentences grouped until semantic similarity drops below threshold
|
|
- **Overlap:** 1-2 sentences at boundaries to preserve context
|
|
- **Max chunk size:** Determined by embedding model limits (typically 512-8192 tokens depending on model)
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
- **Coherent chunks** — complete thoughts, no mid-sentence splits
|
|
- **Consistent pipeline** — one chunking logic for all unstructured content
|
|
- **Debuggable** — markdown is human-readable, easy to inspect chunk quality
|
|
- **Flexible** — can adjust semantic threshold per document type if needed
|
|
|
|
### Negative
|
|
- **Processing overhead** — conversion to markdown adds a step
|
|
- **Variable chunk sizes** — harder to predict storage and latency
|
|
- **Semantic chunking requires embedding model** — adds computation during ingestion
|
|
|
|
### Neutral
|
|
- Need to select semantic similarity threshold (tuning required)
|
|
- Need to handle edge cases: very long sentences, tables in markdown, code blocks
|
|
|
|
## Implementation Notes
|
|
|
|
- Use `unstructured` or `pandoc` for document → markdown conversion
|
|
- Use LlamaIndex's `SemanticSplitter` or custom implementation with embedding model
|
|
- Consider `MarkdownElementNodeParser` for section-aware chunking when headers are present
|
|
- Store original source file path and chunk index in metadata for citation
|