Research/chunking/docs/adr/0002-chunking-strategy.md

ADR 0002: Chunking Strategy

Status

Accepted

Context

The current system chunks CSV files row-by-row, treating each Q&A pair as a single chunk. The migration expands support to PDF, Word, Excel, and text files with predominantly unstructured content (FAQs, guidance, mixed text).

Chunk quality directly impacts retrieval precision. Arbitrary splits (mid-sentence, mid-thought) produce incoherent chunks that degrade both embedding quality and answer generation.

Decision

Adopt a hybrid chunking strategy with markdown as the intermediate format:

1. Markdown-First Pipeline

All documents are converted to markdown before chunking. This provides:

• Consistent processing regardless of source format
• Preserved structure (headers, lists, tables)
• Easier debugging and inspection

2. Type-Specific Chunking

Source Type	Chunking Method	Rationale
CSV	Row-by-row	Already natural Q&A pairs
Excel	Row-by-row	Preserves tabular context per record
PDF	Convert to markdown → semantic chunking	Unstructured prose needs coherent boundaries
Word (.docx)	Convert to markdown → semantic chunking	Same as PDF
Text files	Convert to markdown → semantic chunking	Same as PDF
Structured documents (policies with headers)	Markdown section-aware chunking	Headers define natural boundaries

3. Semantic Chunking Parameters

• Unit: Sentences (not tokens)
• Grouping: Consecutive sentences grouped until semantic similarity drops below threshold
• Overlap: 1-2 sentences at boundaries to preserve context
• Max chunk size: Determined by embedding model limits (typically 512-8192 tokens depending on model)

Consequences

Positive

• Coherent chunks — complete thoughts, no mid-sentence splits
• Consistent pipeline — one chunking logic for all unstructured content
• Debuggable — markdown is human-readable, easy to inspect chunk quality
• Flexible — can adjust semantic threshold per document type if needed

Negative

• Processing overhead — conversion to markdown adds a step
• Variable chunk sizes — harder to predict storage and latency
• Semantic chunking requires embedding model — adds computation during ingestion

Neutral

• Need to select semantic similarity threshold (tuning required)
• Need to handle edge cases: very long sentences, tables in markdown, code blocks

Implementation Notes

• Use unstructured or pandoc for document → markdown conversion
• Use LlamaIndex's SemanticSplitter or custom implementation with embedding model
• Consider MarkdownElementNodeParser for section-aware chunking when headers are present
• Store original source file path and chunk index in metadata for citation