Mahdi Bazrafshan
8715d9c186
Why: - Core evaluation targets — each strategy chunks differently for comparison Changes: - fixed_size: token-based splitting with overlap (baseline) - recursive: cascade splitting (headers → newlines → sentences → words) - semantic: sentence-level embeddings with similarity-based boundaries - contextual_retrieval: LLM-generated context prefixes per chunk (Anthropic research) - semantic_parent_child: paragraph clustering into parent-child hierarchy
chunking_strategies_evaluation
This repo is in order to reach multiple chunking strategies and evaulate them