mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10 Commits 1 Branch 0 Tags
29edbf57c4892528ff5638ad391016fb7a3080a8
Commit Graph

10 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mahdi Bazrafshan
29edbf57c4 feat(api): wire documents router and SQLite init in app factory 
Why:
- App factory needs to mount document routes and initialize SQLite at startup

Changes:
- Import and mount documents_router
- Call init_db() on app creation
 2026-07-26 09:38:02 +03:30
Mahdi Bazrafshan
8715d9c186 feat(strategies): implement all 5 chunking strategies 
Why:
- Core evaluation targets — each strategy chunks differently for comparison

Changes:
- fixed_size: token-based splitting with overlap (baseline)
- recursive: cascade splitting (headers → newlines → sentences → words)
- semantic: sentence-level embeddings with similarity-based boundaries
- contextual_retrieval: LLM-generated context prefixes per chunk (Anthropic research)
- semantic_parent_child: paragraph clustering into parent-child hierarchy
 2026-07-26 09:37:56 +03:30
Mahdi Bazrafshan
143351b96b feat(chunking): add base strategy interface, embedding service, and orchestration 
Why:
- Need abstract base class for all chunking strategies
- Need OpenAI embedding service (text-embedding-3-small)
- Need orchestration to run chunk → embed → store pipeline

Changes:
- Base: ChunkingStrategy ABC, token counting, chunk ID generation, sentence splitting
- Embedding: batch embedding with 2048 text limit per call
- Service: strategy registry, single/multi-strategy runners with per-strategy failure isolation
 2026-07-26 09:37:46 +03:30
Mahdi Bazrafshan
fdc21e3316 feat(documents): add document parser, service, and API routes 
Why:
- Need to parse .docx/.doc files into DocumentTree + markdown
- Need API endpoints for document upload, processing, and deletion

Changes:
- Parser: .doc→.docx conversion via LibreOffice, XML-level extraction for table-heavy docs, heuristic heading detection
- Service: upload, process (delegates to chunking), delete orchestration
- Routes: POST /documents, POST /documents/{id}/process, DELETE /documents/{id}, GET /strategies
 2026-07-26 09:37:40 +03:30
Mahdi Bazrafshan
4aaaccec49 feat(storage): add SQLite and Qdrant storage layers 
Why:
- SQLite stores structured data (documents, queries, experiments)
- Qdrant stores vector embeddings for similarity search

Changes:
- SQLite: schema init, document/query/experiment CRUD, JSON field parsing
- Qdrant: collection management, vector upsert, similarity search, delete operations
 2026-07-26 09:37:32 +03:30
Mahdi Bazrafshan
4f205a7ef7 feat(core): add StrategyName enum, DocumentTree models, and pagination helper 
Why:
- StrategyName enum needed proper values for all 5 strategies
- DocumentTree models needed for document parser output
- PaginatedResponse needed for list endpoints

Changes:
- Renamed StrategyName values to match implementation (fixed_size, contextual_retrieval, semantic_parent_child)
- Added NodeType, DocumentTreeNode, DocumentTree models
- Added PaginatedResponse helper model
 2026-07-26 09:37:24 +03:30
Mahdi Bazrafshan
4edc355ae5 feat(core): add core layer with config, clients, exceptions, models, and app factory 2026-07-21 18:51:57 +03:30
Mahdi Bazrafshan
c5bf8d5e1e build: add project configuration and environment template 2026-07-21 18:51:51 +03:30
Mahdi Bazrafshan
09f9bb53bf docs: add domain glossary and architectural decision records 
15 ADRs covering chunk model, failure handling, evaluation, parsing, directory structure, REST design, and strategy-specific decisions. 62 domain terms across 7 sections.
 2026-07-21 18:51:46 +03:30
Mahdi Bazrafshan
247ea0a16e Initial commit 2026-07-21 11:27:49 +03:30