From 292ae17cfab5ef23c642fefe77d29bdf37e6f968 Mon Sep 17 00:00:00 2001 From: Mahdi Bazrafshan Date: Sun, 26 Jul 2026 09:38:08 +0330 Subject: [PATCH] docs: add phases, tasks, and chunking strategies documentation Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies --- docs/chunking_strategies.md | 144 ++++++++++++++++++++++++++++++++++++ docs/phases.md | 115 ++++++++++++++++++++++++++++ docs/tasks.md | 56 ++++++++++++++ 3 files changed, 315 insertions(+) create mode 100644 docs/chunking_strategies.md create mode 100644 docs/phases.md create mode 100644 docs/tasks.md diff --git a/docs/chunking_strategies.md b/docs/chunking_strategies.md new file mode 100644 index 0000000..bd8e093 --- /dev/null +++ b/docs/chunking_strategies.md @@ -0,0 +1,144 @@ +# Chunking Strategies + +This document explains the 5 chunking strategies implemented in the RAG Chunking Strategy Benchmarking Framework. + +--- + +## 1. FIXED_SIZE (Baseline) + +**How it works:** Splits text into chunks of N tokens with M token overlap. + +### Algorithm + +1. Encode full markdown into tokens +2. Take first `chunk_size` tokens as chunk 1 +3. Slide forward by `chunk_size - overlap` tokens +4. Repeat until end of text + +### Pros & Cons + +- **Pros:** Simple, predictable, fast, no dependencies +- **Cons:** Ignores meaning — can split mid-sentence, mid-word, or across topics + +### Configuration + +Uses `chunk_size` and `chunk_overlap` from settings. + +--- + +## 2. RECURSIVE (Cascade Splitting) + +**How it works:** Tries to split on meaningful boundaries first, falling back to less meaningful ones. + +### Separator Cascade (in order) + +1. Markdown headers (`#`, `##`, `###`) +2. Double newline (`\n\n`) — paragraph breaks +3. Single newline (`\n`) — line breaks +4. Sentence endings (`. ! ?` + space) +5. Space (word-level, last resort) + +### Algorithm + +1. Try splitting by highest-priority separator +2. If parts are still too big, recurse with next separator +3. Merge small parts back up to target size + +### Pros & Cons + +- **Pros:** Respects document structure, produces natural chunks +- **Cons:** Still rule-based, no semantic understanding + +--- + +## 3. SEMANTIC (Similarity-Based) + +**How it works:** Groups sentences by meaning — when similarity drops, it starts a new chunk. + +### Algorithm + +1. Split markdown into sentences +2. Embed each sentence via OpenAI (`text-embedding-3-small`) +3. Compute cosine similarity between adjacent sentences +4. When similarity < `semantic_threshold`, create chunk boundary +5. Enforce minimum chunk size (`semantic_min_chunk_size` sentences) + +### Pros & Cons + +- **Pros:** Respects topic changes, produces coherent chunks +- **Cons:** Requires embeddings at chunk-time (API calls), slower, costs money + +### Configuration + +- `semantic_threshold` (default 0.5) +- `semantic_min_chunk_size` (default 5) + +--- + +## 4. CONTEXTUAL_RETRIEVAL (LLM-Enriched) + +**How it works:** Based on Anthropic's research — prepends a short context summary to each chunk before embedding. + +### Algorithm + +1. Split text using fixed-size token splitting (same as #1) +2. For each chunk, send surrounding text + chunk to LLM +3. LLM generates a 1-2 sentence context prefix +4. Enriched chunk = context prefix + original text + +### Example Output + +``` +This section discusses insurance claim deadlines for property damage... + +[Original chunk text about specific deadlines...] +``` + +### Pros & Cons + +- **Pros:** Improved retrieval by 49% in Anthropic's benchmarks +- **Cons:** Expensive (1 LLM call per chunk), slowest strategy + +### Configuration + +Uses `llm_model` (gpt-4o-mini) for context generation. + +--- + +## 5. SEMANTIC_PARENT_CHILD (Hierarchical) + +**How it works:** Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child. + +### Algorithm + +1. Split markdown into paragraphs +2. Embed each paragraph +3. Cluster consecutive paragraphs by similarity (threshold-based) +4. Each cluster = parent chunk (full cluster text) +5. Each paragraph = child chunk (linked to parent) + +### Query-time Behavior + +- Search finds child paragraph via vector match +- Return full parent cluster as context to LLM + +### Pros & Cons + +- **Pros:** Rich context, no headings needed, works on flat documents +- **Cons:** More storage (both parent + child vectors), complex retrieval + +### Configuration + +Uses `semantic_threshold` for clustering. + +--- + +## Summary Table + +| Strategy | Split Method | Requires LLM | Requires Embeddings | Speed | Cost | +|----------|--------------|--------------|---------------------|-------|------| +| fixed_size | Token count | No | No | Fast | Free | +| recursive | Separator cascade | No | No | Fast | Free | +| semantic | Similarity threshold | No | Yes (at chunk time) | Medium | Low | +| contextual_retrieval | Fixed-size + LLM context | Yes (per chunk) | No | Slow | High | +| semantic_parent_child | Similarity clustering | No | Yes (at chunk time) | Medium | Low | diff --git a/docs/phases.md b/docs/phases.md new file mode 100644 index 0000000..b583d06 --- /dev/null +++ b/docs/phases.md @@ -0,0 +1,115 @@ +# Phases & Steps + +Implementation plan for the RAG Chunking Strategy Benchmarking Framework. +Each step maps back to tasks in [tasks.md](tasks.md). + +--- + +## Phase 1: Document Parsing + Storage + +Foundation layer. Everything else depends on this. + +| Step | What | Files | Status | +|------|------|-------|--------| +| 1 | SQLite Storage Layer | `src/storage/sqlite.py` | DONE | +| 2 | Qdrant Storage Layer | `src/storage/qdrant.py` | DONE | +| 3 | Document Parser | `src/documents/parser.py` | DONE | +| 4 | Document Routes + Service | `src/documents/routes.py`, `src/documents/service.py`, `src/documents/models.py` | DONE | + +**Endpoints delivered:** +- `POST /documents` — upload .docx +- `POST /documents/{id}/process` — run strategies (stubbed) +- `DELETE /documents/{id}` — remove document + vectors +- `GET /strategies` — list all 5 strategies + +--- + +## Phase 2: Chunking Strategies + +The core evaluation targets. Each strategy is an independent module. + +| Step | What | Files | Status | +|------|------|-------|--------| +| 5 | Base Strategy Interface | `src/chunking/base.py` | TODO | +| 6 | Five Strategy Implementations | `src/chunking/strategies/recursive.py`, `markdown_structure.py`, `semantic.py`, `parent_child.py`, `contextual_structure.py` | TODO | +| 7 | Embedding Service | `src/chunking/embedding.py` | TODO | +| 8 | Chunking Orchestration | `src/chunking/service.py` | TODO | + +**ADR alignment:** +- ADR 0001: Unified Chunk model (nullable fields) +- ADR 0002: Hard-fail on contextual enrichment failure +- ADR 0003: Per-strategy failure isolation +- ADR 0005: Hardcoded article-level parents +- ADR 0011: Embed full enriched text for contextual +- ADR 0012: Sentence-level semantic with min chunk size +- ADR 0013: Leaf-level markdown chunks only +- ADR 0015: Direct API usage, no LangChain/LlamaIndex + +**Delivers:** The `POST /documents/{id}/process` endpoint goes from stub to real — chunking, embedding, and Qdrant storage for all 5 strategies. + +--- + +## Phase 3: Query Pipeline + +Retrieval + generation. The bridge between storage and evaluation. + +| Step | What | Files | Status | +|------|------|-------|--------| +| 9 | Query Service | `src/benchmarking/query_service.py` | TODO | +| 10 | Query Routes | `src/benchmarking/routes.py` | TODO | + +**Endpoints delivered:** +- `POST /queries` — ask a question against a strategy +- `GET /queries/{id}` — retrieve past query + +**Pipeline:** question → embed → vector search (Qdrant) → top_k chunks → gpt-4o-mini → answer + metadata. + +--- + +## Phase 4: Benchmarking + Evaluation + +The reason the project exists. Compare strategies head-to-head. + +| Step | What | Files | Status | +|------|------|-------|--------| +| 11 | LLM-as-Judge Evaluation | `src/benchmarking/evaluation.py` | TODO | +| 12 | Benchmark Service | `src/benchmarking/benchmark_service.py` | TODO | +| 13 | Benchmark Routes | extends `src/benchmarking/routes.py` | TODO | +| 14 | HTML Report Template | `src/benchmarking/templates/report.html` | TODO | + +**Endpoints delivered:** +- `POST /benchmarks` — run benchmark (single or dataset) +- `POST /benchmarks?dry_run=true` — cost estimate only +- `GET /benchmarks/{id}` — retrieve experiment results +- `GET /benchmarks/{id}/report` — download HTML report + +**Evaluation metrics (ADR 0004):** +- Context Relevance (1–10) +- Answer Similarity (1–10) +- Faithfulness (1–10) +- Hallucination (bool) + +**Modes:** +- Single question +- Dataset (questions.json) +- Dry run (cost estimation) + +--- + +## Phase 5: Wiring + Verification + +Polish, integration, and proof that it all works end-to-end. + +| Step | What | Files | Status | +|------|------|-------|--------| +| 15 | Wire All Routers + Logging | `src/main.py` | TODO | +| 16 | End-to-End Smoke Test | test suite | TODO | + +**Verification checklist:** +- Upload insurance .docx +- Process with all 5 strategies +- Ask 3 test questions +- Run mini benchmark (3 questions × 5 strategies) +- Verify HTML report renders +- Confirm per-strategy failure isolation (simulate one strategy failure) +- Confirm dry-run cost estimation matches actual cost within 10% diff --git a/docs/tasks.md b/docs/tasks.md new file mode 100644 index 0000000..bcdf4e9 --- /dev/null +++ b/docs/tasks.md @@ -0,0 +1,56 @@ +# Tasks + +All tasks for the RAG Chunking Strategy Benchmarking Framework. +Each task maps to a Phase and Step in [phases.md](phases.md). + +Status legend: `DONE` `IN_PROGRESS` `TODO` + +## Phase 1 — Document Parsing + Storage + +| # | Task | Status | Step | +|---|------|--------|------| +| 1 | Design FastAPI application architecture and project structure | DONE | — | +| 2 | Implement DOCX document loading and processing pipeline | DONE | 3 | +| 3 | Add configurable application settings and experiment parameters | DONE | 1 | +| 4 | Implement Qdrant vector database integration and collection management | DONE | 2 | + +## Phase 2 — Chunking Strategies + +| # | Task | Status | Step | +|---|------|--------|------| +| 5 | Create chunking strategy interface and abstraction layer | TODO | 5 | +| 6 | Implement Structure-Aware Markdown chunking strategy | TODO | 6 | +| 7 | Implement Recursive chunking strategy | TODO | 6 | +| 8 | Implement Semantic chunking strategy | TODO | 6 | +| 9 | Implement Parent-Child chunking strategy | TODO | 6 | +| 10 | Implement Contextual Structure-Aware chunking strategy | TODO | 6 | +| 11 | Implement OpenAI embedding service using text-embedding-3-small | TODO | 7 | +| 12 | Implement strategy-based document processing API | TODO | 8 | + +## Phase 3 — Query Pipeline + +| # | Task | Status | Step | +|---|------|--------|------| +| 13 | Implement question answering API with configurable chunking strategy selection | TODO | 9–10 | + +## Phase 4 — Benchmarking + Evaluation + +| # | Task | Status | Step | +|---|------|--------|------| +| 14 | Implement benchmarking pipeline for comparing chunking strategies | TODO | 12 | +| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | TODO | 11 | +| 16 | Implement experiment tracking and result storage system | TODO | 12 | +| 17 | Create question-answer evaluation dataset from insurance regulation document | TODO | 12 | +| 18 | Implement HTML benchmark report generation system | TODO | 14 | +| 19 | Design HTML report structure for experiment comparison and visualization | TODO | 14 | +| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 | + +## Phase 5 — Wiring + Verification + +| # | Task | Status | Step | +|---|------|--------|------| +| 21 | Add logging, request tracking, and cost monitoring | TODO | 15 | +| 22 | Create API documentation and Swagger examples | TODO | 15 | +| 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 | +| 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 | +| 25 | Analyze and compare chunking strategy performance results | TODO | 16 |