docs: add phases, tasks, and chunking strategies documentation

Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies
2026-07-26 09:38:08 +03:30
parent 29edbf57c4
commit 292ae17cfa
3 changed files with 315 additions and 0 deletions
--- a/docs/chunking_strategies.md
+++ b/docs/chunking_strategies.md
@@ -0,0 +1,144 @@
+# Chunking Strategies
+
+This document explains the 5 chunking strategies implemented in the RAG Chunking Strategy Benchmarking Framework.
+
+---
+
+## 1. FIXED_SIZE (Baseline)
+
+**How it works:** Splits text into chunks of N tokens with M token overlap.
+
+### Algorithm
+
+1. Encode full markdown into tokens
+2. Take first `chunk_size` tokens as chunk 1
+3. Slide forward by `chunk_size - overlap` tokens
+4. Repeat until end of text
+
+### Pros & Cons
+
+- **Pros:** Simple, predictable, fast, no dependencies
+- **Cons:** Ignores meaning — can split mid-sentence, mid-word, or across topics
+
+### Configuration
+
+Uses `chunk_size` and `chunk_overlap` from settings.
+
+---
+
+## 2. RECURSIVE (Cascade Splitting)
+
+**How it works:** Tries to split on meaningful boundaries first, falling back to less meaningful ones.
+
+### Separator Cascade (in order)
+
+1. Markdown headers (`#`, `##`, `###`)
+2. Double newline (`\n\n`) — paragraph breaks
+3. Single newline (`\n`) — line breaks
+4. Sentence endings (`. ! ?` + space)
+5. Space (word-level, last resort)
+
+### Algorithm
+
+1. Try splitting by highest-priority separator
+2. If parts are still too big, recurse with next separator
+3. Merge small parts back up to target size
+
+### Pros & Cons
+
+- **Pros:** Respects document structure, produces natural chunks
+- **Cons:** Still rule-based, no semantic understanding
+
+---
+
+## 3. SEMANTIC (Similarity-Based)
+
+**How it works:** Groups sentences by meaning — when similarity drops, it starts a new chunk.
+
+### Algorithm
+
+1. Split markdown into sentences
+2. Embed each sentence via OpenAI (`text-embedding-3-small`)
+3. Compute cosine similarity between adjacent sentences
+4. When similarity < `semantic_threshold`, create chunk boundary
+5. Enforce minimum chunk size (`semantic_min_chunk_size` sentences)
+
+### Pros & Cons
+
+- **Pros:** Respects topic changes, produces coherent chunks
+- **Cons:** Requires embeddings at chunk-time (API calls), slower, costs money
+
+### Configuration
+
+- `semantic_threshold` (default 0.5)
+- `semantic_min_chunk_size` (default 5)
+
+---
+
+## 4. CONTEXTUAL_RETRIEVAL (LLM-Enriched)
+
+**How it works:** Based on Anthropic's research — prepends a short context summary to each chunk before embedding.
+
+### Algorithm
+
+1. Split text using fixed-size token splitting (same as #1)
+2. For each chunk, send surrounding text + chunk to LLM
+3. LLM generates a 1-2 sentence context prefix
+4. Enriched chunk = context prefix + original text
+
+### Example Output
+
+```
+This section discusses insurance claim deadlines for property damage...
+
+[Original chunk text about specific deadlines...]
+```
+
+### Pros & Cons
+
+- **Pros:** Improved retrieval by 49% in Anthropic's benchmarks
+- **Cons:** Expensive (1 LLM call per chunk), slowest strategy
+
+### Configuration
+
+Uses `llm_model` (gpt-4o-mini) for context generation.
+
+---
+
+## 5. SEMANTIC_PARENT_CHILD (Hierarchical)
+
+**How it works:** Groups paragraphs into semantic clusters. Each cluster is a parent; each paragraph is a child.
+
+### Algorithm
+
+1. Split markdown into paragraphs
+2. Embed each paragraph
+3. Cluster consecutive paragraphs by similarity (threshold-based)
+4. Each cluster = parent chunk (full cluster text)
+5. Each paragraph = child chunk (linked to parent)
+
+### Query-time Behavior
+
+- Search finds child paragraph via vector match
+- Return full parent cluster as context to LLM
+
+### Pros & Cons
+
+- **Pros:** Rich context, no headings needed, works on flat documents
+- **Cons:** More storage (both parent + child vectors), complex retrieval
+
+### Configuration
+
+Uses `semantic_threshold` for clustering.
+
+---
+
+## Summary Table
+
+| Strategy | Split Method | Requires LLM | Requires Embeddings | Speed | Cost |
+|----------|--------------|--------------|---------------------|-------|------|
+| fixed_size | Token count | No | No | Fast | Free |
+| recursive | Separator cascade | No | No | Fast | Free |
+| semantic | Similarity threshold | No | Yes (at chunk time) | Medium | Low |
+| contextual_retrieval | Fixed-size + LLM context | Yes (per chunk) | No | Slow | High |
+| semantic_parent_child | Similarity clustering | No | Yes (at chunk time) | Medium | Low |
--- a/docs/phases.md
+++ b/docs/phases.md
@@ -0,0 +1,115 @@
+# Phases & Steps
+
+Implementation plan for the RAG Chunking Strategy Benchmarking Framework.
+Each step maps back to tasks in [tasks.md](tasks.md).
+
+---
+
+## Phase 1: Document Parsing + Storage
+
+Foundation layer. Everything else depends on this.
+
+| Step | What | Files | Status |
+|------|------|-------|--------|
+| 1 | SQLite Storage Layer | `src/storage/sqlite.py` | DONE |
+| 2 | Qdrant Storage Layer | `src/storage/qdrant.py` | DONE |
+| 3 | Document Parser | `src/documents/parser.py` | DONE |
+| 4 | Document Routes + Service | `src/documents/routes.py`, `src/documents/service.py`, `src/documents/models.py` | DONE |
+
+**Endpoints delivered:**
+- `POST /documents` — upload .docx
+- `POST /documents/{id}/process` — run strategies (stubbed)
+- `DELETE /documents/{id}` — remove document + vectors
+- `GET /strategies` — list all 5 strategies
+
+---
+
+## Phase 2: Chunking Strategies
+
+The core evaluation targets. Each strategy is an independent module.
+
+| Step | What | Files | Status |
+|------|------|-------|--------|
+| 5 | Base Strategy Interface | `src/chunking/base.py` | TODO |
+| 6 | Five Strategy Implementations | `src/chunking/strategies/recursive.py`, `markdown_structure.py`, `semantic.py`, `parent_child.py`, `contextual_structure.py` | TODO |
+| 7 | Embedding Service | `src/chunking/embedding.py` | TODO |
+| 8 | Chunking Orchestration | `src/chunking/service.py` | TODO |
+
+**ADR alignment:**
+- ADR 0001: Unified Chunk model (nullable fields)
+- ADR 0002: Hard-fail on contextual enrichment failure
+- ADR 0003: Per-strategy failure isolation
+- ADR 0005: Hardcoded article-level parents
+- ADR 0011: Embed full enriched text for contextual
+- ADR 0012: Sentence-level semantic with min chunk size
+- ADR 0013: Leaf-level markdown chunks only
+- ADR 0015: Direct API usage, no LangChain/LlamaIndex
+
+**Delivers:** The `POST /documents/{id}/process` endpoint goes from stub to real — chunking, embedding, and Qdrant storage for all 5 strategies.
+
+---
+
+## Phase 3: Query Pipeline
+
+Retrieval + generation. The bridge between storage and evaluation.
+
+| Step | What | Files | Status |
+|------|------|-------|--------|
+| 9 | Query Service | `src/benchmarking/query_service.py` | TODO |
+| 10 | Query Routes | `src/benchmarking/routes.py` | TODO |
+
+**Endpoints delivered:**
+- `POST /queries` — ask a question against a strategy
+- `GET /queries/{id}` — retrieve past query
+
+**Pipeline:** question → embed → vector search (Qdrant) → top_k chunks → gpt-4o-mini → answer + metadata.
+
+---
+
+## Phase 4: Benchmarking + Evaluation
+
+The reason the project exists. Compare strategies head-to-head.
+
+| Step | What | Files | Status |
+|------|------|-------|--------|
+| 11 | LLM-as-Judge Evaluation | `src/benchmarking/evaluation.py` | TODO |
+| 12 | Benchmark Service | `src/benchmarking/benchmark_service.py` | TODO |
+| 13 | Benchmark Routes | extends `src/benchmarking/routes.py` | TODO |
+| 14 | HTML Report Template | `src/benchmarking/templates/report.html` | TODO |
+
+**Endpoints delivered:**
+- `POST /benchmarks` — run benchmark (single or dataset)
+- `POST /benchmarks?dry_run=true` — cost estimate only
+- `GET /benchmarks/{id}` — retrieve experiment results
+- `GET /benchmarks/{id}/report` — download HTML report
+
+**Evaluation metrics (ADR 0004):**
+- Context Relevance (1–10)
+- Answer Similarity (1–10)
+- Faithfulness (1–10)
+- Hallucination (bool)
+
+**Modes:**
+- Single question
+- Dataset (questions.json)
+- Dry run (cost estimation)
+
+---
+
+## Phase 5: Wiring + Verification
+
+Polish, integration, and proof that it all works end-to-end.
+
+| Step | What | Files | Status |
+|------|------|-------|--------|
+| 15 | Wire All Routers + Logging | `src/main.py` | TODO |
+| 16 | End-to-End Smoke Test | test suite | TODO |
+
+**Verification checklist:**
+- Upload insurance .docx
+- Process with all 5 strategies
+- Ask 3 test questions
+- Run mini benchmark (3 questions × 5 strategies)
+- Verify HTML report renders
+- Confirm per-strategy failure isolation (simulate one strategy failure)
+- Confirm dry-run cost estimation matches actual cost within 10%
--- a/docs/tasks.md
+++ b/docs/tasks.md
@@ -0,0 +1,56 @@
+# Tasks
+
+All tasks for the RAG Chunking Strategy Benchmarking Framework.
+Each task maps to a Phase and Step in [phases.md](phases.md).
+
+Status legend: `DONE` `IN_PROGRESS` `TODO`
+
+## Phase 1 — Document Parsing + Storage
+
+| # | Task | Status | Step |
+|---|------|--------|------|
+| 1 | Design FastAPI application architecture and project structure | DONE | — |
+| 2 | Implement DOCX document loading and processing pipeline | DONE | 3 |
+| 3 | Add configurable application settings and experiment parameters | DONE | 1 |
+| 4 | Implement Qdrant vector database integration and collection management | DONE | 2 |
+
+## Phase 2 — Chunking Strategies
+
+| # | Task | Status | Step |
+|---|------|--------|------|
+| 5 | Create chunking strategy interface and abstraction layer | TODO | 5 |
+| 6 | Implement Structure-Aware Markdown chunking strategy | TODO | 6 |
+| 7 | Implement Recursive chunking strategy | TODO | 6 |
+| 8 | Implement Semantic chunking strategy | TODO | 6 |
+| 9 | Implement Parent-Child chunking strategy | TODO | 6 |
+| 10 | Implement Contextual Structure-Aware chunking strategy | TODO | 6 |
+| 11 | Implement OpenAI embedding service using text-embedding-3-small | TODO | 7 |
+| 12 | Implement strategy-based document processing API | TODO | 8 |
+
+## Phase 3 — Query Pipeline
+
+| # | Task | Status | Step |
+|---|------|--------|------|
+| 13 | Implement question answering API with configurable chunking strategy selection | TODO | 9–10 |
+
+## Phase 4 — Benchmarking + Evaluation
+
+| # | Task | Status | Step |
+|---|------|--------|------|
+| 14 | Implement benchmarking pipeline for comparing chunking strategies | TODO | 12 |
+| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | TODO | 11 |
+| 16 | Implement experiment tracking and result storage system | TODO | 12 |
+| 17 | Create question-answer evaluation dataset from insurance regulation document | TODO | 12 |
+| 18 | Implement HTML benchmark report generation system | TODO | 14 |
+| 19 | Design HTML report structure for experiment comparison and visualization | TODO | 14 |
+| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 |
+
+## Phase 5 — Wiring + Verification
+
+| # | Task | Status | Step |
+|---|------|--------|------|
+| 21 | Add logging, request tracking, and cost monitoring | TODO | 15 |
+| 22 | Create API documentation and Swagger examples | TODO | 15 |
+| 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 |
+| 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 |
+| 25 | Analyze and compare chunking strategy performance results | TODO | 16 |