docs: add phases, tasks, and chunking strategies documentation
Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies
This commit is contained in:
115
docs/phases.md
Normal file
115
docs/phases.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Phases & Steps
|
||||
|
||||
Implementation plan for the RAG Chunking Strategy Benchmarking Framework.
|
||||
Each step maps back to tasks in [tasks.md](tasks.md).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Document Parsing + Storage
|
||||
|
||||
Foundation layer. Everything else depends on this.
|
||||
|
||||
| Step | What | Files | Status |
|
||||
|------|------|-------|--------|
|
||||
| 1 | SQLite Storage Layer | `src/storage/sqlite.py` | DONE |
|
||||
| 2 | Qdrant Storage Layer | `src/storage/qdrant.py` | DONE |
|
||||
| 3 | Document Parser | `src/documents/parser.py` | DONE |
|
||||
| 4 | Document Routes + Service | `src/documents/routes.py`, `src/documents/service.py`, `src/documents/models.py` | DONE |
|
||||
|
||||
**Endpoints delivered:**
|
||||
- `POST /documents` — upload .docx
|
||||
- `POST /documents/{id}/process` — run strategies (stubbed)
|
||||
- `DELETE /documents/{id}` — remove document + vectors
|
||||
- `GET /strategies` — list all 5 strategies
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Chunking Strategies
|
||||
|
||||
The core evaluation targets. Each strategy is an independent module.
|
||||
|
||||
| Step | What | Files | Status |
|
||||
|------|------|-------|--------|
|
||||
| 5 | Base Strategy Interface | `src/chunking/base.py` | TODO |
|
||||
| 6 | Five Strategy Implementations | `src/chunking/strategies/recursive.py`, `markdown_structure.py`, `semantic.py`, `parent_child.py`, `contextual_structure.py` | TODO |
|
||||
| 7 | Embedding Service | `src/chunking/embedding.py` | TODO |
|
||||
| 8 | Chunking Orchestration | `src/chunking/service.py` | TODO |
|
||||
|
||||
**ADR alignment:**
|
||||
- ADR 0001: Unified Chunk model (nullable fields)
|
||||
- ADR 0002: Hard-fail on contextual enrichment failure
|
||||
- ADR 0003: Per-strategy failure isolation
|
||||
- ADR 0005: Hardcoded article-level parents
|
||||
- ADR 0011: Embed full enriched text for contextual
|
||||
- ADR 0012: Sentence-level semantic with min chunk size
|
||||
- ADR 0013: Leaf-level markdown chunks only
|
||||
- ADR 0015: Direct API usage, no LangChain/LlamaIndex
|
||||
|
||||
**Delivers:** The `POST /documents/{id}/process` endpoint goes from stub to real — chunking, embedding, and Qdrant storage for all 5 strategies.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Query Pipeline
|
||||
|
||||
Retrieval + generation. The bridge between storage and evaluation.
|
||||
|
||||
| Step | What | Files | Status |
|
||||
|------|------|-------|--------|
|
||||
| 9 | Query Service | `src/benchmarking/query_service.py` | TODO |
|
||||
| 10 | Query Routes | `src/benchmarking/routes.py` | TODO |
|
||||
|
||||
**Endpoints delivered:**
|
||||
- `POST /queries` — ask a question against a strategy
|
||||
- `GET /queries/{id}` — retrieve past query
|
||||
|
||||
**Pipeline:** question → embed → vector search (Qdrant) → top_k chunks → gpt-4o-mini → answer + metadata.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Benchmarking + Evaluation
|
||||
|
||||
The reason the project exists. Compare strategies head-to-head.
|
||||
|
||||
| Step | What | Files | Status |
|
||||
|------|------|-------|--------|
|
||||
| 11 | LLM-as-Judge Evaluation | `src/benchmarking/evaluation.py` | TODO |
|
||||
| 12 | Benchmark Service | `src/benchmarking/benchmark_service.py` | TODO |
|
||||
| 13 | Benchmark Routes | extends `src/benchmarking/routes.py` | TODO |
|
||||
| 14 | HTML Report Template | `src/benchmarking/templates/report.html` | TODO |
|
||||
|
||||
**Endpoints delivered:**
|
||||
- `POST /benchmarks` — run benchmark (single or dataset)
|
||||
- `POST /benchmarks?dry_run=true` — cost estimate only
|
||||
- `GET /benchmarks/{id}` — retrieve experiment results
|
||||
- `GET /benchmarks/{id}/report` — download HTML report
|
||||
|
||||
**Evaluation metrics (ADR 0004):**
|
||||
- Context Relevance (1–10)
|
||||
- Answer Similarity (1–10)
|
||||
- Faithfulness (1–10)
|
||||
- Hallucination (bool)
|
||||
|
||||
**Modes:**
|
||||
- Single question
|
||||
- Dataset (questions.json)
|
||||
- Dry run (cost estimation)
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Wiring + Verification
|
||||
|
||||
Polish, integration, and proof that it all works end-to-end.
|
||||
|
||||
| Step | What | Files | Status |
|
||||
|------|------|-------|--------|
|
||||
| 15 | Wire All Routers + Logging | `src/main.py` | TODO |
|
||||
| 16 | End-to-End Smoke Test | test suite | TODO |
|
||||
|
||||
**Verification checklist:**
|
||||
- Upload insurance .docx
|
||||
- Process with all 5 strategies
|
||||
- Ask 3 test questions
|
||||
- Run mini benchmark (3 questions × 5 strategies)
|
||||
- Verify HTML report renders
|
||||
- Confirm per-strategy failure isolation (simulate one strategy failure)
|
||||
- Confirm dry-run cost estimation matches actual cost within 10%
|
||||
Reference in New Issue
Block a user