Mahdi Bazrafshan
292ae17cfa
Why: - Need implementation plan and task tracking documentation - Need strategy explanations for reference Changes: - phases.md: 5-phase implementation plan with status tracking - tasks.md: 25 tasks mapped to phases and steps - chunking_strategies.md: detailed explanations of all 5 strategies
116 lines
4.0 KiB
Markdown
116 lines
4.0 KiB
Markdown
# Phases & Steps
|
||
|
||
Implementation plan for the RAG Chunking Strategy Benchmarking Framework.
|
||
Each step maps back to tasks in [tasks.md](tasks.md).
|
||
|
||
---
|
||
|
||
## Phase 1: Document Parsing + Storage
|
||
|
||
Foundation layer. Everything else depends on this.
|
||
|
||
| Step | What | Files | Status |
|
||
|------|------|-------|--------|
|
||
| 1 | SQLite Storage Layer | `src/storage/sqlite.py` | DONE |
|
||
| 2 | Qdrant Storage Layer | `src/storage/qdrant.py` | DONE |
|
||
| 3 | Document Parser | `src/documents/parser.py` | DONE |
|
||
| 4 | Document Routes + Service | `src/documents/routes.py`, `src/documents/service.py`, `src/documents/models.py` | DONE |
|
||
|
||
**Endpoints delivered:**
|
||
- `POST /documents` — upload .docx
|
||
- `POST /documents/{id}/process` — run strategies (stubbed)
|
||
- `DELETE /documents/{id}` — remove document + vectors
|
||
- `GET /strategies` — list all 5 strategies
|
||
|
||
---
|
||
|
||
## Phase 2: Chunking Strategies
|
||
|
||
The core evaluation targets. Each strategy is an independent module.
|
||
|
||
| Step | What | Files | Status |
|
||
|------|------|-------|--------|
|
||
| 5 | Base Strategy Interface | `src/chunking/base.py` | TODO |
|
||
| 6 | Five Strategy Implementations | `src/chunking/strategies/recursive.py`, `markdown_structure.py`, `semantic.py`, `parent_child.py`, `contextual_structure.py` | TODO |
|
||
| 7 | Embedding Service | `src/chunking/embedding.py` | TODO |
|
||
| 8 | Chunking Orchestration | `src/chunking/service.py` | TODO |
|
||
|
||
**ADR alignment:**
|
||
- ADR 0001: Unified Chunk model (nullable fields)
|
||
- ADR 0002: Hard-fail on contextual enrichment failure
|
||
- ADR 0003: Per-strategy failure isolation
|
||
- ADR 0005: Hardcoded article-level parents
|
||
- ADR 0011: Embed full enriched text for contextual
|
||
- ADR 0012: Sentence-level semantic with min chunk size
|
||
- ADR 0013: Leaf-level markdown chunks only
|
||
- ADR 0015: Direct API usage, no LangChain/LlamaIndex
|
||
|
||
**Delivers:** The `POST /documents/{id}/process` endpoint goes from stub to real — chunking, embedding, and Qdrant storage for all 5 strategies.
|
||
|
||
---
|
||
|
||
## Phase 3: Query Pipeline
|
||
|
||
Retrieval + generation. The bridge between storage and evaluation.
|
||
|
||
| Step | What | Files | Status |
|
||
|------|------|-------|--------|
|
||
| 9 | Query Service | `src/benchmarking/query_service.py` | TODO |
|
||
| 10 | Query Routes | `src/benchmarking/routes.py` | TODO |
|
||
|
||
**Endpoints delivered:**
|
||
- `POST /queries` — ask a question against a strategy
|
||
- `GET /queries/{id}` — retrieve past query
|
||
|
||
**Pipeline:** question → embed → vector search (Qdrant) → top_k chunks → gpt-4o-mini → answer + metadata.
|
||
|
||
---
|
||
|
||
## Phase 4: Benchmarking + Evaluation
|
||
|
||
The reason the project exists. Compare strategies head-to-head.
|
||
|
||
| Step | What | Files | Status |
|
||
|------|------|-------|--------|
|
||
| 11 | LLM-as-Judge Evaluation | `src/benchmarking/evaluation.py` | TODO |
|
||
| 12 | Benchmark Service | `src/benchmarking/benchmark_service.py` | TODO |
|
||
| 13 | Benchmark Routes | extends `src/benchmarking/routes.py` | TODO |
|
||
| 14 | HTML Report Template | `src/benchmarking/templates/report.html` | TODO |
|
||
|
||
**Endpoints delivered:**
|
||
- `POST /benchmarks` — run benchmark (single or dataset)
|
||
- `POST /benchmarks?dry_run=true` — cost estimate only
|
||
- `GET /benchmarks/{id}` — retrieve experiment results
|
||
- `GET /benchmarks/{id}/report` — download HTML report
|
||
|
||
**Evaluation metrics (ADR 0004):**
|
||
- Context Relevance (1–10)
|
||
- Answer Similarity (1–10)
|
||
- Faithfulness (1–10)
|
||
- Hallucination (bool)
|
||
|
||
**Modes:**
|
||
- Single question
|
||
- Dataset (questions.json)
|
||
- Dry run (cost estimation)
|
||
|
||
---
|
||
|
||
## Phase 5: Wiring + Verification
|
||
|
||
Polish, integration, and proof that it all works end-to-end.
|
||
|
||
| Step | What | Files | Status |
|
||
|------|------|-------|--------|
|
||
| 15 | Wire All Routers + Logging | `src/main.py` | TODO |
|
||
| 16 | End-to-End Smoke Test | test suite | TODO |
|
||
|
||
**Verification checklist:**
|
||
- Upload insurance .docx
|
||
- Process with all 5 strategies
|
||
- Ask 3 test questions
|
||
- Run mini benchmark (3 questions × 5 strategies)
|
||
- Verify HTML report renders
|
||
- Confirm per-strategy failure isolation (simulate one strategy failure)
|
||
- Confirm dry-run cost estimation matches actual cost within 10%
|