Mahdi Bazrafshan
d3f8a9a5e5
Why: - Need to track domain model, ADRs, task progress, and feature planning Changes: - CONTEXT.md: domain model with ADRs 0001-0014 - docs/tasks.md: updated task list with Phase 6 (Dashboard) - docs/cant-do-yet.md: backend-ready but no UI features - docs/out-of-scope-v1.md: intentionally excluded features Impact: - Project documentation centralized for reference
85 lines
5.1 KiB
Markdown
85 lines
5.1 KiB
Markdown
# Tasks
|
||
|
||
All tasks for the RAG Chunking Strategy Benchmarking Framework.
|
||
Each task maps to a Phase and Step in [phases.md](phases.md).
|
||
|
||
Status legend: `DONE` `IN_PROGRESS` `TODO`
|
||
|
||
## Phase 1 — Document Parsing + Storage
|
||
|
||
| # | Task | Status | Step |
|
||
|---|------|--------|------|
|
||
| 1 | Design FastAPI application architecture and project structure | DONE | — |
|
||
| 2 | Implement DOCX document loading and processing pipeline | DONE | 3 |
|
||
| 3 | Add configurable application settings and experiment parameters | DONE | 1 |
|
||
| 4 | Implement Qdrant vector database integration and collection management | DONE | 2 |
|
||
|
||
## Phase 2 — Chunking Strategies
|
||
|
||
| # | Task | Status | Step |
|
||
|---|------|--------|------|
|
||
| 5 | Create chunking strategy interface and abstraction layer | DONE | 5 |
|
||
| 6 | Implement Structure-Aware Markdown chunking strategy | DONE | 6 |
|
||
| 7 | Implement Recursive chunking strategy | DONE | 6 |
|
||
| 8 | Implement Semantic chunking strategy | DONE | 6 |
|
||
| 9 | Implement Parent-Child chunking strategy | DONE | 6 |
|
||
| 10 | Implement Contextual Structure-Aware chunking strategy | DONE | 6 |
|
||
| 11 | Implement OpenAI embedding service using text-embedding-3-small | DONE | 7 |
|
||
| 12 | Implement strategy-based document processing API | DONE | 8 |
|
||
|
||
## Phase 3 — Query Pipeline
|
||
|
||
| # | Task | Status | Step |
|
||
|---|------|--------|------|
|
||
| 13 | Implement question answering API with configurable chunking strategy selection | DONE | 9–10 |
|
||
|
||
## Phase 4 — Benchmarking + Evaluation
|
||
|
||
| # | Task | Status | Step |
|
||
|---|------|--------|------|
|
||
| 14 | Implement benchmarking pipeline for comparing chunking strategies | DONE | 12 |
|
||
| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | DONE | 11 |
|
||
| 16 | Implement experiment tracking and result storage system | DONE | 12 |
|
||
| 17 | Create question-answer evaluation dataset from insurance regulation document | DONE | 12 |
|
||
| 18 | Implement HTML benchmark report generation system | DONE | 14 |
|
||
| 19 | Design HTML report structure for experiment comparison and visualization | DONE | 14 |
|
||
| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 |
|
||
|
||
## Phase 5 — Wiring + Verification
|
||
|
||
| # | Task | Status | Step |
|
||
|---|------|--------|------|
|
||
| 21 | Add logging, request tracking, and cost monitoring | DONE | 15 |
|
||
| 22 | Create API documentation and Swagger examples | DONE | 15 |
|
||
| 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 |
|
||
| 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 |
|
||
| 25 | Analyze and compare chunking strategy performance results | TODO | 16 |
|
||
|
||
## Phase 6 — Admin Dashboard (UI)
|
||
|
||
| # | Task | Status | Note |
|
||
|---|------|--------|------|
|
||
| 26 | Design dashboard architecture (ADR-0001 through ADR-0009) | DONE | Babel JSX, persistent tabs, dark theme, /app mount |
|
||
| 27 | Create frontend skeleton: index.html with tab bar, dark theme, design tokens | DONE | src/static/index.html, 889 lines |
|
||
| 28 | Wire FastAPI to serve dashboard at /app (StaticFiles mount) | DONE | src/main.py modified |
|
||
| 29 | Implement admin backend: /admin/health endpoint | DONE | src/admin/routes.py + service.py |
|
||
| 30 | Implement admin backend: /admin/qdrant/* CRUD endpoints | DONE | Collections list, create, delete, wipe points |
|
||
| 31 | Implement admin backend: /admin/chunks/{doc_id} chunk preview | DONE | Backend only, no frontend panel |
|
||
| 32 | Implement admin backend: /admin/questions/* CRUD endpoints | DONE | Backend only, no frontend panel |
|
||
| 33 | Implement admin backend: /admin/cost-estimate endpoint | DONE | Backend only, no frontend panel |
|
||
| 34 | Build Documents tab: upload, list, process, delete | DONE | Drag-drop, strategy checkboxes, all wired |
|
||
| 35 | Build Home tab: status cards + quick start guide | DONE | Documents count, experiments count, server status |
|
||
| 36 | Build Query tab: form + answer display + chunk table | DONE | Document/strategy/top_k selectors, question textarea |
|
||
| 37 | Build Benchmarks tab: run form + experiments list | DONE | Strategies checkboxes, cost estimate, report links |
|
||
| 38 | Build Admin tab: health cards + Qdrant collections panel | DONE | Health status, collection list with delete |
|
||
| 39 | Build Admin tab: chunk preview panel | TODO | Backend ready, UI not built |
|
||
| 40 | Build Admin tab: questions dataset management panel | TODO | Backend ready, UI not built |
|
||
| 41 | Build Admin tab: standalone cost estimator panel | TODO | Backend ready, UI not built |
|
||
| 42 | Build Query tab: past queries history panel | TODO | Backend storage exists, no UI list |
|
||
| 43 | Build Benchmarks tab: in-dashboard experiment detail view | TODO | Report link exists, no drill-down UI |
|
||
| 44 | Build Documents tab: document content preview | TODO | Parsed text stored, no preview panel |
|
||
| 45 | Add real-time processing progress indicator | TODO | Spinner only, no per-strategy progress |
|
||
| 46 | Add global error handling and retry logic | TODO | Toast only, no retry button or banner |
|
||
| 47 | Create docs: cant-do-yet.md (backend ready, no UI) | DONE | docs/cant-do-yet.md |
|
||
| 48 | Create docs: out-of-scope-v1.md (intentionally excluded) | DONE | docs/out-of-scope-v1.md |
|