mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add project documentation and task tracking

Browse Source 
Why:
- Need to track domain model, ADRs, task progress, and feature planning

Changes:
- CONTEXT.md: domain model with ADRs 0001-0014
- docs/tasks.md: updated task list with Phase 6 (Dashboard)
- docs/cant-do-yet.md: backend-ready but no UI features
- docs/out-of-scope-v1.md: intentionally excluded features

Impact:
- Project documentation centralized for reference
This commit is contained in:
Mahdi Bazrafshan
2026-07-29 17:55:01 +03:30
parent a49795e5eb
commit d3f8a9a5e5
4 changed files with 186 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/cant-do-yet.md Normal file
View File

@@ -0,0 +1,53 @@
# What You Can't Do Yet (But Backend Is Ready)
Features where the backend API is implemented and tested, but the dashboard UI does not yet expose them.
## 1. Chunk Preview Panel (Admin Tab)
**Backend**: `GET /admin/chunks/{doc_id}?strategy=...` returns all chunks for a document and strategy — chunk text (truncated to 500 chars), chunk index, token count, character count, parent_id.
**Missing UI**: No panel in the Admin tab to select a document + strategy and visualize the actual chunk text blocks, their sizes, and hierarchy.
## 2. Questions Dataset Management (Admin Tab)
**Backend**:
- `GET /admin/questions` — lists JSON files in `files/` with question counts
- `POST /admin/questions/upload` — uploads a new questions JSON file
- `GET /admin/questions/{file_id}` — returns full question set
- `DELETE /admin/questions/{file_id}` — deletes a question file
**Missing UI**: No section in the Admin tab to browse, upload, view, or delete question datasets. You can only use the default `files/questions.json` by typing its path in the Benchmarks tab.
## 3. Standalone Cost Estimator (Admin Tab)
**Backend**: `POST /admin/cost-estimate` accepts `num_questions` and `num_strategies`, returns full cost breakdown (embedding, queries, evaluation, token estimates, total USD).
**Missing UI**: No standalone cost estimation panel in the Admin tab. The Benchmarks tab has a "Cost Estimate" button, but it is tied to the benchmark form and requires a document selection.
## 4. Experiment Detail View (Benchmarks Tab)
**Backend**: `GET /benchmarks/{experiment_id}` returns full experiment data — per-question results, per-strategy scores, aggregate metrics, benchmark config.
**Missing UI**: Dashboard shows a list of experiments with a "Report" link that opens the raw HTML report in a new tab. No in-dashboard detail view with side-by-side strategy comparison tables, per-question drill-down, or visual charts.
## 5. Query History (Query Tab)
**Backend**: `GET /queries/{query_id}` retrieves any past query. All queries are stored in SQLite with full results (answer, retrieved chunks, latency, tokens).
**Missing UI**: No list of past queries. When you navigate away from the Query tab, the current result disappears from the screen. There is no history panel to browse, search, or re-view previous questions and answers.
## 6. Document Content Preview (Documents Tab)
**Backend**: `GET /documents` returns `parsed_text_preview` and `document_tree` fields in the detail response model (`DocumentDetailResponse`). SQLite stores the full parsed text and document tree.
**Missing UI**: You can upload and process documents, but you cannot preview the actual document text or the hierarchical document tree (sections, articles, paragraphs) inside the dashboard.
## 7. Real-Time Processing Status (Documents Tab)
**Backend**: The processing endpoint (`POST /documents/{id}/process`) returns per-strategy results — `strategies_completed` and `strategies_failed` with status, chunk counts, and error messages.
**Missing UI**: When you click "Process", the button shows a spinner. You don't see which strategies are running, which have completed, or which failed. If it fails partway through 5 strategies, you don't know which one broke without checking the server logs.
## 8. Global Error Handling and Retry
**Missing**: If the server returns an error (network down, Qdrant disconnected, OpenAI rate limit), you see a toast notification that disappears after 3 seconds. There is no retry button, no persistent error banner, and no automatic reconnection. A failed operation must be manually re-attempted by clicking the button again.

46
docs/out-of-scope-v1.md Normal file
View File

@@ -0,0 +1,46 @@
# What's Completely Out of Scope for V1
Features that are intentionally excluded from the first version of the dashboard. These would require significant additional architecture, infrastructure, or design work that doesn't serve the core use case of a single-user localhost benchmarking tool.
## User Authentication and Multi-User Support
No login, no sessions, no role-based access control. The dashboard runs on localhost and is operated by a single user. Adding auth would introduce JWT/session management, user storage, and permission logic — none of which serve the current workflow.
## Exporting Results to CSV/PDF
No built-in export of experiment results, query logs, or benchmark comparisons to CSV, PDF, or Excel. The HTML report endpoint (`/benchmarks/{id}/report`) is the closest thing to an export. If you need raw data, use the API endpoints directly or query SQLite.
## Custom Chunking Parameters from the UI
You cannot change `chunk_size`, `chunk_overlap`, `semantic_threshold`, or `semantic_min_chunk_size` from the dashboard. These are hardcoded in `src/core/config.py` with defaults (512 tokens, 50 overlap, 0.3 threshold). Changing them requires editing the config and restarting the server. The UI uses whatever the server has configured.
## Streaming Responses
Answers from GPT-4o-mini appear all at once after the full response is generated. There is no Server-Sent Events (SSE) or WebSocket connection to stream the answer word-by-word as it is generated. The latency breakdown shows how long generation took, but you wait for the complete answer.
## Dark/Light Theme Toggle
The dashboard is locked to the custom dark theme (#111113 base, #eab308 amber accent). There is no theme switcher or light mode alternative. The design tokens are hardcoded as CSS custom properties in the HTML file.
## Mobile Responsive Layout
The dashboard is designed for desktop screens (max-width 1200px, tab bar, tables). It is not optimized for phones or tablets. Tables will overflow, the tab bar may wrap, and the drop zone will be small on mobile. A responsive redesign with breakpoints is not planned for V1.
## WebSocket for Real-Time Updates
No persistent WebSocket connections. All communication is via standard HTTP REST requests. This means:
- Processing status updates require polling or a full page refresh
- No live push when a benchmark completes
- No live updating of collection point counts as chunks are inserted
## Batch Operations
You cannot process multiple documents at once, run multiple benchmarks in parallel, or delete several documents simultaneously. Each operation targets a single document. If you need to process 10 documents, you click "Process" 10 times.
## Search Across Documents or Experiments
No full-text search or filtering within the dashboard. You cannot search for a specific question text, filter experiments by date range, or find documents by partial filename. The document list and experiment list show everything in chronological order with no search or filter controls.
## Configuration Management from the UI
You cannot view or edit the server configuration (OpenAI model, temperature, max_tokens, embedding model, Qdrant URL, database path) from the dashboard. The `/admin/config` endpoint was deferred to V1.1. All configuration is managed via the `.env` file and `src/core/config.py`.

62
docs/tasks.md
View File

@@ -18,39 +18,67 @@ Status legend: `DONE` `IN_PROGRESS` `TODO`
| # | Task | Status | Step |
|---|------|--------|------|
| 5 | Create chunking strategy interface and abstraction layer | TODO | 5 |
| 6 | Implement Structure-Aware Markdown chunking strategy | TODO | 6 |
| 7 | Implement Recursive chunking strategy | TODO | 6 |
| 8 | Implement Semantic chunking strategy | TODO | 6 |
| 9 | Implement Parent-Child chunking strategy | TODO | 6 |
| 10 | Implement Contextual Structure-Aware chunking strategy | TODO | 6 |
| 11 | Implement OpenAI embedding service using text-embedding-3-small | TODO | 7 |
| 12 | Implement strategy-based document processing API | TODO | 8 |
| 5 | Create chunking strategy interface and abstraction layer | DONE | 5 |
| 6 | Implement Structure-Aware Markdown chunking strategy | DONE | 6 |
| 7 | Implement Recursive chunking strategy | DONE | 6 |
| 8 | Implement Semantic chunking strategy | DONE | 6 |
| 9 | Implement Parent-Child chunking strategy | DONE | 6 |
| 10 | Implement Contextual Structure-Aware chunking strategy | DONE | 6 |
| 11 | Implement OpenAI embedding service using text-embedding-3-small | DONE | 7 |
| 12 | Implement strategy-based document processing API | DONE | 8 |
## Phase 3 — Query Pipeline
| # | Task | Status | Step |
|---|------|--------|------|
| 13 | Implement question answering API with configurable chunking strategy selection | TODO | 9–10 |
| 13 | Implement question answering API with configurable chunking strategy selection | DONE | 9–10 |
## Phase 4 — Benchmarking + Evaluation
| # | Task | Status | Step |
|---|------|--------|------|
| 14 | Implement benchmarking pipeline for comparing chunking strategies | TODO | 12 |
| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | TODO | 11 |
| 16 | Implement experiment tracking and result storage system | TODO | 12 |
| 17 | Create question-answer evaluation dataset from insurance regulation document | TODO | 12 |
| 18 | Implement HTML benchmark report generation system | TODO | 14 |
| 19 | Design HTML report structure for experiment comparison and visualization | TODO | 14 |
| 14 | Implement benchmarking pipeline for comparing chunking strategies | DONE | 12 |
| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | DONE | 11 |
| 16 | Implement experiment tracking and result storage system | DONE | 12 |
| 17 | Create question-answer evaluation dataset from insurance regulation document | DONE | 12 |
| 18 | Implement HTML benchmark report generation system | DONE | 14 |
| 19 | Design HTML report structure for experiment comparison and visualization | DONE | 14 |
| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 |
## Phase 5 — Wiring + Verification
| # | Task | Status | Step |
|---|------|--------|------|
| 21 | Add logging, request tracking, and cost monitoring | TODO | 15 |
| 22 | Create API documentation and Swagger examples | TODO | 15 |
| 21 | Add logging, request tracking, and cost monitoring | DONE | 15 |
| 22 | Create API documentation and Swagger examples | DONE | 15 |
| 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 |
| 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 |
| 25 | Analyze and compare chunking strategy performance results | TODO | 16 |
## Phase 6 — Admin Dashboard (UI)
| # | Task | Status | Note |
|---|------|--------|------|
| 26 | Design dashboard architecture (ADR-0001 through ADR-0009) | DONE | Babel JSX, persistent tabs, dark theme, /app mount |
| 27 | Create frontend skeleton: index.html with tab bar, dark theme, design tokens | DONE | src/static/index.html, 889 lines |
| 28 | Wire FastAPI to serve dashboard at /app (StaticFiles mount) | DONE | src/main.py modified |
| 29 | Implement admin backend: /admin/health endpoint | DONE | src/admin/routes.py + service.py |
| 30 | Implement admin backend: /admin/qdrant/* CRUD endpoints | DONE | Collections list, create, delete, wipe points |
| 31 | Implement admin backend: /admin/chunks/{doc_id} chunk preview | DONE | Backend only, no frontend panel |
| 32 | Implement admin backend: /admin/questions/* CRUD endpoints | DONE | Backend only, no frontend panel |
| 33 | Implement admin backend: /admin/cost-estimate endpoint | DONE | Backend only, no frontend panel |
| 34 | Build Documents tab: upload, list, process, delete | DONE | Drag-drop, strategy checkboxes, all wired |
| 35 | Build Home tab: status cards + quick start guide | DONE | Documents count, experiments count, server status |
| 36 | Build Query tab: form + answer display + chunk table | DONE | Document/strategy/top_k selectors, question textarea |
| 37 | Build Benchmarks tab: run form + experiments list | DONE | Strategies checkboxes, cost estimate, report links |
| 38 | Build Admin tab: health cards + Qdrant collections panel | DONE | Health status, collection list with delete |
| 39 | Build Admin tab: chunk preview panel | TODO | Backend ready, UI not built |
| 40 | Build Admin tab: questions dataset management panel | TODO | Backend ready, UI not built |
| 41 | Build Admin tab: standalone cost estimator panel | TODO | Backend ready, UI not built |
| 42 | Build Query tab: past queries history panel | TODO | Backend storage exists, no UI list |
| 43 | Build Benchmarks tab: in-dashboard experiment detail view | TODO | Report link exists, no drill-down UI |
| 44 | Build Documents tab: document content preview | TODO | Parsed text stored, no preview panel |
| 45 | Add real-time processing progress indicator | TODO | Spinner only, no per-strategy progress |
| 46 | Add global error handling and retry logic | TODO | Toast only, no retry button or banner |
| 47 | Create docs: cant-do-yet.md (backend ready, no UI) | DONE | docs/cant-do-yet.md |
| 48 | Create docs: out-of-scope-v1.md (intentionally excluded) | DONE | docs/out-of-scope-v1.md |