mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add project documentation and task tracking

Browse Source 
Why:
- Need to track domain model, ADRs, task progress, and feature planning

Changes:
- CONTEXT.md: domain model with ADRs 0001-0014
- docs/tasks.md: updated task list with Phase 6 (Dashboard)
- docs/cant-do-yet.md: backend-ready but no UI features
- docs/out-of-scope-v1.md: intentionally excluded features

Impact:
- Project documentation centralized for reference
This commit is contained in:
Mahdi Bazrafshan
2026-07-29 17:55:01 +03:30
parent a49795e5eb
commit d3f8a9a5e5
4 changed files with 186 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files

347
CONTEXT.md
View File

@@ -1,305 +1,42 @@
# RAG Chunking Benchmarker # Context — RAG Chunking Benchmarker Admin Dashboard
A benchmarking application that compares chunking strategies on regulatory documents, measuring retrieval quality, answer faithfulness, and cost across strategies. ## Language
## Domain **Dashboard**:
A single-file React app (CDN-loaded, no build step) served by FastAPI at `/app`. Replaces Swagger as the primary UI for operating the RAG benchmarking platform.
**Chunk:** _Avoid_: Admin panel, web UI, frontend
A discrete unit of text extracted from a document, ready for embedding and storage. Every strategy produces chunks conforming to the unified Chunk model.
_Avoid_: fragment, segment, block **Strategy**:
One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child.
**Unified Chunk Model:** _Avoid_: Method, approach, technique
A single Pydantic model that all five chunking strategies must return. Strategy-specific fields (e.g., parent_id) are nullable when not applicable. This ensures Qdrant payload schemas are consistent and benchmarking compares equivalent structures.
_Avoid_: chunk schema, chunk type **Experiment**:
A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics.
**Chunk Metadata:** _Avoid_: Run, trial, benchmark
Fields stored with every chunk in Qdrant: document_name, chunk_id, strategy_name, chunk_index, token_count, character_count. No page_number (unreliable in DOCX). chunk_index is the position in the full document's chunk sequence.
_Avoid_: chunk fields, payload data **Chunk Preview**:
The ability to visualize what a specific strategy produces for a given document — the raw text blocks, their sizes, and hierarchy.
**Chunking Strategy:** _Avoid_: Chunk inspection, chunk view
An algorithm that determines how a document is split into chunks. Must inherit from the ChunkingStrategy base class and implement the chunk() method. The five strategies are: contextual_structure, parent_child, semantic, markdown_structure, recursive.
_Avoid_: splitter, chunker, method **Tab**:
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives.
**Strategy Name:** _Avoid_: Page, route, view
The canonical identifier for each chunking strategy: contextual_structure, parent_child, semantic, markdown_structure, recursive. Used in collection names, metadata, API parameters, and benchmark comparisons.
_Avoid_: strategy type, strategy id ## Architecture Decisions
**Contextual Structure Strategy:** | # | Decision | Status |
Pipeline: DOCX > Markdown > Structure Extraction > LLM enrichment (GPT-4o-mini per section) > Chunks > Embeddings > Qdrant. Each chunk is enriched with Document/Section/Article/Context/Content labels. The full enriched text is what gets embedded (not just the content). Hard-fails if the LLM enrichment call fails — no degraded chunks stored. |---|----------|--------|
_Avoid_: contextual chunking, enriched strategy ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
**Parent-Child Strategy:** ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved |
Parent chunks are Article-level units (hardcoded for v1). Child chunks are sentences or paragraphs within each Article. Only child vectors are stored in Qdrant with a parent_id reference. At retrieval time, the matching child is found via vector search, then its full parent Article is returned as context to the LLM. ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
_Avoid_: hierarchical chunking, nested strategy ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
**Parent Level:** ADR-0007 | Admin router v1 endpoints: /admin/health, /admin/qdrant/collections (CRUD + wipe), /admin/chunks/{doc_id}, /admin/questions (CRUD + upload), /admin/cost-estimate. Config/SQLite/request-log deferred to v1.1. | Approved |
The document hierarchy level used as parent in parent-child chunking. Hardcoded to "article" for v1. Configurable in future versions (article, section, paragraph) when additional document types are supported. ADR-0008 | Custom dark theme: bg #111113/#1a1a1e/#222228, border #2a2a30, text #e4e4e7/#fafafa/#71717a, accent #eab308 (amber). Inter font from Google Fonts CDN. Continuity with existing amber/green report palette. | Approved |
_Avoid_: parent hierarchy, parent depth ADR-0009 | File structure: dashboard HTML in src/static/index.html, admin backend in src/admin/ (routes.py, service.py). FastAPI mounts /app -> src/static via StaticFiles(html=True). | Approved |
ADR-0010 | Cant-do-yet implementation order: Admin panels first (chunk preview, questions mgmt, cost estimator), then cross-tab UX (experiment detail, query history, doc preview, progress), then cross-cutting (error handling). | Approved |
**Semantic Strategy:** ADR-0011 | Admin tab uses collapsible sections (accordion pattern). Each panel is a card with a clickable header toggling display. Health starts expanded, others collapsed by default. | Approved |
Pipeline: DOCX > Markdown > Sentence extraction > Sentence embeddings > Cosine similarity between adjacent sentences > Dynamic chunk creation. When similarity drops below the semantic threshold, a new chunk boundary is created. Enforces a minimum chunk size (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences). Boundary sentence stays with the previous chunk. ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved |
_Avoid_: embedding-based chunking, similarity chunking ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved |
ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
**Semantic Threshold:**
The cosine similarity value below which adjacent sentences are considered a topic shift. When cosine_sim(sentence_n, sentence_n+1) < threshold, a new chunk boundary is created. Configurable via SEMANTIC_THRESHOLD in .env.
_Avoid_: similarity cutoff, cut threshold
**Minimum Chunk Size:**
The minimum number of sentences a semantic chunk must contain. If a boundary is detected before the minimum is reached, accumulation continues. Configurable via SEMANTIC_MIN_CHUNK_SIZE in .env.
_Avoid_: min chunk, smallest chunk
**Semantic Boundary:**
A point in the document where adjacent sentence similarity drops below the semantic threshold, marking where one chunk ends and another begins.
_Avoid_: cut point, split boundary
**Markdown Structure Strategy:**
Pipeline: DOCX > Markdown > Header detection > Paragraph grouping > Leaf-level chunk creation > Qdrant. Chunks are created only at the deepest header level (leaf chunks). No overlap between parent and child sections — a ## section is NOT a chunk if it contains ### subsections.
_Avoid_: header-based chunking, markdown chunking
**Leaf Chunk:**
A chunk at the deepest header level in markdown chunking. Only leaf-level chunks are created (no overlap with parent sections).
_Avoid_: atomic chunk, granular chunk
**Recursive Strategy:**
Pipeline: DOCX > Markdown > Cascade splitting using a separator hierarchy. Separators applied in order: (1) Markdown headers (#, ##, ###), (2) double newline (\n\n), (3) single newline (\n), (4) sentence-ending punctuation (. ! ? followed by space), (5) space (word-level, last resort). Splitting stops when chunks reach the target size. Each chunk stores which separator level was used.
_Avoid_: hierarchical splitting, recursive splitting
**Separator Cascade:**
The ordered list of separators used by recursive chunking: headers > double newline > single newline > sentence-ending punctuation > word boundary. Applied top-down; each level is tried before falling back to the next.
_Avoid_: separator list, split pattern
**Document Tree:**
The hierarchical structure of a parsed document: Document > Section > Article > Paragraph. Extracted once via python-docx at upload time, stored in SQLite, and shared across all strategies. Strategies that need hierarchy (contextual, parent_child, markdown) read from the stored tree; they do not re-parse.
_Avoid_: doc tree, document structure, hierarchy
**Embedding:**
The vector representation of a chunk's text, produced by text-embedding-3-small. For the contextual_structure strategy, the full enriched text is embedded. For all other strategies, the raw chunk text is embedded.
_Avoid_: vector, vector representation, encoding
**Top-k:**
The number of chunks to retrieve during vector similarity search. Default is 5. Configurable per query via the API (POST /queries) and per benchmark via the dataset configuration.
_Avoid_: k, retrieve count, result limit
**Embedding Model:**
The fixed OpenAI model used for all embedding operations: text-embedding-3-small. Not configurable — every strategy uses the same embedding model to ensure fair comparison.
_Avoid_: vector model, encoder model
**LLM Model:**
The fixed OpenAI model used for all generation and evaluation: gpt-4o-mini. Used for: (1) contextual enrichment during chunking, (2) answer generation during queries, (3) LLM-as-judge evaluation during benchmarks.
_Avoid_: generation model, chat model
## Storage
**Strategy Collection:**
A Qdrant collection named after a strategy. Naming convention: {strategy_name}_collection (e.g., contextual_structure_collection, semantic_collection). Each strategy writes to its own collection. All collections share the same vector dimension (from text-embedding-3-small) and payload schema (unified Chunk model).
_Avoid_: vector store, collection only
**Parent:**
In parent-child chunking, a larger semantic unit (Article-level) whose child chunks are finer-grained pieces. Only child chunks are embedded and stored in Qdrant; the parent is retrieved at query time for context expansion.
_Avoid_: source chunk, container
**Child:**
In parent-child chunking, a fine-grained piece (sentence or paragraph within an Article) that gets embedded and stored in Qdrant. Carries a parent_id reference to its Article parent.
_Avoid_: sub-chunk, fragment
**Experiment:**
A completed benchmark run. Every benchmark (single question or dataset) stores an experiment record in SQLite with: all questions, generated answers, retrieved chunks per strategy, evaluation metrics, configuration snapshot, and timestamps.
_Avoid_: run, trial, test
**SQLite:**
The persistence layer for structured data. Stores: documents (parsed text + document tree JSON), experiments (benchmark results), queries (single question/answer history), per-strategy metrics, and timestamps. Does NOT store vectors — those live in Qdrant.
_Avoid_: database, DB, local storage
**Qdrant:**
The vector database. Stores chunk embeddings and their payload metadata. One collection per strategy. Used for vector similarity search during retrieval. No hybrid search, no BM25, no filters — pure vector similarity only.
_Avoid_: vector DB, vector store
## REST API
**Resource:**
A noun-based URL entity in the REST API. The four primary resources are Documents, Queries, Benchmarks, and the read-only Strategies list.
_Avoid_: endpoint, API, route
**Endpoint:**
A specific URL path combined with an HTTP method. The complete endpoint map:
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | /documents | Upload .docx file (multipart) |
| POST | /documents/{id}/process | Run selected strategies (synchronous) |
| DELETE | /documents/{id} | Remove document + all its chunks |
| GET | /strategies | List available strategies |
| POST | /queries | Ask a question against a strategy |
| GET | /queries/{id} | Retrieve past query |
| POST | /benchmarks | Run benchmark comparison |
| POST | /benchmarks?dry_run=true | Estimate cost without executing |
| GET | /benchmarks/{id} | Retrieve benchmark results |
| GET | /benchmarks/{id}/report | Download HTML report |
_Avoid_: API, route, handler (use "endpoint" for URL+method, "handler" only for the Python function)
**Query Parameter:**
A key-value pair appended to the URL after `?` (e.g., ?dry_run=true). Used for optional modifiers that don't change the resource identity. Contrast with path parameters that identify the resource (e.g., /benchmarks/{id}).
_Avoid_: URL parameter, request parameter
**Path Parameter:**
A variable segment in the URL path identified by curly braces (e.g., /documents/{id}). Identifies a specific resource instance. Always an ID string.
_Avoid_: URL variable, route parameter
**Multipart Upload:**
The format used for uploading .docx files via POST /documents. The request Content-Type is multipart/form-data with the file attached as a binary field.
_Avoid_: file upload, binary upload
**Request ID:**
A unique identifier attached to every inbound request, propagated through logs and stored in experiment records. Enables tracing a single operation across multiple service calls (document parsing, embedding, LLM calls).
_Avoid_: trace ID, correlation ID, request UUID
## Architecture
**Core Layer:**
The shared foundational directory (core/) holding: config.py (pydantic-settings from .env), dependencies.py (OpenAI/Qdrant client singletons), exceptions.py (custom exception classes + FastAPI handlers), models.py (Chunk, StrategyName, pagination). All other domains import upward to core, never sideways to each other.
_Avoid_: common, shared, base
**Domain Directory:**
A self-contained vertical slice of the application containing its own routes, service logic, models, and templates. The four domains: documents/, chunking/, benchmarking/, storage/.
_Avoid_: module, feature folder
**Service Layer:**
The business logic layer within each domain directory. Services contain orchestration logic, call external APIs (OpenAI, Qdrant), and are imported by route handlers. A service never imports from another domain's service — all shared logic goes through core/.
_Avoid_: manager, logic, use case
**Router:**
A FastAPI APIRouter instance, one per domain, mounted onto the main FastAPI app in main.py. Each domain's routes.py exports its own router. The main app composes them: app.include_router(documents_router), etc.
_Avoid_: blueprint, handler group
**Strategy Pattern:**
The design pattern used for chunking strategies. All strategies inherit from a base ChunkingStrategy class with a chunk(document) method returning a list of Chunk objects. New strategies are added by implementing the interface, not by modifying existing code.
_Avoid_: polymorphism, duck typing
**Pipeline:**
The sequential processing chain from DOCX to Qdrant. Each strategy defines its own pipeline. Pipelines are synchronous and run within a single request lifecycle (POST /documents/{id}/process).
_Avoid_: workflow, chain, flow
**Retrieval Pipeline:**
The query-time chain: user question > vector search in Qdrant > retrieve top_k chunks > send chunks + question to LLM > return answer with metadata (retrieved chunks, latency, token usage).
_Avoid_: search flow, query pipeline
**Sync Processing:**
All API endpoints execute synchronously and return results directly. No background job queue, no polling endpoint, no async task tracking. Document processing takes 30-60 seconds; benchmarking takes 2-5 minutes. Both are acceptable within the request lifecycle.
_Avoid_: async, background job, task queue
**Import Rules:**
Strict unidirectional dependency: core/ imports nothing from the project. documents/, chunking/, benchmarking/, storage/ each import only from core/. No domain imports from another domain. If benchmarking needs to reference a query, it calls a function via the Query service, not by importing chunking/ directly.
_Avoid_: dependency direction, import graph
**Shared DocumentTree:**
The parsed document hierarchy extracted once at upload time by documents/parser.py (python-docx). Stored as JSON in SQLite. All strategies that need hierarchy (contextual, parent_child, markdown) read from this stored tree. Structure extraction happens once regardless of how many strategies are requested.
_Avoid_: parsed structure, document hierarchy
**App Factory:**
The main.py module that creates and configures the FastAPI application. Responsibilities: create FastAPI instance, mount all domain routers, register exception handlers from core/, configure middleware (CORS, request ID injection).
_Avoid_: app init, application setup
## Evaluation
**Query:**
A single question asked against a specific chunking strategy. POST /queries with {question, strategy, top_k}. Returns: generated answer, retrieved chunks (with metadata), latency breakdown, token usage. Stored in SQLite for history.
_Avoid_: request, question, ask
**Benchmark:**
A comparison run that evaluates one or more strategies against one or more questions. Runs synchronously. Two modes: single question (POST /benchmarks) or dataset (POST /benchmarks with questions.json body). Produces an experiment record in SQLite with per-strategy metrics.
_Avoid_: comparison, evaluation run, test
**Dataset Benchmark:**
A benchmark mode that processes multiple questions from a JSON file. Input: array of {question, answer} objects. Runs each question against all selected strategies, aggregates metrics, produces a single experiment with overall scores.
_Avoid_: batch benchmark, multi-question benchmark
**Dry Run:**
A benchmark mode that estimates the number of LLM calls and approximate cost without executing. Triggered via query parameter: POST /benchmarks?dry_run=true. Returns: estimated_calls, estimated_cost_usd. Used for cost visibility before committing to a run.
_Avoid_: preview, simulation
**LLM-as-Judge:**
The evaluation paradigm where gpt-4o-mini scores the quality of answers and retrieval. Same model used for generation and evaluation. Scores are on a 1-10 scale. Applied to: context relevance, answer similarity, faithfulness. Hallucination is binary.
_Avoid_: automated evaluation, model evaluation
**Experiment Metrics:**
Composite scores for a benchmark run stored in SQLite. Includes: context_relevance (1-10), answer_similarity (1-10), faithfulness (1-10), hallucination (bool), plus cost breakdowns per strategy.
_Avoid_: results, evaluation data
**Latency Breakdown:**
Timing measurements recorded for each operation: chunking_time (strategy execution), embedding_time (API calls to text-embedding-3-small), retrieval_time (Qdrant vector search), llm_time (OpenAI generation + evaluation calls). All measured in seconds. Total latency is the sum.
_Avoid_: timing data, performance metrics
**Context Relevance:**
LLM-judged score (1-10) measuring how relevant the retrieved chunks are to the question. Evaluated by asking gpt-4o-mini: "Given this question, how relevant are these chunks?"
_Avoid_: retrieval relevance, chunk relevance
**Answer Similarity:**
LLM-judged score (1-10) measuring how close the generated answer is to the expected answer. Evaluated by asking gpt-4o-mini: "Compare these two answers for semantic similarity."
_Avoid_: answer match, correctness
**Faithfulness:**
LLM-judged score (1-10) measuring whether the generated answer is supported by the retrieved chunks. Low faithfulness = the answer uses information outside the retrieved context. Evaluated by asking gpt-4o-mini: "Is this answer supported by these chunks?"
_Avoid_: groundedness, support
**Hallucination:**
Binary flag indicating the generated answer contains claims not present in the retrieved chunks. Set to true when the LLM-as-judge detects unsupported information in the answer.
_Avoid_: fabrication, confabulation
**Cost Estimation:**
The dry-run calculation that estimates: number of LLM calls needed (per strategy per question for generation + evaluation), approximate tokens per call, estimated cost at current pricing. Does not execute — returns estimate only.
_Avoid_: price estimate, budget calculation
**Token Tracking:**
Token usage recorded at two levels: (1) tiktoken for offline chunk metadata (token_count per chunk, stored at chunking time), (2) OpenAI API response usage field for actual billing data on generation and evaluation calls.
_Avoid_: token counting, usage tracking
## Tech Stack
**Framework:**
FastAPI — the web framework. Chosen for: auto-generated OpenAPI docs (Swagger), Pydantic validation, async support (available for future use), and clean dependency injection via Depends().
_Avoid_: web framework, HTTP framework
**python-docx:**
The library used for DOCX parsing and document structure extraction. Provides direct access to paragraph styles, heading levels, and document hierarchy. Chosen over pandoc/mammoth because it preserves native document structure needed by contextual, parent-child, and markdown strategies.
_Avoid_: docx parser, word parser
**OpenAI API:**
The external API for: (1) text-embedding-3-small embeddings, (2) gpt-4o-mini for enrichment, generation, and evaluation. Accessed via the openai Python SDK. All calls are synchronous.
_Avoid_: LLM API, embedding API
**Qdrant SDK:**
The Python client for Qdrant vector database. Used for: collection creation, vector upsert, similarity search. Accessed via the qdrant-client package.
_Avoid_: vector client, qdrant client
**tiktoken:**
OpenAI's tokenizer library used for offline token counting. Applied to chunks at chunking time to compute token_count metadata. Not used for API cost calculation (that uses the OpenAI response usage field).
_Avoid_: tokenizer, token counter
**Jinja2:**
The templating engine used for HTML benchmark reports. Template lives in benchmarking/templates/report.html. Renders per-strategy comparison tables, per-question breakdowns, and metric visualizations.
_Avoid_: HTML generator, template engine
**pydantic-settings:**
The library used for configuration management. Settings are loaded from .env with type validation, defaults, and environment variable overrides.
_Avoid_: config loader, settings manager
## Configuration
**Environment Variables (.env):**
| Variable | Default | Description |
|----------|---------|-------------|
| OPENAI_API_KEY | (required) | OpenAI API authentication |
| QDRANT_URL | http://localhost:6333 | Qdrant server URL |
| QDRANT_API_KEY | (optional) | Qdrant auth for cloud instances |
| EMBEDDING_MODEL | text-embedding-3-small | Fixed embedding model |
| LLM_MODEL | gpt-4o-mini | Fixed LLM for generation + evaluation |
| TOP_K | 5 | Default chunks to retrieve |
| TEMPERATURE | 0.0 | LLM temperature for deterministic output |
| MAX_TOKENS | 1024 | Max tokens in LLM response |
| CHUNK_SIZE | 512 | Target chunk size in characters |
| CHUNK_OVERLAP | 50 | Overlap between adjacent chunks |
| SEMANTIC_THRESHOLD | 0.3 | Cosine similarity cutoff for semantic chunking |
| SEMANTIC_MIN_CHUNK_SIZE | 3 | Minimum sentences per semantic chunk |
_Avoid_: config variables, env vars

53
docs/cant-do-yet.md Normal file
View File

@@ -0,0 +1,53 @@
# What You Can't Do Yet (But Backend Is Ready)
Features where the backend API is implemented and tested, but the dashboard UI does not yet expose them.
## 1. Chunk Preview Panel (Admin Tab)
**Backend**: `GET /admin/chunks/{doc_id}?strategy=...` returns all chunks for a document and strategy — chunk text (truncated to 500 chars), chunk index, token count, character count, parent_id.
**Missing UI**: No panel in the Admin tab to select a document + strategy and visualize the actual chunk text blocks, their sizes, and hierarchy.
## 2. Questions Dataset Management (Admin Tab)
**Backend**:
- `GET /admin/questions` — lists JSON files in `files/` with question counts
- `POST /admin/questions/upload` — uploads a new questions JSON file
- `GET /admin/questions/{file_id}` — returns full question set
- `DELETE /admin/questions/{file_id}` — deletes a question file
**Missing UI**: No section in the Admin tab to browse, upload, view, or delete question datasets. You can only use the default `files/questions.json` by typing its path in the Benchmarks tab.
## 3. Standalone Cost Estimator (Admin Tab)
**Backend**: `POST /admin/cost-estimate` accepts `num_questions` and `num_strategies`, returns full cost breakdown (embedding, queries, evaluation, token estimates, total USD).
**Missing UI**: No standalone cost estimation panel in the Admin tab. The Benchmarks tab has a "Cost Estimate" button, but it is tied to the benchmark form and requires a document selection.
## 4. Experiment Detail View (Benchmarks Tab)
**Backend**: `GET /benchmarks/{experiment_id}` returns full experiment data — per-question results, per-strategy scores, aggregate metrics, benchmark config.
**Missing UI**: Dashboard shows a list of experiments with a "Report" link that opens the raw HTML report in a new tab. No in-dashboard detail view with side-by-side strategy comparison tables, per-question drill-down, or visual charts.
## 5. Query History (Query Tab)
**Backend**: `GET /queries/{query_id}` retrieves any past query. All queries are stored in SQLite with full results (answer, retrieved chunks, latency, tokens).
**Missing UI**: No list of past queries. When you navigate away from the Query tab, the current result disappears from the screen. There is no history panel to browse, search, or re-view previous questions and answers.
## 6. Document Content Preview (Documents Tab)
**Backend**: `GET /documents` returns `parsed_text_preview` and `document_tree` fields in the detail response model (`DocumentDetailResponse`). SQLite stores the full parsed text and document tree.
**Missing UI**: You can upload and process documents, but you cannot preview the actual document text or the hierarchical document tree (sections, articles, paragraphs) inside the dashboard.
## 7. Real-Time Processing Status (Documents Tab)
**Backend**: The processing endpoint (`POST /documents/{id}/process`) returns per-strategy results — `strategies_completed` and `strategies_failed` with status, chunk counts, and error messages.
**Missing UI**: When you click "Process", the button shows a spinner. You don't see which strategies are running, which have completed, or which failed. If it fails partway through 5 strategies, you don't know which one broke without checking the server logs.
## 8. Global Error Handling and Retry
**Missing**: If the server returns an error (network down, Qdrant disconnected, OpenAI rate limit), you see a toast notification that disappears after 3 seconds. There is no retry button, no persistent error banner, and no automatic reconnection. A failed operation must be manually re-attempted by clicking the button again.

46
docs/out-of-scope-v1.md Normal file
View File

@@ -0,0 +1,46 @@
# What's Completely Out of Scope for V1
Features that are intentionally excluded from the first version of the dashboard. These would require significant additional architecture, infrastructure, or design work that doesn't serve the core use case of a single-user localhost benchmarking tool.
## User Authentication and Multi-User Support
No login, no sessions, no role-based access control. The dashboard runs on localhost and is operated by a single user. Adding auth would introduce JWT/session management, user storage, and permission logic — none of which serve the current workflow.
## Exporting Results to CSV/PDF
No built-in export of experiment results, query logs, or benchmark comparisons to CSV, PDF, or Excel. The HTML report endpoint (`/benchmarks/{id}/report`) is the closest thing to an export. If you need raw data, use the API endpoints directly or query SQLite.
## Custom Chunking Parameters from the UI
You cannot change `chunk_size`, `chunk_overlap`, `semantic_threshold`, or `semantic_min_chunk_size` from the dashboard. These are hardcoded in `src/core/config.py` with defaults (512 tokens, 50 overlap, 0.3 threshold). Changing them requires editing the config and restarting the server. The UI uses whatever the server has configured.
## Streaming Responses
Answers from GPT-4o-mini appear all at once after the full response is generated. There is no Server-Sent Events (SSE) or WebSocket connection to stream the answer word-by-word as it is generated. The latency breakdown shows how long generation took, but you wait for the complete answer.
## Dark/Light Theme Toggle
The dashboard is locked to the custom dark theme (#111113 base, #eab308 amber accent). There is no theme switcher or light mode alternative. The design tokens are hardcoded as CSS custom properties in the HTML file.
## Mobile Responsive Layout
The dashboard is designed for desktop screens (max-width 1200px, tab bar, tables). It is not optimized for phones or tablets. Tables will overflow, the tab bar may wrap, and the drop zone will be small on mobile. A responsive redesign with breakpoints is not planned for V1.
## WebSocket for Real-Time Updates
No persistent WebSocket connections. All communication is via standard HTTP REST requests. This means:
- Processing status updates require polling or a full page refresh
- No live push when a benchmark completes
- No live updating of collection point counts as chunks are inserted
## Batch Operations
You cannot process multiple documents at once, run multiple benchmarks in parallel, or delete several documents simultaneously. Each operation targets a single document. If you need to process 10 documents, you click "Process" 10 times.
## Search Across Documents or Experiments
No full-text search or filtering within the dashboard. You cannot search for a specific question text, filter experiments by date range, or find documents by partial filename. The document list and experiment list show everything in chronological order with no search or filter controls.
## Configuration Management from the UI
You cannot view or edit the server configuration (OpenAI model, temperature, max_tokens, embedding model, Qdrant URL, database path) from the dashboard. The `/admin/config` endpoint was deferred to V1.1. All configuration is managed via the `.env` file and `src/core/config.py`.

62
docs/tasks.md
View File

@@ -18,39 +18,67 @@ Status legend: `DONE` `IN_PROGRESS` `TODO`
| # | Task | Status | Step | | # | Task | Status | Step |
|---|------|--------|------| |---|------|--------|------|
| 5 | Create chunking strategy interface and abstraction layer | TODO | 5 | | 5 | Create chunking strategy interface and abstraction layer | DONE | 5 |
| 6 | Implement Structure-Aware Markdown chunking strategy | TODO | 6 | | 6 | Implement Structure-Aware Markdown chunking strategy | DONE | 6 |
| 7 | Implement Recursive chunking strategy | TODO | 6 | | 7 | Implement Recursive chunking strategy | DONE | 6 |
| 8 | Implement Semantic chunking strategy | TODO | 6 | | 8 | Implement Semantic chunking strategy | DONE | 6 |
| 9 | Implement Parent-Child chunking strategy | TODO | 6 | | 9 | Implement Parent-Child chunking strategy | DONE | 6 |
| 10 | Implement Contextual Structure-Aware chunking strategy | TODO | 6 | | 10 | Implement Contextual Structure-Aware chunking strategy | DONE | 6 |
| 11 | Implement OpenAI embedding service using text-embedding-3-small | TODO | 7 | | 11 | Implement OpenAI embedding service using text-embedding-3-small | DONE | 7 |
| 12 | Implement strategy-based document processing API | TODO | 8 | | 12 | Implement strategy-based document processing API | DONE | 8 |
## Phase 3 — Query Pipeline ## Phase 3 — Query Pipeline
| # | Task | Status | Step | | # | Task | Status | Step |
|---|------|--------|------| |---|------|--------|------|
| 13 | Implement question answering API with configurable chunking strategy selection | TODO | 9–10 | | 13 | Implement question answering API with configurable chunking strategy selection | DONE | 9–10 |
## Phase 4 — Benchmarking + Evaluation ## Phase 4 — Benchmarking + Evaluation
| # | Task | Status | Step | | # | Task | Status | Step |
|---|------|--------|------| |---|------|--------|------|
| 14 | Implement benchmarking pipeline for comparing chunking strategies | TODO | 12 | | 14 | Implement benchmarking pipeline for comparing chunking strategies | DONE | 12 |
| 15 | Implement RAG evaluation pipeline using GPT-4o-mini | TODO | 11 | | 15 | Implement RAG evaluation pipeline using GPT-4o-mini | DONE | 11 |
| 16 | Implement experiment tracking and result storage system | TODO | 12 | | 16 | Implement experiment tracking and result storage system | DONE | 12 |
| 17 | Create question-answer evaluation dataset from insurance regulation document | TODO | 12 | | 17 | Create question-answer evaluation dataset from insurance regulation document | DONE | 12 |
| 18 | Implement HTML benchmark report generation system | TODO | 14 | | 18 | Implement HTML benchmark report generation system | DONE | 14 |
| 19 | Design HTML report structure for experiment comparison and visualization | TODO | 14 | | 19 | Design HTML report structure for experiment comparison and visualization | DONE | 14 |
| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 | | 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 |
## Phase 5 — Wiring + Verification ## Phase 5 — Wiring + Verification
| # | Task | Status | Step | | # | Task | Status | Step |
|---|------|--------|------| |---|------|--------|------|
| 21 | Add logging, request tracking, and cost monitoring | TODO | 15 | | 21 | Add logging, request tracking, and cost monitoring | DONE | 15 |
| 22 | Create API documentation and Swagger examples | TODO | 15 | | 22 | Create API documentation and Swagger examples | DONE | 15 |
| 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 | | 23 | Create automated tests for chunking strategies and RAG workflow | TODO | 16 |
| 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 | | 24 | Validate end-to-end benchmarking workflow with insurance regulation dataset | TODO | 16 |
| 25 | Analyze and compare chunking strategy performance results | TODO | 16 | | 25 | Analyze and compare chunking strategy performance results | TODO | 16 |
## Phase 6 — Admin Dashboard (UI)
| # | Task | Status | Note |
|---|------|--------|------|
| 26 | Design dashboard architecture (ADR-0001 through ADR-0009) | DONE | Babel JSX, persistent tabs, dark theme, /app mount |
| 27 | Create frontend skeleton: index.html with tab bar, dark theme, design tokens | DONE | src/static/index.html, 889 lines |
| 28 | Wire FastAPI to serve dashboard at /app (StaticFiles mount) | DONE | src/main.py modified |
| 29 | Implement admin backend: /admin/health endpoint | DONE | src/admin/routes.py + service.py |
| 30 | Implement admin backend: /admin/qdrant/* CRUD endpoints | DONE | Collections list, create, delete, wipe points |
| 31 | Implement admin backend: /admin/chunks/{doc_id} chunk preview | DONE | Backend only, no frontend panel |
| 32 | Implement admin backend: /admin/questions/* CRUD endpoints | DONE | Backend only, no frontend panel |
| 33 | Implement admin backend: /admin/cost-estimate endpoint | DONE | Backend only, no frontend panel |
| 34 | Build Documents tab: upload, list, process, delete | DONE | Drag-drop, strategy checkboxes, all wired |
| 35 | Build Home tab: status cards + quick start guide | DONE | Documents count, experiments count, server status |
| 36 | Build Query tab: form + answer display + chunk table | DONE | Document/strategy/top_k selectors, question textarea |
| 37 | Build Benchmarks tab: run form + experiments list | DONE | Strategies checkboxes, cost estimate, report links |
| 38 | Build Admin tab: health cards + Qdrant collections panel | DONE | Health status, collection list with delete |
| 39 | Build Admin tab: chunk preview panel | TODO | Backend ready, UI not built |
| 40 | Build Admin tab: questions dataset management panel | TODO | Backend ready, UI not built |
| 41 | Build Admin tab: standalone cost estimator panel | TODO | Backend ready, UI not built |
| 42 | Build Query tab: past queries history panel | TODO | Backend storage exists, no UI list |
| 43 | Build Benchmarks tab: in-dashboard experiment detail view | TODO | Report link exists, no drill-down UI |
| 44 | Build Documents tab: document content preview | TODO | Parsed text stored, no preview panel |
| 45 | Add real-time processing progress indicator | TODO | Spinner only, no per-strategy progress |
| 46 | Add global error handling and retry logic | TODO | Toast only, no retry button or banner |
| 47 | Create docs: cant-do-yet.md (backend ready, no UI) | DONE | docs/cant-do-yet.md |
| 48 | Create docs: out-of-scope-v1.md (intentionally excluded) | DONE | docs/out-of-scope-v1.md |