diff --git a/CONTEXT.md b/CONTEXT.md index 47f6b68..b2db2e9 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,305 +1,42 @@ -# RAG Chunking Benchmarker - -A benchmarking application that compares chunking strategies on regulatory documents, measuring retrieval quality, answer faithfulness, and cost across strategies. - -## Domain - -**Chunk:** -A discrete unit of text extracted from a document, ready for embedding and storage. Every strategy produces chunks conforming to the unified Chunk model. -_Avoid_: fragment, segment, block - -**Unified Chunk Model:** -A single Pydantic model that all five chunking strategies must return. Strategy-specific fields (e.g., parent_id) are nullable when not applicable. This ensures Qdrant payload schemas are consistent and benchmarking compares equivalent structures. -_Avoid_: chunk schema, chunk type - -**Chunk Metadata:** -Fields stored with every chunk in Qdrant: document_name, chunk_id, strategy_name, chunk_index, token_count, character_count. No page_number (unreliable in DOCX). chunk_index is the position in the full document's chunk sequence. -_Avoid_: chunk fields, payload data - -**Chunking Strategy:** -An algorithm that determines how a document is split into chunks. Must inherit from the ChunkingStrategy base class and implement the chunk() method. The five strategies are: contextual_structure, parent_child, semantic, markdown_structure, recursive. -_Avoid_: splitter, chunker, method - -**Strategy Name:** -The canonical identifier for each chunking strategy: contextual_structure, parent_child, semantic, markdown_structure, recursive. Used in collection names, metadata, API parameters, and benchmark comparisons. -_Avoid_: strategy type, strategy id - -**Contextual Structure Strategy:** -Pipeline: DOCX > Markdown > Structure Extraction > LLM enrichment (GPT-4o-mini per section) > Chunks > Embeddings > Qdrant. Each chunk is enriched with Document/Section/Article/Context/Content labels. The full enriched text is what gets embedded (not just the content). Hard-fails if the LLM enrichment call fails — no degraded chunks stored. -_Avoid_: contextual chunking, enriched strategy - -**Parent-Child Strategy:** -Parent chunks are Article-level units (hardcoded for v1). Child chunks are sentences or paragraphs within each Article. Only child vectors are stored in Qdrant with a parent_id reference. At retrieval time, the matching child is found via vector search, then its full parent Article is returned as context to the LLM. -_Avoid_: hierarchical chunking, nested strategy - -**Parent Level:** -The document hierarchy level used as parent in parent-child chunking. Hardcoded to "article" for v1. Configurable in future versions (article, section, paragraph) when additional document types are supported. -_Avoid_: parent hierarchy, parent depth - -**Semantic Strategy:** -Pipeline: DOCX > Markdown > Sentence extraction > Sentence embeddings > Cosine similarity between adjacent sentences > Dynamic chunk creation. When similarity drops below the semantic threshold, a new chunk boundary is created. Enforces a minimum chunk size (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences). Boundary sentence stays with the previous chunk. -_Avoid_: embedding-based chunking, similarity chunking - -**Semantic Threshold:** -The cosine similarity value below which adjacent sentences are considered a topic shift. When cosine_sim(sentence_n, sentence_n+1) < threshold, a new chunk boundary is created. Configurable via SEMANTIC_THRESHOLD in .env. -_Avoid_: similarity cutoff, cut threshold - -**Minimum Chunk Size:** -The minimum number of sentences a semantic chunk must contain. If a boundary is detected before the minimum is reached, accumulation continues. Configurable via SEMANTIC_MIN_CHUNK_SIZE in .env. -_Avoid_: min chunk, smallest chunk - -**Semantic Boundary:** -A point in the document where adjacent sentence similarity drops below the semantic threshold, marking where one chunk ends and another begins. -_Avoid_: cut point, split boundary - -**Markdown Structure Strategy:** -Pipeline: DOCX > Markdown > Header detection > Paragraph grouping > Leaf-level chunk creation > Qdrant. Chunks are created only at the deepest header level (leaf chunks). No overlap between parent and child sections — a ## section is NOT a chunk if it contains ### subsections. -_Avoid_: header-based chunking, markdown chunking - -**Leaf Chunk:** -A chunk at the deepest header level in markdown chunking. Only leaf-level chunks are created (no overlap with parent sections). -_Avoid_: atomic chunk, granular chunk - -**Recursive Strategy:** -Pipeline: DOCX > Markdown > Cascade splitting using a separator hierarchy. Separators applied in order: (1) Markdown headers (#, ##, ###), (2) double newline (\n\n), (3) single newline (\n), (4) sentence-ending punctuation (. ! ? followed by space), (5) space (word-level, last resort). Splitting stops when chunks reach the target size. Each chunk stores which separator level was used. -_Avoid_: hierarchical splitting, recursive splitting - -**Separator Cascade:** -The ordered list of separators used by recursive chunking: headers > double newline > single newline > sentence-ending punctuation > word boundary. Applied top-down; each level is tried before falling back to the next. -_Avoid_: separator list, split pattern - -**Document Tree:** -The hierarchical structure of a parsed document: Document > Section > Article > Paragraph. Extracted once via python-docx at upload time, stored in SQLite, and shared across all strategies. Strategies that need hierarchy (contextual, parent_child, markdown) read from the stored tree; they do not re-parse. -_Avoid_: doc tree, document structure, hierarchy - -**Embedding:** -The vector representation of a chunk's text, produced by text-embedding-3-small. For the contextual_structure strategy, the full enriched text is embedded. For all other strategies, the raw chunk text is embedded. -_Avoid_: vector, vector representation, encoding - -**Top-k:** -The number of chunks to retrieve during vector similarity search. Default is 5. Configurable per query via the API (POST /queries) and per benchmark via the dataset configuration. -_Avoid_: k, retrieve count, result limit - -**Embedding Model:** -The fixed OpenAI model used for all embedding operations: text-embedding-3-small. Not configurable — every strategy uses the same embedding model to ensure fair comparison. -_Avoid_: vector model, encoder model - -**LLM Model:** -The fixed OpenAI model used for all generation and evaluation: gpt-4o-mini. Used for: (1) contextual enrichment during chunking, (2) answer generation during queries, (3) LLM-as-judge evaluation during benchmarks. -_Avoid_: generation model, chat model - -## Storage - -**Strategy Collection:** -A Qdrant collection named after a strategy. Naming convention: {strategy_name}_collection (e.g., contextual_structure_collection, semantic_collection). Each strategy writes to its own collection. All collections share the same vector dimension (from text-embedding-3-small) and payload schema (unified Chunk model). -_Avoid_: vector store, collection only - -**Parent:** -In parent-child chunking, a larger semantic unit (Article-level) whose child chunks are finer-grained pieces. Only child chunks are embedded and stored in Qdrant; the parent is retrieved at query time for context expansion. -_Avoid_: source chunk, container - -**Child:** -In parent-child chunking, a fine-grained piece (sentence or paragraph within an Article) that gets embedded and stored in Qdrant. Carries a parent_id reference to its Article parent. -_Avoid_: sub-chunk, fragment - -**Experiment:** -A completed benchmark run. Every benchmark (single question or dataset) stores an experiment record in SQLite with: all questions, generated answers, retrieved chunks per strategy, evaluation metrics, configuration snapshot, and timestamps. -_Avoid_: run, trial, test - -**SQLite:** -The persistence layer for structured data. Stores: documents (parsed text + document tree JSON), experiments (benchmark results), queries (single question/answer history), per-strategy metrics, and timestamps. Does NOT store vectors — those live in Qdrant. -_Avoid_: database, DB, local storage - -**Qdrant:** -The vector database. Stores chunk embeddings and their payload metadata. One collection per strategy. Used for vector similarity search during retrieval. No hybrid search, no BM25, no filters — pure vector similarity only. -_Avoid_: vector DB, vector store - -## REST API - -**Resource:** -A noun-based URL entity in the REST API. The four primary resources are Documents, Queries, Benchmarks, and the read-only Strategies list. -_Avoid_: endpoint, API, route - -**Endpoint:** -A specific URL path combined with an HTTP method. The complete endpoint map: - -| Method | Endpoint | Description | -|--------|----------|-------------| -| POST | /documents | Upload .docx file (multipart) | -| POST | /documents/{id}/process | Run selected strategies (synchronous) | -| DELETE | /documents/{id} | Remove document + all its chunks | -| GET | /strategies | List available strategies | -| POST | /queries | Ask a question against a strategy | -| GET | /queries/{id} | Retrieve past query | -| POST | /benchmarks | Run benchmark comparison | -| POST | /benchmarks?dry_run=true | Estimate cost without executing | -| GET | /benchmarks/{id} | Retrieve benchmark results | -| GET | /benchmarks/{id}/report | Download HTML report | -_Avoid_: API, route, handler (use "endpoint" for URL+method, "handler" only for the Python function) - -**Query Parameter:** -A key-value pair appended to the URL after `?` (e.g., ?dry_run=true). Used for optional modifiers that don't change the resource identity. Contrast with path parameters that identify the resource (e.g., /benchmarks/{id}). -_Avoid_: URL parameter, request parameter - -**Path Parameter:** -A variable segment in the URL path identified by curly braces (e.g., /documents/{id}). Identifies a specific resource instance. Always an ID string. -_Avoid_: URL variable, route parameter - -**Multipart Upload:** -The format used for uploading .docx files via POST /documents. The request Content-Type is multipart/form-data with the file attached as a binary field. -_Avoid_: file upload, binary upload - -**Request ID:** -A unique identifier attached to every inbound request, propagated through logs and stored in experiment records. Enables tracing a single operation across multiple service calls (document parsing, embedding, LLM calls). -_Avoid_: trace ID, correlation ID, request UUID - -## Architecture - -**Core Layer:** -The shared foundational directory (core/) holding: config.py (pydantic-settings from .env), dependencies.py (OpenAI/Qdrant client singletons), exceptions.py (custom exception classes + FastAPI handlers), models.py (Chunk, StrategyName, pagination). All other domains import upward to core, never sideways to each other. -_Avoid_: common, shared, base - -**Domain Directory:** -A self-contained vertical slice of the application containing its own routes, service logic, models, and templates. The four domains: documents/, chunking/, benchmarking/, storage/. -_Avoid_: module, feature folder - -**Service Layer:** -The business logic layer within each domain directory. Services contain orchestration logic, call external APIs (OpenAI, Qdrant), and are imported by route handlers. A service never imports from another domain's service — all shared logic goes through core/. -_Avoid_: manager, logic, use case - -**Router:** -A FastAPI APIRouter instance, one per domain, mounted onto the main FastAPI app in main.py. Each domain's routes.py exports its own router. The main app composes them: app.include_router(documents_router), etc. -_Avoid_: blueprint, handler group - -**Strategy Pattern:** -The design pattern used for chunking strategies. All strategies inherit from a base ChunkingStrategy class with a chunk(document) method returning a list of Chunk objects. New strategies are added by implementing the interface, not by modifying existing code. -_Avoid_: polymorphism, duck typing - -**Pipeline:** -The sequential processing chain from DOCX to Qdrant. Each strategy defines its own pipeline. Pipelines are synchronous and run within a single request lifecycle (POST /documents/{id}/process). -_Avoid_: workflow, chain, flow - -**Retrieval Pipeline:** -The query-time chain: user question > vector search in Qdrant > retrieve top_k chunks > send chunks + question to LLM > return answer with metadata (retrieved chunks, latency, token usage). -_Avoid_: search flow, query pipeline - -**Sync Processing:** -All API endpoints execute synchronously and return results directly. No background job queue, no polling endpoint, no async task tracking. Document processing takes 30-60 seconds; benchmarking takes 2-5 minutes. Both are acceptable within the request lifecycle. -_Avoid_: async, background job, task queue - -**Import Rules:** -Strict unidirectional dependency: core/ imports nothing from the project. documents/, chunking/, benchmarking/, storage/ each import only from core/. No domain imports from another domain. If benchmarking needs to reference a query, it calls a function via the Query service, not by importing chunking/ directly. -_Avoid_: dependency direction, import graph - -**Shared DocumentTree:** -The parsed document hierarchy extracted once at upload time by documents/parser.py (python-docx). Stored as JSON in SQLite. All strategies that need hierarchy (contextual, parent_child, markdown) read from this stored tree. Structure extraction happens once regardless of how many strategies are requested. -_Avoid_: parsed structure, document hierarchy - -**App Factory:** -The main.py module that creates and configures the FastAPI application. Responsibilities: create FastAPI instance, mount all domain routers, register exception handlers from core/, configure middleware (CORS, request ID injection). -_Avoid_: app init, application setup - -## Evaluation - -**Query:** -A single question asked against a specific chunking strategy. POST /queries with {question, strategy, top_k}. Returns: generated answer, retrieved chunks (with metadata), latency breakdown, token usage. Stored in SQLite for history. -_Avoid_: request, question, ask - -**Benchmark:** -A comparison run that evaluates one or more strategies against one or more questions. Runs synchronously. Two modes: single question (POST /benchmarks) or dataset (POST /benchmarks with questions.json body). Produces an experiment record in SQLite with per-strategy metrics. -_Avoid_: comparison, evaluation run, test - -**Dataset Benchmark:** -A benchmark mode that processes multiple questions from a JSON file. Input: array of {question, answer} objects. Runs each question against all selected strategies, aggregates metrics, produces a single experiment with overall scores. -_Avoid_: batch benchmark, multi-question benchmark - -**Dry Run:** -A benchmark mode that estimates the number of LLM calls and approximate cost without executing. Triggered via query parameter: POST /benchmarks?dry_run=true. Returns: estimated_calls, estimated_cost_usd. Used for cost visibility before committing to a run. -_Avoid_: preview, simulation - -**LLM-as-Judge:** -The evaluation paradigm where gpt-4o-mini scores the quality of answers and retrieval. Same model used for generation and evaluation. Scores are on a 1-10 scale. Applied to: context relevance, answer similarity, faithfulness. Hallucination is binary. -_Avoid_: automated evaluation, model evaluation - -**Experiment Metrics:** -Composite scores for a benchmark run stored in SQLite. Includes: context_relevance (1-10), answer_similarity (1-10), faithfulness (1-10), hallucination (bool), plus cost breakdowns per strategy. -_Avoid_: results, evaluation data - -**Latency Breakdown:** -Timing measurements recorded for each operation: chunking_time (strategy execution), embedding_time (API calls to text-embedding-3-small), retrieval_time (Qdrant vector search), llm_time (OpenAI generation + evaluation calls). All measured in seconds. Total latency is the sum. -_Avoid_: timing data, performance metrics - -**Context Relevance:** -LLM-judged score (1-10) measuring how relevant the retrieved chunks are to the question. Evaluated by asking gpt-4o-mini: "Given this question, how relevant are these chunks?" -_Avoid_: retrieval relevance, chunk relevance - -**Answer Similarity:** -LLM-judged score (1-10) measuring how close the generated answer is to the expected answer. Evaluated by asking gpt-4o-mini: "Compare these two answers for semantic similarity." -_Avoid_: answer match, correctness - -**Faithfulness:** -LLM-judged score (1-10) measuring whether the generated answer is supported by the retrieved chunks. Low faithfulness = the answer uses information outside the retrieved context. Evaluated by asking gpt-4o-mini: "Is this answer supported by these chunks?" -_Avoid_: groundedness, support - -**Hallucination:** -Binary flag indicating the generated answer contains claims not present in the retrieved chunks. Set to true when the LLM-as-judge detects unsupported information in the answer. -_Avoid_: fabrication, confabulation - -**Cost Estimation:** -The dry-run calculation that estimates: number of LLM calls needed (per strategy per question for generation + evaluation), approximate tokens per call, estimated cost at current pricing. Does not execute — returns estimate only. -_Avoid_: price estimate, budget calculation - -**Token Tracking:** -Token usage recorded at two levels: (1) tiktoken for offline chunk metadata (token_count per chunk, stored at chunking time), (2) OpenAI API response usage field for actual billing data on generation and evaluation calls. -_Avoid_: token counting, usage tracking - -## Tech Stack - -**Framework:** -FastAPI — the web framework. Chosen for: auto-generated OpenAPI docs (Swagger), Pydantic validation, async support (available for future use), and clean dependency injection via Depends(). -_Avoid_: web framework, HTTP framework - -**python-docx:** -The library used for DOCX parsing and document structure extraction. Provides direct access to paragraph styles, heading levels, and document hierarchy. Chosen over pandoc/mammoth because it preserves native document structure needed by contextual, parent-child, and markdown strategies. -_Avoid_: docx parser, word parser - -**OpenAI API:** -The external API for: (1) text-embedding-3-small embeddings, (2) gpt-4o-mini for enrichment, generation, and evaluation. Accessed via the openai Python SDK. All calls are synchronous. -_Avoid_: LLM API, embedding API - -**Qdrant SDK:** -The Python client for Qdrant vector database. Used for: collection creation, vector upsert, similarity search. Accessed via the qdrant-client package. -_Avoid_: vector client, qdrant client - -**tiktoken:** -OpenAI's tokenizer library used for offline token counting. Applied to chunks at chunking time to compute token_count metadata. Not used for API cost calculation (that uses the OpenAI response usage field). -_Avoid_: tokenizer, token counter - -**Jinja2:** -The templating engine used for HTML benchmark reports. Template lives in benchmarking/templates/report.html. Renders per-strategy comparison tables, per-question breakdowns, and metric visualizations. -_Avoid_: HTML generator, template engine - -**pydantic-settings:** -The library used for configuration management. Settings are loaded from .env with type validation, defaults, and environment variable overrides. -_Avoid_: config loader, settings manager - -## Configuration - -**Environment Variables (.env):** - -| Variable | Default | Description | -|----------|---------|-------------| -| OPENAI_API_KEY | (required) | OpenAI API authentication | -| QDRANT_URL | http://localhost:6333 | Qdrant server URL | -| QDRANT_API_KEY | (optional) | Qdrant auth for cloud instances | -| EMBEDDING_MODEL | text-embedding-3-small | Fixed embedding model | -| LLM_MODEL | gpt-4o-mini | Fixed LLM for generation + evaluation | -| TOP_K | 5 | Default chunks to retrieve | -| TEMPERATURE | 0.0 | LLM temperature for deterministic output | -| MAX_TOKENS | 1024 | Max tokens in LLM response | -| CHUNK_SIZE | 512 | Target chunk size in characters | -| CHUNK_OVERLAP | 50 | Overlap between adjacent chunks | -| SEMANTIC_THRESHOLD | 0.3 | Cosine similarity cutoff for semantic chunking | -| SEMANTIC_MIN_CHUNK_SIZE | 3 | Minimum sentences per semantic chunk | - -_Avoid_: config variables, env vars +# Context — RAG Chunking Benchmarker Admin Dashboard + +## Language + +**Dashboard**: +A single-file React app (CDN-loaded, no build step) served by FastAPI at `/app`. Replaces Swagger as the primary UI for operating the RAG benchmarking platform. +_Avoid_: Admin panel, web UI, frontend + +**Strategy**: +One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child. +_Avoid_: Method, approach, technique + +**Experiment**: +A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics. +_Avoid_: Run, trial, benchmark + +**Chunk Preview**: +The ability to visualize what a specific strategy produces for a given document — the raw text blocks, their sizes, and hierarchy. +_Avoid_: Chunk inspection, chunk view + +**Tab**: +A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives. +_Avoid_: Page, route, view + +## Architecture Decisions + +| # | Decision | Status | +|---|----------|--------| + ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved | + ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved | + ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved | + ADR-0004 | Babel-in-browser JSX: single index.html with inline