chunking_strategies_evaluation/CONTEXT.md

# RAG Chunking Benchmarker

A benchmarking application that compares chunking strategies on regulatory documents, measuring retrieval quality, answer faithfulness, and cost across strategies.

## Domain

**Chunk:**
A discrete unit of text extracted from a document, ready for embedding and storage. Every strategy produces chunks conforming to the unified Chunk model.
_Avoid_: fragment, segment, block

**Unified Chunk Model:**
A single Pydantic model that all five chunking strategies must return. Strategy-specific fields (e.g., parent_id) are nullable when not applicable. This ensures Qdrant payload schemas are consistent and benchmarking compares equivalent structures.
_Avoid_: chunk schema, chunk type

**Chunk Metadata:**
Fields stored with every chunk in Qdrant: document_name, chunk_id, strategy_name, chunk_index, token_count, character_count. No page_number (unreliable in DOCX). chunk_index is the position in the full document's chunk sequence.
_Avoid_: chunk fields, payload data

**Chunking Strategy:**
An algorithm that determines how a document is split into chunks. Must inherit from the ChunkingStrategy base class and implement the chunk() method. The five strategies are: contextual_structure, parent_child, semantic, markdown_structure, recursive.
_Avoid_: splitter, chunker, method

**Strategy Name:**
The canonical identifier for each chunking strategy: contextual_structure, parent_child, semantic, markdown_structure, recursive. Used in collection names, metadata, API parameters, and benchmark comparisons.
_Avoid_: strategy type, strategy id

**Contextual Structure Strategy:**
Pipeline: DOCX > Markdown > Structure Extraction > LLM enrichment (GPT-4o-mini per section) > Chunks > Embeddings > Qdrant. Each chunk is enriched with Document/Section/Article/Context/Content labels. The full enriched text is what gets embedded (not just the content). Hard-fails if the LLM enrichment call fails — no degraded chunks stored.
_Avoid_: contextual chunking, enriched strategy

**Parent-Child Strategy:**
Parent chunks are Article-level units (hardcoded for v1). Child chunks are sentences or paragraphs within each Article. Only child vectors are stored in Qdrant with a parent_id reference. At retrieval time, the matching child is found via vector search, then its full parent Article is returned as context to the LLM.
_Avoid_: hierarchical chunking, nested strategy

**Parent Level:**
The document hierarchy level used as parent in parent-child chunking. Hardcoded to "article" for v1. Configurable in future versions (article, section, paragraph) when additional document types are supported.
_Avoid_: parent hierarchy, parent depth

**Semantic Strategy:**
Pipeline: DOCX > Markdown > Sentence extraction > Sentence embeddings > Cosine similarity between adjacent sentences > Dynamic chunk creation. When similarity drops below the semantic threshold, a new chunk boundary is created. Enforces a minimum chunk size (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences). Boundary sentence stays with the previous chunk.
_Avoid_: embedding-based chunking, similarity chunking

**Semantic Threshold:**
The cosine similarity value below which adjacent sentences are considered a topic shift. When cosine_sim(sentence_n, sentence_n+1) < threshold, a new chunk boundary is created. Configurable via SEMANTIC_THRESHOLD in .env.
_Avoid_: similarity cutoff, cut threshold

**Minimum Chunk Size:**
The minimum number of sentences a semantic chunk must contain. If a boundary is detected before the minimum is reached, accumulation continues. Configurable via SEMANTIC_MIN_CHUNK_SIZE in .env.
_Avoid_: min chunk, smallest chunk

**Semantic Boundary:**
A point in the document where adjacent sentence similarity drops below the semantic threshold, marking where one chunk ends and another begins.
_Avoid_: cut point, split boundary

**Markdown Structure Strategy:**
Pipeline: DOCX > Markdown > Header detection > Paragraph grouping > Leaf-level chunk creation > Qdrant. Chunks are created only at the deepest header level (leaf chunks). No overlap between parent and child sections — a ## section is NOT a chunk if it contains ### subsections.
_Avoid_: header-based chunking, markdown chunking

**Leaf Chunk:**
A chunk at the deepest header level in markdown chunking. Only leaf-level chunks are created (no overlap with parent sections).
_Avoid_: atomic chunk, granular chunk

**Recursive Strategy:**
Pipeline: DOCX > Markdown > Cascade splitting using a separator hierarchy. Separators applied in order: (1) Markdown headers (#, ##, ###), (2) double newline (\n\n), (3) single newline (\n), (4) sentence-ending punctuation (. ! ? followed by space), (5) space (word-level, last resort). Splitting stops when chunks reach the target size. Each chunk stores which separator level was used.
_Avoid_: hierarchical splitting, recursive splitting

**Separator Cascade:**
The ordered list of separators used by recursive chunking: headers > double newline > single newline > sentence-ending punctuation > word boundary. Applied top-down; each level is tried before falling back to the next.
_Avoid_: separator list, split pattern

**Document Tree:**
The hierarchical structure of a parsed document: Document > Section > Article > Paragraph. Extracted once via python-docx at upload time, stored in SQLite, and shared across all strategies. Strategies that need hierarchy (contextual, parent_child, markdown) read from the stored tree; they do not re-parse.
_Avoid_: doc tree, document structure, hierarchy

**Embedding:**
The vector representation of a chunk's text, produced by text-embedding-3-small. For the contextual_structure strategy, the full enriched text is embedded. For all other strategies, the raw chunk text is embedded.
_Avoid_: vector, vector representation, encoding

**Top-k:**
The number of chunks to retrieve during vector similarity search. Default is 5. Configurable per query via the API (POST /queries) and per benchmark via the dataset configuration.
_Avoid_: k, retrieve count, result limit

**Embedding Model:**
The fixed OpenAI model used for all embedding operations: text-embedding-3-small. Not configurable — every strategy uses the same embedding model to ensure fair comparison.
_Avoid_: vector model, encoder model

**LLM Model:**
The fixed OpenAI model used for all generation and evaluation: gpt-4o-mini. Used for: (1) contextual enrichment during chunking, (2) answer generation during queries, (3) LLM-as-judge evaluation during benchmarks.
_Avoid_: generation model, chat model

## Storage

**Strategy Collection:**
A Qdrant collection named after a strategy. Naming convention: {strategy_name}_collection (e.g., contextual_structure_collection, semantic_collection). Each strategy writes to its own collection. All collections share the same vector dimension (from text-embedding-3-small) and payload schema (unified Chunk model).
_Avoid_: vector store, collection only

**Parent:**
In parent-child chunking, a larger semantic unit (Article-level) whose child chunks are finer-grained pieces. Only child chunks are embedded and stored in Qdrant; the parent is retrieved at query time for context expansion.
_Avoid_: source chunk, container

**Child:**
In parent-child chunking, a fine-grained piece (sentence or paragraph within an Article) that gets embedded and stored in Qdrant. Carries a parent_id reference to its Article parent.
_Avoid_: sub-chunk, fragment

**Experiment:**
A completed benchmark run. Every benchmark (single question or dataset) stores an experiment record in SQLite with: all questions, generated answers, retrieved chunks per strategy, evaluation metrics, configuration snapshot, and timestamps.
_Avoid_: run, trial, test

**SQLite:**
The persistence layer for structured data. Stores: documents (parsed text + document tree JSON), experiments (benchmark results), queries (single question/answer history), per-strategy metrics, and timestamps. Does NOT store vectors — those live in Qdrant.
_Avoid_: database, DB, local storage

**Qdrant:**
The vector database. Stores chunk embeddings and their payload metadata. One collection per strategy. Used for vector similarity search during retrieval. No hybrid search, no BM25, no filters — pure vector similarity only.
_Avoid_: vector DB, vector store

## REST API

**Resource:**
A noun-based URL entity in the REST API. The four primary resources are Documents, Queries, Benchmarks, and the read-only Strategies list.
_Avoid_: endpoint, API, route

**Endpoint:**
A specific URL path combined with an HTTP method. The complete endpoint map:

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | /documents | Upload .docx file (multipart) |
| POST | /documents/{id}/process | Run selected strategies (synchronous) |
| DELETE | /documents/{id} | Remove document + all its chunks |
| GET | /strategies | List available strategies |
| POST | /queries | Ask a question against a strategy |
| GET | /queries/{id} | Retrieve past query |
| POST | /benchmarks | Run benchmark comparison |
| POST | /benchmarks?dry_run=true | Estimate cost without executing |
| GET | /benchmarks/{id} | Retrieve benchmark results |
| GET | /benchmarks/{id}/report | Download HTML report |
_Avoid_: API, route, handler (use "endpoint" for URL+method, "handler" only for the Python function)

**Query Parameter:**
A key-value pair appended to the URL after `?` (e.g., ?dry_run=true). Used for optional modifiers that don't change the resource identity. Contrast with path parameters that identify the resource (e.g., /benchmarks/{id}).
_Avoid_: URL parameter, request parameter

**Path Parameter:**
A variable segment in the URL path identified by curly braces (e.g., /documents/{id}). Identifies a specific resource instance. Always an ID string.
_Avoid_: URL variable, route parameter

**Multipart Upload:**
The format used for uploading .docx files via POST /documents. The request Content-Type is multipart/form-data with the file attached as a binary field.
_Avoid_: file upload, binary upload

**Request ID:**
A unique identifier attached to every inbound request, propagated through logs and stored in experiment records. Enables tracing a single operation across multiple service calls (document parsing, embedding, LLM calls).
_Avoid_: trace ID, correlation ID, request UUID

## Architecture

**Core Layer:**
The shared foundational directory (core/) holding: config.py (pydantic-settings from .env), dependencies.py (OpenAI/Qdrant client singletons), exceptions.py (custom exception classes + FastAPI handlers), models.py (Chunk, StrategyName, pagination). All other domains import upward to core, never sideways to each other.
_Avoid_: common, shared, base

**Domain Directory:**
A self-contained vertical slice of the application containing its own routes, service logic, models, and templates. The four domains: documents/, chunking/, benchmarking/, storage/.
_Avoid_: module, feature folder

**Service Layer:**
The business logic layer within each domain directory. Services contain orchestration logic, call external APIs (OpenAI, Qdrant), and are imported by route handlers. A service never imports from another domain's service — all shared logic goes through core/.
_Avoid_: manager, logic, use case

**Router:**
A FastAPI APIRouter instance, one per domain, mounted onto the main FastAPI app in main.py. Each domain's routes.py exports its own router. The main app composes them: app.include_router(documents_router), etc.
_Avoid_: blueprint, handler group

**Strategy Pattern:**
The design pattern used for chunking strategies. All strategies inherit from a base ChunkingStrategy class with a chunk(document) method returning a list of Chunk objects. New strategies are added by implementing the interface, not by modifying existing code.
_Avoid_: polymorphism, duck typing

**Pipeline:**
The sequential processing chain from DOCX to Qdrant. Each strategy defines its own pipeline. Pipelines are synchronous and run within a single request lifecycle (POST /documents/{id}/process).
_Avoid_: workflow, chain, flow

**Retrieval Pipeline:**
The query-time chain: user question > vector search in Qdrant > retrieve top_k chunks > send chunks + question to LLM > return answer with metadata (retrieved chunks, latency, token usage).
_Avoid_: search flow, query pipeline

**Sync Processing:**
All API endpoints execute synchronously and return results directly. No background job queue, no polling endpoint, no async task tracking. Document processing takes 30-60 seconds; benchmarking takes 2-5 minutes. Both are acceptable within the request lifecycle.
_Avoid_: async, background job, task queue

**Import Rules:**
Strict unidirectional dependency: core/ imports nothing from the project. documents/, chunking/, benchmarking/, storage/ each import only from core/. No domain imports from another domain. If benchmarking needs to reference a query, it calls a function via the Query service, not by importing chunking/ directly.
_Avoid_: dependency direction, import graph

**Shared DocumentTree:**
The parsed document hierarchy extracted once at upload time by documents/parser.py (python-docx). Stored as JSON in SQLite. All strategies that need hierarchy (contextual, parent_child, markdown) read from this stored tree. Structure extraction happens once regardless of how many strategies are requested.
_Avoid_: parsed structure, document hierarchy

**App Factory:**
The main.py module that creates and configures the FastAPI application. Responsibilities: create FastAPI instance, mount all domain routers, register exception handlers from core/, configure middleware (CORS, request ID injection).
_Avoid_: app init, application setup

## Evaluation

**Query:**
A single question asked against a specific chunking strategy. POST /queries with {question, strategy, top_k}. Returns: generated answer, retrieved chunks (with metadata), latency breakdown, token usage. Stored in SQLite for history.
_Avoid_: request, question, ask

**Benchmark:**
A comparison run that evaluates one or more strategies against one or more questions. Runs synchronously. Two modes: single question (POST /benchmarks) or dataset (POST /benchmarks with questions.json body). Produces an experiment record in SQLite with per-strategy metrics.
_Avoid_: comparison, evaluation run, test

**Dataset Benchmark:**
A benchmark mode that processes multiple questions from a JSON file. Input: array of {question, answer} objects. Runs each question against all selected strategies, aggregates metrics, produces a single experiment with overall scores.
_Avoid_: batch benchmark, multi-question benchmark

**Dry Run:**
A benchmark mode that estimates the number of LLM calls and approximate cost without executing. Triggered via query parameter: POST /benchmarks?dry_run=true. Returns: estimated_calls, estimated_cost_usd. Used for cost visibility before committing to a run.
_Avoid_: preview, simulation

**LLM-as-Judge:**
The evaluation paradigm where gpt-4o-mini scores the quality of answers and retrieval. Same model used for generation and evaluation. Scores are on a 1-10 scale. Applied to: context relevance, answer similarity, faithfulness. Hallucination is binary.
_Avoid_: automated evaluation, model evaluation

**Experiment Metrics:**
Composite scores for a benchmark run stored in SQLite. Includes: context_relevance (1-10), answer_similarity (1-10), faithfulness (1-10), hallucination (bool), plus cost breakdowns per strategy.
_Avoid_: results, evaluation data

**Latency Breakdown:**
Timing measurements recorded for each operation: chunking_time (strategy execution), embedding_time (API calls to text-embedding-3-small), retrieval_time (Qdrant vector search), llm_time (OpenAI generation + evaluation calls). All measured in seconds. Total latency is the sum.
_Avoid_: timing data, performance metrics

**Context Relevance:**
LLM-judged score (1-10) measuring how relevant the retrieved chunks are to the question. Evaluated by asking gpt-4o-mini: "Given this question, how relevant are these chunks?"
_Avoid_: retrieval relevance, chunk relevance

**Answer Similarity:**
LLM-judged score (1-10) measuring how close the generated answer is to the expected answer. Evaluated by asking gpt-4o-mini: "Compare these two answers for semantic similarity."
_Avoid_: answer match, correctness

**Faithfulness:**
LLM-judged score (1-10) measuring whether the generated answer is supported by the retrieved chunks. Low faithfulness = the answer uses information outside the retrieved context. Evaluated by asking gpt-4o-mini: "Is this answer supported by these chunks?"
_Avoid_: groundedness, support

**Hallucination:**
Binary flag indicating the generated answer contains claims not present in the retrieved chunks. Set to true when the LLM-as-judge detects unsupported information in the answer.
_Avoid_: fabrication, confabulation

**Cost Estimation:**
The dry-run calculation that estimates: number of LLM calls needed (per strategy per question for generation + evaluation), approximate tokens per call, estimated cost at current pricing. Does not execute — returns estimate only.
_Avoid_: price estimate, budget calculation

**Token Tracking:**
Token usage recorded at two levels: (1) tiktoken for offline chunk metadata (token_count per chunk, stored at chunking time), (2) OpenAI API response usage field for actual billing data on generation and evaluation calls.
_Avoid_: token counting, usage tracking

## Tech Stack

**Framework:**
FastAPI — the web framework. Chosen for: auto-generated OpenAPI docs (Swagger), Pydantic validation, async support (available for future use), and clean dependency injection via Depends().
_Avoid_: web framework, HTTP framework

**python-docx:**
The library used for DOCX parsing and document structure extraction. Provides direct access to paragraph styles, heading levels, and document hierarchy. Chosen over pandoc/mammoth because it preserves native document structure needed by contextual, parent-child, and markdown strategies.
_Avoid_: docx parser, word parser

**OpenAI API:**
The external API for: (1) text-embedding-3-small embeddings, (2) gpt-4o-mini for enrichment, generation, and evaluation. Accessed via the openai Python SDK. All calls are synchronous.
_Avoid_: LLM API, embedding API

**Qdrant SDK:**
The Python client for Qdrant vector database. Used for: collection creation, vector upsert, similarity search. Accessed via the qdrant-client package.
_Avoid_: vector client, qdrant client

**tiktoken:**
OpenAI's tokenizer library used for offline token counting. Applied to chunks at chunking time to compute token_count metadata. Not used for API cost calculation (that uses the OpenAI response usage field).
_Avoid_: tokenizer, token counter

**Jinja2:**
The templating engine used for HTML benchmark reports. Template lives in benchmarking/templates/report.html. Renders per-strategy comparison tables, per-question breakdowns, and metric visualizations.
_Avoid_: HTML generator, template engine

**pydantic-settings:**
The library used for configuration management. Settings are loaded from .env with type validation, defaults, and environment variable overrides.
_Avoid_: config loader, settings manager

## Configuration

**Environment Variables (.env):**

| Variable | Default | Description |
|----------|---------|-------------|
| OPENAI_API_KEY | (required) | OpenAI API authentication |
| QDRANT_URL | http://localhost:6333 | Qdrant server URL |
| QDRANT_API_KEY | (optional) | Qdrant auth for cloud instances |
| EMBEDDING_MODEL | text-embedding-3-small | Fixed embedding model |
| LLM_MODEL | gpt-4o-mini | Fixed LLM for generation + evaluation |
| TOP_K | 5 | Default chunks to retrieve |
| TEMPERATURE | 0.0 | LLM temperature for deterministic output |
| MAX_TOKENS | 1024 | Max tokens in LLM response |
| CHUNK_SIZE | 512 | Target chunk size in characters |
| CHUNK_OVERLAP | 50 | Overlap between adjacent chunks |
| SEMANTIC_THRESHOLD | 0.3 | Cosine similarity cutoff for semantic chunking |
| SEMANTIC_MIN_CHUNK_SIZE | 3 | Minimum sentences per semantic chunk |

_Avoid_: config variables, env vars