chunking_strategies_evaluation/CONTEXT.md

RAG Chunking Benchmarker

A benchmarking application that compares chunking strategies on regulatory documents, measuring retrieval quality, answer faithfulness, and cost across strategies.

Domain

Chunk: A discrete unit of text extracted from a document, ready for embedding and storage. Every strategy produces chunks conforming to the unified Chunk model. Avoid: fragment, segment, block

Unified Chunk Model: A single Pydantic model that all five chunking strategies must return. Strategy-specific fields (e.g., parent_id) are nullable when not applicable. This ensures Qdrant payload schemas are consistent and benchmarking compares equivalent structures. Avoid: chunk schema, chunk type

Chunk Metadata: Fields stored with every chunk in Qdrant: document_name, chunk_id, strategy_name, chunk_index, token_count, character_count. No page_number (unreliable in DOCX). chunk_index is the position in the full document's chunk sequence. Avoid: chunk fields, payload data

Chunking Strategy: An algorithm that determines how a document is split into chunks. Must inherit from the ChunkingStrategy base class and implement the chunk() method. The five strategies are: contextual_structure, parent_child, semantic, markdown_structure, recursive. Avoid: splitter, chunker, method

Strategy Name: The canonical identifier for each chunking strategy: contextual_structure, parent_child, semantic, markdown_structure, recursive. Used in collection names, metadata, API parameters, and benchmark comparisons. Avoid: strategy type, strategy id

Contextual Structure Strategy: Pipeline: DOCX > Markdown > Structure Extraction > LLM enrichment (GPT-4o-mini per section) > Chunks > Embeddings > Qdrant. Each chunk is enriched with Document/Section/Article/Context/Content labels. The full enriched text is what gets embedded (not just the content). Hard-fails if the LLM enrichment call fails — no degraded chunks stored. Avoid: contextual chunking, enriched strategy

Parent-Child Strategy: Parent chunks are Article-level units (hardcoded for v1). Child chunks are sentences or paragraphs within each Article. Only child vectors are stored in Qdrant with a parent_id reference. At retrieval time, the matching child is found via vector search, then its full parent Article is returned as context to the LLM. Avoid: hierarchical chunking, nested strategy

Parent Level: The document hierarchy level used as parent in parent-child chunking. Hardcoded to "article" for v1. Configurable in future versions (article, section, paragraph) when additional document types are supported. Avoid: parent hierarchy, parent depth

Semantic Strategy: Pipeline: DOCX > Markdown > Sentence extraction > Sentence embeddings > Cosine similarity between adjacent sentences > Dynamic chunk creation. When similarity drops below the semantic threshold, a new chunk boundary is created. Enforces a minimum chunk size (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences). Boundary sentence stays with the previous chunk. Avoid: embedding-based chunking, similarity chunking

Semantic Threshold: The cosine similarity value below which adjacent sentences are considered a topic shift. When cosine_sim(sentence_n, sentence_n+1) < threshold, a new chunk boundary is created. Configurable via SEMANTIC_THRESHOLD in .env. Avoid: similarity cutoff, cut threshold

Minimum Chunk Size: The minimum number of sentences a semantic chunk must contain. If a boundary is detected before the minimum is reached, accumulation continues. Configurable via SEMANTIC_MIN_CHUNK_SIZE in .env. Avoid: min chunk, smallest chunk

Semantic Boundary: A point in the document where adjacent sentence similarity drops below the semantic threshold, marking where one chunk ends and another begins. Avoid: cut point, split boundary

Markdown Structure Strategy: Pipeline: DOCX > Markdown > Header detection > Paragraph grouping > Leaf-level chunk creation > Qdrant. Chunks are created only at the deepest header level (leaf chunks). No overlap between parent and child sections — a ## section is NOT a chunk if it contains ### subsections. Avoid: header-based chunking, markdown chunking

Leaf Chunk: A chunk at the deepest header level in markdown chunking. Only leaf-level chunks are created (no overlap with parent sections). Avoid: atomic chunk, granular chunk

Recursive Strategy: Pipeline: DOCX > Markdown > Cascade splitting using a separator hierarchy. Separators applied in order: (1) Markdown headers (#, ##, ###), (2) double newline (\n\n), (3) single newline (\n), (4) sentence-ending punctuation (. ! ? followed by space), (5) space (word-level, last resort). Splitting stops when chunks reach the target size. Each chunk stores which separator level was used. Avoid: hierarchical splitting, recursive splitting

Separator Cascade: The ordered list of separators used by recursive chunking: headers > double newline > single newline > sentence-ending punctuation > word boundary. Applied top-down; each level is tried before falling back to the next. Avoid: separator list, split pattern

Document Tree: The hierarchical structure of a parsed document: Document > Section > Article > Paragraph. Extracted once via python-docx at upload time, stored in SQLite, and shared across all strategies. Strategies that need hierarchy (contextual, parent_child, markdown) read from the stored tree; they do not re-parse. Avoid: doc tree, document structure, hierarchy

Embedding: The vector representation of a chunk's text, produced by text-embedding-3-small. For the contextual_structure strategy, the full enriched text is embedded. For all other strategies, the raw chunk text is embedded. Avoid: vector, vector representation, encoding

Top-k: The number of chunks to retrieve during vector similarity search. Default is 5. Configurable per query via the API (POST /queries) and per benchmark via the dataset configuration. Avoid: k, retrieve count, result limit

Embedding Model: The fixed OpenAI model used for all embedding operations: text-embedding-3-small. Not configurable — every strategy uses the same embedding model to ensure fair comparison. Avoid: vector model, encoder model

LLM Model: The fixed OpenAI model used for all generation and evaluation: gpt-4o-mini. Used for: (1) contextual enrichment during chunking, (2) answer generation during queries, (3) LLM-as-judge evaluation during benchmarks. Avoid: generation model, chat model

Storage

Strategy Collection: A Qdrant collection named after a strategy. Naming convention: {strategy_name}_collection (e.g., contextual_structure_collection, semantic_collection). Each strategy writes to its own collection. All collections share the same vector dimension (from text-embedding-3-small) and payload schema (unified Chunk model). Avoid: vector store, collection only

Parent: In parent-child chunking, a larger semantic unit (Article-level) whose child chunks are finer-grained pieces. Only child chunks are embedded and stored in Qdrant; the parent is retrieved at query time for context expansion. Avoid: source chunk, container

Child: In parent-child chunking, a fine-grained piece (sentence or paragraph within an Article) that gets embedded and stored in Qdrant. Carries a parent_id reference to its Article parent. Avoid: sub-chunk, fragment

Experiment: A completed benchmark run. Every benchmark (single question or dataset) stores an experiment record in SQLite with: all questions, generated answers, retrieved chunks per strategy, evaluation metrics, configuration snapshot, and timestamps. Avoid: run, trial, test

SQLite: The persistence layer for structured data. Stores: documents (parsed text + document tree JSON), experiments (benchmark results), queries (single question/answer history), per-strategy metrics, and timestamps. Does NOT store vectors — those live in Qdrant. Avoid: database, DB, local storage

Qdrant: The vector database. Stores chunk embeddings and their payload metadata. One collection per strategy. Used for vector similarity search during retrieval. No hybrid search, no BM25, no filters — pure vector similarity only. Avoid: vector DB, vector store

REST API

Resource: A noun-based URL entity in the REST API. The four primary resources are Documents, Queries, Benchmarks, and the read-only Strategies list. Avoid: endpoint, API, route

Endpoint: A specific URL path combined with an HTTP method. The complete endpoint map:

Method	Endpoint	Description
POST	/documents	Upload .docx file (multipart)
POST	/documents/{id}/process	Run selected strategies (synchronous)
DELETE	/documents/{id}	Remove document + all its chunks
GET	/strategies	List available strategies
POST	/queries	Ask a question against a strategy
GET	/queries/{id}	Retrieve past query
POST	/benchmarks	Run benchmark comparison
POST	/benchmarks?dry_run=true	Estimate cost without executing
GET	/benchmarks/{id}	Retrieve benchmark results
GET	/benchmarks/{id}/report	Download HTML report
Avoid: API, route, handler (use "endpoint" for URL+method, "handler" only for the Python function)

Query Parameter: A key-value pair appended to the URL after ? (e.g., ?dry_run=true). Used for optional modifiers that don't change the resource identity. Contrast with path parameters that identify the resource (e.g., /benchmarks/{id}). Avoid: URL parameter, request parameter

Path Parameter: A variable segment in the URL path identified by curly braces (e.g., /documents/{id}). Identifies a specific resource instance. Always an ID string. Avoid: URL variable, route parameter

Multipart Upload: The format used for uploading .docx files via POST /documents. The request Content-Type is multipart/form-data with the file attached as a binary field. Avoid: file upload, binary upload

Request ID: A unique identifier attached to every inbound request, propagated through logs and stored in experiment records. Enables tracing a single operation across multiple service calls (document parsing, embedding, LLM calls). Avoid: trace ID, correlation ID, request UUID

Architecture

Core Layer: The shared foundational directory (core/) holding: config.py (pydantic-settings from .env), dependencies.py (OpenAI/Qdrant client singletons), exceptions.py (custom exception classes + FastAPI handlers), models.py (Chunk, StrategyName, pagination). All other domains import upward to core, never sideways to each other. Avoid: common, shared, base

Domain Directory: A self-contained vertical slice of the application containing its own routes, service logic, models, and templates. The four domains: documents/, chunking/, benchmarking/, storage/. Avoid: module, feature folder

Service Layer: The business logic layer within each domain directory. Services contain orchestration logic, call external APIs (OpenAI, Qdrant), and are imported by route handlers. A service never imports from another domain's service — all shared logic goes through core/. Avoid: manager, logic, use case

Router: A FastAPI APIRouter instance, one per domain, mounted onto the main FastAPI app in main.py. Each domain's routes.py exports its own router. The main app composes them: app.include_router(documents_router), etc. Avoid: blueprint, handler group

Strategy Pattern: The design pattern used for chunking strategies. All strategies inherit from a base ChunkingStrategy class with a chunk(document) method returning a list of Chunk objects. New strategies are added by implementing the interface, not by modifying existing code. Avoid: polymorphism, duck typing

Pipeline: The sequential processing chain from DOCX to Qdrant. Each strategy defines its own pipeline. Pipelines are synchronous and run within a single request lifecycle (POST /documents/{id}/process). Avoid: workflow, chain, flow

Retrieval Pipeline: The query-time chain: user question > vector search in Qdrant > retrieve top_k chunks > send chunks + question to LLM > return answer with metadata (retrieved chunks, latency, token usage). Avoid: search flow, query pipeline

Sync Processing: All API endpoints execute synchronously and return results directly. No background job queue, no polling endpoint, no async task tracking. Document processing takes 30-60 seconds; benchmarking takes 2-5 minutes. Both are acceptable within the request lifecycle. Avoid: async, background job, task queue

Import Rules: Strict unidirectional dependency: core/ imports nothing from the project. documents/, chunking/, benchmarking/, storage/ each import only from core/. No domain imports from another domain. If benchmarking needs to reference a query, it calls a function via the Query service, not by importing chunking/ directly. Avoid: dependency direction, import graph

Shared DocumentTree: The parsed document hierarchy extracted once at upload time by documents/parser.py (python-docx). Stored as JSON in SQLite. All strategies that need hierarchy (contextual, parent_child, markdown) read from this stored tree. Structure extraction happens once regardless of how many strategies are requested. Avoid: parsed structure, document hierarchy

App Factory: The main.py module that creates and configures the FastAPI application. Responsibilities: create FastAPI instance, mount all domain routers, register exception handlers from core/, configure middleware (CORS, request ID injection). Avoid: app init, application setup

Evaluation

Query: A single question asked against a specific chunking strategy. POST /queries with {question, strategy, top_k}. Returns: generated answer, retrieved chunks (with metadata), latency breakdown, token usage. Stored in SQLite for history. Avoid: request, question, ask

Benchmark: A comparison run that evaluates one or more strategies against one or more questions. Runs synchronously. Two modes: single question (POST /benchmarks) or dataset (POST /benchmarks with questions.json body). Produces an experiment record in SQLite with per-strategy metrics. Avoid: comparison, evaluation run, test

Dataset Benchmark: A benchmark mode that processes multiple questions from a JSON file. Input: array of {question, answer} objects. Runs each question against all selected strategies, aggregates metrics, produces a single experiment with overall scores. Avoid: batch benchmark, multi-question benchmark

Dry Run: A benchmark mode that estimates the number of LLM calls and approximate cost without executing. Triggered via query parameter: POST /benchmarks?dry_run=true. Returns: estimated_calls, estimated_cost_usd. Used for cost visibility before committing to a run. Avoid: preview, simulation

LLM-as-Judge: The evaluation paradigm where gpt-4o-mini scores the quality of answers and retrieval. Same model used for generation and evaluation. Scores are on a 1-10 scale. Applied to: context relevance, answer similarity, faithfulness. Hallucination is binary. Avoid: automated evaluation, model evaluation

Experiment Metrics: Composite scores for a benchmark run stored in SQLite. Includes: context_relevance (1-10), answer_similarity (1-10), faithfulness (1-10), hallucination (bool), plus cost breakdowns per strategy. Avoid: results, evaluation data

Latency Breakdown: Timing measurements recorded for each operation: chunking_time (strategy execution), embedding_time (API calls to text-embedding-3-small), retrieval_time (Qdrant vector search), llm_time (OpenAI generation + evaluation calls). All measured in seconds. Total latency is the sum. Avoid: timing data, performance metrics

Context Relevance: LLM-judged score (1-10) measuring how relevant the retrieved chunks are to the question. Evaluated by asking gpt-4o-mini: "Given this question, how relevant are these chunks?" Avoid: retrieval relevance, chunk relevance

Answer Similarity: LLM-judged score (1-10) measuring how close the generated answer is to the expected answer. Evaluated by asking gpt-4o-mini: "Compare these two answers for semantic similarity." Avoid: answer match, correctness

Faithfulness: LLM-judged score (1-10) measuring whether the generated answer is supported by the retrieved chunks. Low faithfulness = the answer uses information outside the retrieved context. Evaluated by asking gpt-4o-mini: "Is this answer supported by these chunks?" Avoid: groundedness, support

Hallucination: Binary flag indicating the generated answer contains claims not present in the retrieved chunks. Set to true when the LLM-as-judge detects unsupported information in the answer. Avoid: fabrication, confabulation

Cost Estimation: The dry-run calculation that estimates: number of LLM calls needed (per strategy per question for generation + evaluation), approximate tokens per call, estimated cost at current pricing. Does not execute — returns estimate only. Avoid: price estimate, budget calculation

Token Tracking: Token usage recorded at two levels: (1) tiktoken for offline chunk metadata (token_count per chunk, stored at chunking time), (2) OpenAI API response usage field for actual billing data on generation and evaluation calls. Avoid: token counting, usage tracking

Tech Stack

Framework: FastAPI — the web framework. Chosen for: auto-generated OpenAPI docs (Swagger), Pydantic validation, async support (available for future use), and clean dependency injection via Depends(). Avoid: web framework, HTTP framework

python-docx: The library used for DOCX parsing and document structure extraction. Provides direct access to paragraph styles, heading levels, and document hierarchy. Chosen over pandoc/mammoth because it preserves native document structure needed by contextual, parent-child, and markdown strategies. Avoid: docx parser, word parser

OpenAI API: The external API for: (1) text-embedding-3-small embeddings, (2) gpt-4o-mini for enrichment, generation, and evaluation. Accessed via the openai Python SDK. All calls are synchronous. Avoid: LLM API, embedding API

Qdrant SDK: The Python client for Qdrant vector database. Used for: collection creation, vector upsert, similarity search. Accessed via the qdrant-client package. Avoid: vector client, qdrant client

tiktoken: OpenAI's tokenizer library used for offline token counting. Applied to chunks at chunking time to compute token_count metadata. Not used for API cost calculation (that uses the OpenAI response usage field). Avoid: tokenizer, token counter

Jinja2: The templating engine used for HTML benchmark reports. Template lives in benchmarking/templates/report.html. Renders per-strategy comparison tables, per-question breakdowns, and metric visualizations. Avoid: HTML generator, template engine

pydantic-settings: The library used for configuration management. Settings are loaded from .env with type validation, defaults, and environment variable overrides. Avoid: config loader, settings manager

Configuration

Environment Variables (.env):

Variable	Default	Description
OPENAI_API_KEY	(required)	OpenAI API authentication
QDRANT_URL	http://localhost:6333	Qdrant server URL
QDRANT_API_KEY	(optional)	Qdrant auth for cloud instances
EMBEDDING_MODEL	text-embedding-3-small	Fixed embedding model
LLM_MODEL	gpt-4o-mini	Fixed LLM for generation + evaluation
TOP_K	5	Default chunks to retrieve
TEMPERATURE	0.0	LLM temperature for deterministic output
MAX_TOKENS	1024	Max tokens in LLM response
CHUNK_SIZE	512	Target chunk size in characters
CHUNK_OVERLAP	50	Overlap between adjacent chunks
SEMANTIC_THRESHOLD	0.3	Cosine similarity cutoff for semantic chunking
SEMANTIC_MIN_CHUNK_SIZE	3	Minimum sentences per semantic chunk

Avoid: config variables, env vars