From 09f9bb53bf9641228deb53127b0ff47437240cae Mon Sep 17 00:00:00 2001 From: Mahdi Bazrafshan Date: Tue, 21 Jul 2026 18:51:46 +0330 Subject: [PATCH] docs: add domain glossary and architectural decision records 15 ADRs covering chunk model, failure handling, evaluation, parsing, directory structure, REST design, and strategy-specific decisions. 62 domain terms across 7 sections. --- CONTEXT.md | 305 ++++++++++++++++++ ...01-through-0015-architectural-decisions.md | 263 +++++++++++++++ 2 files changed, 568 insertions(+) create mode 100644 CONTEXT.md create mode 100644 docs/adr/0001-through-0015-architectural-decisions.md diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..47f6b68 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,305 @@ +# RAG Chunking Benchmarker + +A benchmarking application that compares chunking strategies on regulatory documents, measuring retrieval quality, answer faithfulness, and cost across strategies. + +## Domain + +**Chunk:** +A discrete unit of text extracted from a document, ready for embedding and storage. Every strategy produces chunks conforming to the unified Chunk model. +_Avoid_: fragment, segment, block + +**Unified Chunk Model:** +A single Pydantic model that all five chunking strategies must return. Strategy-specific fields (e.g., parent_id) are nullable when not applicable. This ensures Qdrant payload schemas are consistent and benchmarking compares equivalent structures. +_Avoid_: chunk schema, chunk type + +**Chunk Metadata:** +Fields stored with every chunk in Qdrant: document_name, chunk_id, strategy_name, chunk_index, token_count, character_count. No page_number (unreliable in DOCX). chunk_index is the position in the full document's chunk sequence. +_Avoid_: chunk fields, payload data + +**Chunking Strategy:** +An algorithm that determines how a document is split into chunks. Must inherit from the ChunkingStrategy base class and implement the chunk() method. The five strategies are: contextual_structure, parent_child, semantic, markdown_structure, recursive. +_Avoid_: splitter, chunker, method + +**Strategy Name:** +The canonical identifier for each chunking strategy: contextual_structure, parent_child, semantic, markdown_structure, recursive. Used in collection names, metadata, API parameters, and benchmark comparisons. +_Avoid_: strategy type, strategy id + +**Contextual Structure Strategy:** +Pipeline: DOCX > Markdown > Structure Extraction > LLM enrichment (GPT-4o-mini per section) > Chunks > Embeddings > Qdrant. Each chunk is enriched with Document/Section/Article/Context/Content labels. The full enriched text is what gets embedded (not just the content). Hard-fails if the LLM enrichment call fails — no degraded chunks stored. +_Avoid_: contextual chunking, enriched strategy + +**Parent-Child Strategy:** +Parent chunks are Article-level units (hardcoded for v1). Child chunks are sentences or paragraphs within each Article. Only child vectors are stored in Qdrant with a parent_id reference. At retrieval time, the matching child is found via vector search, then its full parent Article is returned as context to the LLM. +_Avoid_: hierarchical chunking, nested strategy + +**Parent Level:** +The document hierarchy level used as parent in parent-child chunking. Hardcoded to "article" for v1. Configurable in future versions (article, section, paragraph) when additional document types are supported. +_Avoid_: parent hierarchy, parent depth + +**Semantic Strategy:** +Pipeline: DOCX > Markdown > Sentence extraction > Sentence embeddings > Cosine similarity between adjacent sentences > Dynamic chunk creation. When similarity drops below the semantic threshold, a new chunk boundary is created. Enforces a minimum chunk size (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences). Boundary sentence stays with the previous chunk. +_Avoid_: embedding-based chunking, similarity chunking + +**Semantic Threshold:** +The cosine similarity value below which adjacent sentences are considered a topic shift. When cosine_sim(sentence_n, sentence_n+1) < threshold, a new chunk boundary is created. Configurable via SEMANTIC_THRESHOLD in .env. +_Avoid_: similarity cutoff, cut threshold + +**Minimum Chunk Size:** +The minimum number of sentences a semantic chunk must contain. If a boundary is detected before the minimum is reached, accumulation continues. Configurable via SEMANTIC_MIN_CHUNK_SIZE in .env. +_Avoid_: min chunk, smallest chunk + +**Semantic Boundary:** +A point in the document where adjacent sentence similarity drops below the semantic threshold, marking where one chunk ends and another begins. +_Avoid_: cut point, split boundary + +**Markdown Structure Strategy:** +Pipeline: DOCX > Markdown > Header detection > Paragraph grouping > Leaf-level chunk creation > Qdrant. Chunks are created only at the deepest header level (leaf chunks). No overlap between parent and child sections — a ## section is NOT a chunk if it contains ### subsections. +_Avoid_: header-based chunking, markdown chunking + +**Leaf Chunk:** +A chunk at the deepest header level in markdown chunking. Only leaf-level chunks are created (no overlap with parent sections). +_Avoid_: atomic chunk, granular chunk + +**Recursive Strategy:** +Pipeline: DOCX > Markdown > Cascade splitting using a separator hierarchy. Separators applied in order: (1) Markdown headers (#, ##, ###), (2) double newline (\n\n), (3) single newline (\n), (4) sentence-ending punctuation (. ! ? followed by space), (5) space (word-level, last resort). Splitting stops when chunks reach the target size. Each chunk stores which separator level was used. +_Avoid_: hierarchical splitting, recursive splitting + +**Separator Cascade:** +The ordered list of separators used by recursive chunking: headers > double newline > single newline > sentence-ending punctuation > word boundary. Applied top-down; each level is tried before falling back to the next. +_Avoid_: separator list, split pattern + +**Document Tree:** +The hierarchical structure of a parsed document: Document > Section > Article > Paragraph. Extracted once via python-docx at upload time, stored in SQLite, and shared across all strategies. Strategies that need hierarchy (contextual, parent_child, markdown) read from the stored tree; they do not re-parse. +_Avoid_: doc tree, document structure, hierarchy + +**Embedding:** +The vector representation of a chunk's text, produced by text-embedding-3-small. For the contextual_structure strategy, the full enriched text is embedded. For all other strategies, the raw chunk text is embedded. +_Avoid_: vector, vector representation, encoding + +**Top-k:** +The number of chunks to retrieve during vector similarity search. Default is 5. Configurable per query via the API (POST /queries) and per benchmark via the dataset configuration. +_Avoid_: k, retrieve count, result limit + +**Embedding Model:** +The fixed OpenAI model used for all embedding operations: text-embedding-3-small. Not configurable — every strategy uses the same embedding model to ensure fair comparison. +_Avoid_: vector model, encoder model + +**LLM Model:** +The fixed OpenAI model used for all generation and evaluation: gpt-4o-mini. Used for: (1) contextual enrichment during chunking, (2) answer generation during queries, (3) LLM-as-judge evaluation during benchmarks. +_Avoid_: generation model, chat model + +## Storage + +**Strategy Collection:** +A Qdrant collection named after a strategy. Naming convention: {strategy_name}_collection (e.g., contextual_structure_collection, semantic_collection). Each strategy writes to its own collection. All collections share the same vector dimension (from text-embedding-3-small) and payload schema (unified Chunk model). +_Avoid_: vector store, collection only + +**Parent:** +In parent-child chunking, a larger semantic unit (Article-level) whose child chunks are finer-grained pieces. Only child chunks are embedded and stored in Qdrant; the parent is retrieved at query time for context expansion. +_Avoid_: source chunk, container + +**Child:** +In parent-child chunking, a fine-grained piece (sentence or paragraph within an Article) that gets embedded and stored in Qdrant. Carries a parent_id reference to its Article parent. +_Avoid_: sub-chunk, fragment + +**Experiment:** +A completed benchmark run. Every benchmark (single question or dataset) stores an experiment record in SQLite with: all questions, generated answers, retrieved chunks per strategy, evaluation metrics, configuration snapshot, and timestamps. +_Avoid_: run, trial, test + +**SQLite:** +The persistence layer for structured data. Stores: documents (parsed text + document tree JSON), experiments (benchmark results), queries (single question/answer history), per-strategy metrics, and timestamps. Does NOT store vectors — those live in Qdrant. +_Avoid_: database, DB, local storage + +**Qdrant:** +The vector database. Stores chunk embeddings and their payload metadata. One collection per strategy. Used for vector similarity search during retrieval. No hybrid search, no BM25, no filters — pure vector similarity only. +_Avoid_: vector DB, vector store + +## REST API + +**Resource:** +A noun-based URL entity in the REST API. The four primary resources are Documents, Queries, Benchmarks, and the read-only Strategies list. +_Avoid_: endpoint, API, route + +**Endpoint:** +A specific URL path combined with an HTTP method. The complete endpoint map: + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | /documents | Upload .docx file (multipart) | +| POST | /documents/{id}/process | Run selected strategies (synchronous) | +| DELETE | /documents/{id} | Remove document + all its chunks | +| GET | /strategies | List available strategies | +| POST | /queries | Ask a question against a strategy | +| GET | /queries/{id} | Retrieve past query | +| POST | /benchmarks | Run benchmark comparison | +| POST | /benchmarks?dry_run=true | Estimate cost without executing | +| GET | /benchmarks/{id} | Retrieve benchmark results | +| GET | /benchmarks/{id}/report | Download HTML report | +_Avoid_: API, route, handler (use "endpoint" for URL+method, "handler" only for the Python function) + +**Query Parameter:** +A key-value pair appended to the URL after `?` (e.g., ?dry_run=true). Used for optional modifiers that don't change the resource identity. Contrast with path parameters that identify the resource (e.g., /benchmarks/{id}). +_Avoid_: URL parameter, request parameter + +**Path Parameter:** +A variable segment in the URL path identified by curly braces (e.g., /documents/{id}). Identifies a specific resource instance. Always an ID string. +_Avoid_: URL variable, route parameter + +**Multipart Upload:** +The format used for uploading .docx files via POST /documents. The request Content-Type is multipart/form-data with the file attached as a binary field. +_Avoid_: file upload, binary upload + +**Request ID:** +A unique identifier attached to every inbound request, propagated through logs and stored in experiment records. Enables tracing a single operation across multiple service calls (document parsing, embedding, LLM calls). +_Avoid_: trace ID, correlation ID, request UUID + +## Architecture + +**Core Layer:** +The shared foundational directory (core/) holding: config.py (pydantic-settings from .env), dependencies.py (OpenAI/Qdrant client singletons), exceptions.py (custom exception classes + FastAPI handlers), models.py (Chunk, StrategyName, pagination). All other domains import upward to core, never sideways to each other. +_Avoid_: common, shared, base + +**Domain Directory:** +A self-contained vertical slice of the application containing its own routes, service logic, models, and templates. The four domains: documents/, chunking/, benchmarking/, storage/. +_Avoid_: module, feature folder + +**Service Layer:** +The business logic layer within each domain directory. Services contain orchestration logic, call external APIs (OpenAI, Qdrant), and are imported by route handlers. A service never imports from another domain's service — all shared logic goes through core/. +_Avoid_: manager, logic, use case + +**Router:** +A FastAPI APIRouter instance, one per domain, mounted onto the main FastAPI app in main.py. Each domain's routes.py exports its own router. The main app composes them: app.include_router(documents_router), etc. +_Avoid_: blueprint, handler group + +**Strategy Pattern:** +The design pattern used for chunking strategies. All strategies inherit from a base ChunkingStrategy class with a chunk(document) method returning a list of Chunk objects. New strategies are added by implementing the interface, not by modifying existing code. +_Avoid_: polymorphism, duck typing + +**Pipeline:** +The sequential processing chain from DOCX to Qdrant. Each strategy defines its own pipeline. Pipelines are synchronous and run within a single request lifecycle (POST /documents/{id}/process). +_Avoid_: workflow, chain, flow + +**Retrieval Pipeline:** +The query-time chain: user question > vector search in Qdrant > retrieve top_k chunks > send chunks + question to LLM > return answer with metadata (retrieved chunks, latency, token usage). +_Avoid_: search flow, query pipeline + +**Sync Processing:** +All API endpoints execute synchronously and return results directly. No background job queue, no polling endpoint, no async task tracking. Document processing takes 30-60 seconds; benchmarking takes 2-5 minutes. Both are acceptable within the request lifecycle. +_Avoid_: async, background job, task queue + +**Import Rules:** +Strict unidirectional dependency: core/ imports nothing from the project. documents/, chunking/, benchmarking/, storage/ each import only from core/. No domain imports from another domain. If benchmarking needs to reference a query, it calls a function via the Query service, not by importing chunking/ directly. +_Avoid_: dependency direction, import graph + +**Shared DocumentTree:** +The parsed document hierarchy extracted once at upload time by documents/parser.py (python-docx). Stored as JSON in SQLite. All strategies that need hierarchy (contextual, parent_child, markdown) read from this stored tree. Structure extraction happens once regardless of how many strategies are requested. +_Avoid_: parsed structure, document hierarchy + +**App Factory:** +The main.py module that creates and configures the FastAPI application. Responsibilities: create FastAPI instance, mount all domain routers, register exception handlers from core/, configure middleware (CORS, request ID injection). +_Avoid_: app init, application setup + +## Evaluation + +**Query:** +A single question asked against a specific chunking strategy. POST /queries with {question, strategy, top_k}. Returns: generated answer, retrieved chunks (with metadata), latency breakdown, token usage. Stored in SQLite for history. +_Avoid_: request, question, ask + +**Benchmark:** +A comparison run that evaluates one or more strategies against one or more questions. Runs synchronously. Two modes: single question (POST /benchmarks) or dataset (POST /benchmarks with questions.json body). Produces an experiment record in SQLite with per-strategy metrics. +_Avoid_: comparison, evaluation run, test + +**Dataset Benchmark:** +A benchmark mode that processes multiple questions from a JSON file. Input: array of {question, answer} objects. Runs each question against all selected strategies, aggregates metrics, produces a single experiment with overall scores. +_Avoid_: batch benchmark, multi-question benchmark + +**Dry Run:** +A benchmark mode that estimates the number of LLM calls and approximate cost without executing. Triggered via query parameter: POST /benchmarks?dry_run=true. Returns: estimated_calls, estimated_cost_usd. Used for cost visibility before committing to a run. +_Avoid_: preview, simulation + +**LLM-as-Judge:** +The evaluation paradigm where gpt-4o-mini scores the quality of answers and retrieval. Same model used for generation and evaluation. Scores are on a 1-10 scale. Applied to: context relevance, answer similarity, faithfulness. Hallucination is binary. +_Avoid_: automated evaluation, model evaluation + +**Experiment Metrics:** +Composite scores for a benchmark run stored in SQLite. Includes: context_relevance (1-10), answer_similarity (1-10), faithfulness (1-10), hallucination (bool), plus cost breakdowns per strategy. +_Avoid_: results, evaluation data + +**Latency Breakdown:** +Timing measurements recorded for each operation: chunking_time (strategy execution), embedding_time (API calls to text-embedding-3-small), retrieval_time (Qdrant vector search), llm_time (OpenAI generation + evaluation calls). All measured in seconds. Total latency is the sum. +_Avoid_: timing data, performance metrics + +**Context Relevance:** +LLM-judged score (1-10) measuring how relevant the retrieved chunks are to the question. Evaluated by asking gpt-4o-mini: "Given this question, how relevant are these chunks?" +_Avoid_: retrieval relevance, chunk relevance + +**Answer Similarity:** +LLM-judged score (1-10) measuring how close the generated answer is to the expected answer. Evaluated by asking gpt-4o-mini: "Compare these two answers for semantic similarity." +_Avoid_: answer match, correctness + +**Faithfulness:** +LLM-judged score (1-10) measuring whether the generated answer is supported by the retrieved chunks. Low faithfulness = the answer uses information outside the retrieved context. Evaluated by asking gpt-4o-mini: "Is this answer supported by these chunks?" +_Avoid_: groundedness, support + +**Hallucination:** +Binary flag indicating the generated answer contains claims not present in the retrieved chunks. Set to true when the LLM-as-judge detects unsupported information in the answer. +_Avoid_: fabrication, confabulation + +**Cost Estimation:** +The dry-run calculation that estimates: number of LLM calls needed (per strategy per question for generation + evaluation), approximate tokens per call, estimated cost at current pricing. Does not execute — returns estimate only. +_Avoid_: price estimate, budget calculation + +**Token Tracking:** +Token usage recorded at two levels: (1) tiktoken for offline chunk metadata (token_count per chunk, stored at chunking time), (2) OpenAI API response usage field for actual billing data on generation and evaluation calls. +_Avoid_: token counting, usage tracking + +## Tech Stack + +**Framework:** +FastAPI — the web framework. Chosen for: auto-generated OpenAPI docs (Swagger), Pydantic validation, async support (available for future use), and clean dependency injection via Depends(). +_Avoid_: web framework, HTTP framework + +**python-docx:** +The library used for DOCX parsing and document structure extraction. Provides direct access to paragraph styles, heading levels, and document hierarchy. Chosen over pandoc/mammoth because it preserves native document structure needed by contextual, parent-child, and markdown strategies. +_Avoid_: docx parser, word parser + +**OpenAI API:** +The external API for: (1) text-embedding-3-small embeddings, (2) gpt-4o-mini for enrichment, generation, and evaluation. Accessed via the openai Python SDK. All calls are synchronous. +_Avoid_: LLM API, embedding API + +**Qdrant SDK:** +The Python client for Qdrant vector database. Used for: collection creation, vector upsert, similarity search. Accessed via the qdrant-client package. +_Avoid_: vector client, qdrant client + +**tiktoken:** +OpenAI's tokenizer library used for offline token counting. Applied to chunks at chunking time to compute token_count metadata. Not used for API cost calculation (that uses the OpenAI response usage field). +_Avoid_: tokenizer, token counter + +**Jinja2:** +The templating engine used for HTML benchmark reports. Template lives in benchmarking/templates/report.html. Renders per-strategy comparison tables, per-question breakdowns, and metric visualizations. +_Avoid_: HTML generator, template engine + +**pydantic-settings:** +The library used for configuration management. Settings are loaded from .env with type validation, defaults, and environment variable overrides. +_Avoid_: config loader, settings manager + +## Configuration + +**Environment Variables (.env):** + +| Variable | Default | Description | +|----------|---------|-------------| +| OPENAI_API_KEY | (required) | OpenAI API authentication | +| QDRANT_URL | http://localhost:6333 | Qdrant server URL | +| QDRANT_API_KEY | (optional) | Qdrant auth for cloud instances | +| EMBEDDING_MODEL | text-embedding-3-small | Fixed embedding model | +| LLM_MODEL | gpt-4o-mini | Fixed LLM for generation + evaluation | +| TOP_K | 5 | Default chunks to retrieve | +| TEMPERATURE | 0.0 | LLM temperature for deterministic output | +| MAX_TOKENS | 1024 | Max tokens in LLM response | +| CHUNK_SIZE | 512 | Target chunk size in characters | +| CHUNK_OVERLAP | 50 | Overlap between adjacent chunks | +| SEMANTIC_THRESHOLD | 0.3 | Cosine similarity cutoff for semantic chunking | +| SEMANTIC_MIN_CHUNK_SIZE | 3 | Minimum sentences per semantic chunk | + +_Avoid_: config variables, env vars diff --git a/docs/adr/0001-through-0015-architectural-decisions.md b/docs/adr/0001-through-0015-architectural-decisions.md new file mode 100644 index 0000000..d66a631 --- /dev/null +++ b/docs/adr/0001-through-0015-architectural-decisions.md @@ -0,0 +1,263 @@ +# ADR 0001: Unified Chunk Model Across All Strategies + +All chunking strategies return chunks conforming to a single Pydantic model. Strategies that don't produce certain fields (e.g., parent_id for non-parent-child strategies) populate those fields as null. This keeps Qdrant payload schemas consistent and ensures benchmarking metrics compare equivalent structures. + +The unified model fields: +- document_name (str) +- chunk_id (str) +- strategy_name (StrategyName enum) +- chunk_index (int) — position in the full document's chunk sequence +- token_count (int) — computed via tiktoken at chunking time +- character_count (int) +- parent_id (Optional[str]) — only populated by parent_child strategy +- enriched_content (Optional[str]) — only populated by contextual_structure strategy + +The alternative — strategy-specific chunk shapes — would require per-strategy normalization before storage and make retrieval comparisons unreliable. With a unified model, every strategy's output is directly comparable in Qdrant payload structure, SQLite experiment records, and HTML benchmark reports. + +## Considered Options + +- Option A: Unified Chunk model with nullable fields for strategy-specific data +- Option B: Strategy-specific chunk shapes, normalized on ingestion + +Option A was chosen because it eliminates schema divergence and makes the benchmarking pipeline uniform across strategies. + +## Consequences + +All five strategies must conform to the same Pydantic model. Adding new fields later is straightforward (add field with default null). Removing fields requires updating all strategies and re-indexing Qdrant collections. The nullable fields approach means strategies only populate what they produce — no forced defaults or sentinel values. + +--- + +# ADR 0002: Hard-Fail on Contextual Enrichment Failure + +If the LLM call for contextual enrichment fails (rate limit, service outage, API error), the contextual_structure strategy raises an exception immediately. Chunks are NOT stored in degraded form (without enrichment). This is because a "contextual" chunk without context is just a markdown chunk and would corrupt the benchmark comparison — you'd be measuring markdown chunking quality while labeling it as contextual. + +The exception is caught at the strategy orchestration level (chunking/service.py), which marks the strategy as failed in the processing response. Other strategies in the same request proceed independently. + +## Considered Options + +- Hard-fail: raise exception, mark strategy as failed, store nothing +- Graceful degradation: store chunks without enrichment, mark as degraded +- Retry: retry the LLM call up to N times before failing + +Hard-fail was chosen because degraded chunks would produce misleading benchmark results. Retry was rejected because if the LLM is rate-limited, retrying within the same request just delays the inevitable and blocks other strategies. + +--- + +# ADR 0003: Per-Strategy Failure Isolation + +Each chunking strategy runs independently within a synchronous processing request. If one strategy fails (e.g., contextual_structure due to LLM unavailability), the other strategies' results are still committed. The processing response includes a strategies_completed list and a strategies_failed list with error details. + +The alternative — fail the entire request if any strategy fails — was rejected because it wastes already-computed results and forces the user to re-process strategies that already succeeded. + +Note: Although we initially considered background job status tracking (ADR 0009), this isolation principle still applies at the response level — partial results are returned rather than all-or-nothing. + +--- + +# ADR 0004: LLM-Judge Evaluation Without Retrieval Precision/Recall + +Benchmark evaluation uses LLM-as-judge scoring rather than retrieval Precision and Recall. The four evaluation metrics are: + +1. **Context Relevance** (1-10): Are the retrieved chunks relevant to the question? +2. **Answer Similarity** (1-10): How close is the generated answer to the expected answer? +3. **Faithfulness** (1-10): Is the answer supported by the retrieved chunks? +4. **Hallucination** (bool): Does the answer contain claims not in the retrieved context? + +This was chosen because Precision/Recall require ground truth labels mapping questions to "correct" chunks, which don't exist for a single-document benchmark. Since chunks are strategy-specific, the set of "relevant" chunks differs across strategies — requiring per-strategy labeling that would be a separate project. + +The architecture does not preclude adding labeled retrieval evaluation later. A labeled ground truth file could be loaded and Precision/Recall computed alongside the LLM-judge metrics without changing any existing code. + +--- + +# ADR 0005: Hardcoded Article-Level Parents in Parent-Child Chunking + +The parent-child strategy uses Article as the parent level. This is hardcoded for v1 because we have a single regulation document with clear Article structure within Sections. + +Making it configurable (article/section/paragraph) was considered but rejected because: +- It adds a configuration parameter that won't be used for v1 +- Each level requires different grouping logic (section contains multiple articles, paragraph is too granular) +- Testing the parameter adds overhead with no practical benefit + +The extension point is trivial: add a PARENT_CHILD_PARENT_LEVEL config variable and a conditional branch in the strategy. This is a 30-minute change when a second document type requires it. + +--- + +# ADR 0006: python-docx for DOCX Parsing and Structure Extraction + +Document parsing uses python-docx directly rather than routing through pandoc, mammoth, or other converters. + +The rationale: the contextual, parent-child, and markdown strategies all need the document's native structure tree (section/article/paragraph hierarchy from heading styles), not just flat text. python-docx provides direct access to paragraph styles and document hierarchy via its Document.paragraphs API, where each paragraph has a style.name property that identifies it as "Heading 1", "Heading 2", "Normal", etc. + +External converters (pandoc, mammoth) output flat markdown, which would require re-parsing the markdown text to recover heading levels — a fragile approach that loses style information and edge cases (nested lists, tables, special formatting). + +The parsed structure is stored as a JSON-serializable DocumentTree in SQLite, shared across all strategies. Structure extraction happens once per upload regardless of how many strategies are requested. + +--- + +# ADR 0007: Store Parsed Document in SQLite for Re-Processability + +After a document is uploaded, its parsed representation (extracted text, DocumentTree JSON, file metadata) is persisted in SQLite alongside experiment data. + +This enables re-running individual strategies with different parameters without re-uploading and re-parsing the document. For example: tuning SEMANTIC_THRESHOLD from 0.3 to 0.4 and re-running only the semantic strategy, reading the already-parsed document from SQLite. + +Without this, every parameter change would require re-processing the full pipeline from .docx through python-docx parsing — approximately 2-5 seconds of unnecessary work per re-run. + +SQLite was chosen over PostgreSQL for v1 because: single-user benchmarking tool, no concurrent access requirements, zero infrastructure setup, and the data volume is small (documents + experiments, not transactional data). + +--- + +# ADR 0008: Domain-Based Directory Structure + +The project uses a vertical-slice (domain-based) directory layout with four domains plus a core layer: + +``` +src/ +├── main.py +├── core/ # config, dependencies, exceptions, shared models +├── documents/ # upload, parsing, document tree +├── chunking/ # strategies, orchestration, embedding +├── storage/ # qdrant, sqlite +└── benchmarking/ # queries, benchmarks, evaluation, HTML reports +``` + +Each domain contains its own routes.py, service.py, and models.py. This means adding a new feature (e.g., a new endpoint) only touches one directory. + +The alternative — horizontal/layer-based (api/, services/, models/) — was rejected because adding a new feature requires touching multiple directories (api for routes, services for logic, models for schemas). This creates implicit coupling and makes it unclear where new code belongs. + +A core/ directory holds configuration (pydantic-settings), dependency injection (OpenAI/Qdrant singletons), shared models (Chunk, StrategyName), and exception handlers. All domains import upward to core; no domain imports from another domain. + +--- + +# ADR 0009: Synchronous REST Without Background Jobs + +All endpoints execute synchronously and return results directly. There is no Job model, no polling endpoint, and no async task queue. + +Measured timing for v1 scale: +- Document upload + parsing: < 1 second +- Single strategy processing (chunk + embed + store): 5-15 seconds +- All 5 strategies: 30-60 seconds +- Single query (retrieval + generation): 2-5 seconds +- Dataset benchmark (10 questions x 5 strategies): 2-5 minutes + +All of these are acceptable within an HTTP request lifecycle with a loading indicator. + +Background job tracking was considered (returns job_id, polls status via GET /jobs/{id}) but rejected because it adds: a Job model, a status state machine (started -> running -> completed/failed), a polling endpoint, and stale job cleanup logic — none of which are needed at v1 scale. + +If processing times grow beyond a few minutes (e.g., 100-question datasets with 20+ page documents), async can be introduced via a task queue (Celery or FastAPI BackgroundTasks) without changing the resource model — the POST endpoints would still accept the same request body and return a job_id, just with actual background execution. + +--- + +# ADR 0010: RESTful Resource Model + +The API exposes four primary resources following standard REST conventions: + +| Resource | Create | Read | Delete | Actions | +|----------|--------|------|--------|---------| +| Documents | POST /documents | — | DELETE /documents/{id} | POST /documents/{id}/process | +| Queries | POST /queries | GET /queries/{id} | — | — | +| Benchmarks | POST /benchmarks | GET /benchmarks/{id} | — | GET /benchmarks/{id}/report | +| Strategies | — | GET /strategies | — | — | + +The alternative — verb-based endpoints (POST /ask, POST /benchmark/run) — was rejected because it conflates actions with resources and breaks REST's noun-based resource model. + +Key naming decisions: +- "ask" is implemented as POST /queries (a Query is the resource, not the action) +- Dry run uses a query parameter: POST /benchmarks?dry_run=true (same resource, different mode) +- Process is a sub-action on documents, not a standalone endpoint: POST /documents/{id}/process +- Dataset vs single-question is determined by request body, not URL + +--- + +# ADR 0011: Embed Full Enriched Text in Contextual Strategy + +For the contextual_structure strategy, the full enriched text is embedded — including Document/Section/Article/Context/Content labels — not just the raw content. This is a deliberate design choice that affects what the benchmark measures. + +Enriched text example: +``` +Document: Insurance Regulation +Section: Claims and Compensation +Article: 15 +Context: This section explains compensation payment rules. +Content: The insurer must pay compensation within 15 days. +``` + +The whole enriched string is what gets sent to text-embedding-3-small. This means the contextual strategy's vectors encode structural metadata and generated context, not just the regulation text. The hypothesis being tested is: does adding contextual enrichment to the embedding improve retrieval quality compared to raw content embeddings? + +The alternative — embed only the Content field, store enriched version in payload for display only — was rejected because it would test whether enrichment helps at answer-generation time (when the LLM sees the enriched payload), but NOT whether it helps at retrieval time (when vector similarity determines which chunks are returned). Since retrieval is the primary bottleneck in RAG, the embedding must reflect the enrichment. + +--- + +# ADR 0012: Sentence-Level Semantic Chunking with Minimum Chunk Size + +The semantic strategy operates at sentence granularity, not paragraph granularity. Each sentence gets its own embedding, and cosine similarity between adjacent sentences determines chunk boundaries. + +Minimum chunk size is enforced: if a boundary is detected before the minimum is reached (configurable via SEMANTIC_MIN_CHUNK_SIZE, default 3 sentences), accumulation continues. The boundary sentence stays with the previous chunk, not the next. + +The alternatives considered were: + +1. **Paragraph-level semantic**: Embed each paragraph, cut when paragraph similarity drops. Simpler but less precise — a long paragraph with a mid-paragraph topic shift won't be caught. Chosen against because sentence-level is the canonical approach in the literature and provides the finest boundary detection. + +2. **Sentence-level without minimum size**: Allows single-sentence chunks. Rejected because degenerate tiny chunks hurt retrieval quality — a single sentence lacks enough context for the LLM to generate a meaningful answer. + +3. **Adaptive threshold**: Instead of a fixed cosine similarity cutoff, use a rolling average that adapts to the document's baseline similarity. More complex to implement and explain. Rejected for v1 because we're testing a single document — the fixed threshold is sufficient and the results are easier to interpret. + +The minimum chunk size is a direct consequence of sentence-level granularity — without it, the strategy would produce many unusable single-sentence chunks. + +--- + +# ADR 0013: Leaf-Level Markdown Chunks Without Overlap + +The markdown_structure strategy creates chunks only at the deepest header level (leaf chunks). If a document has: + +``` +## Claims (Section) +### Article 15 +### Article 16 +``` + +Chunks are created for Article 15 and Article 16. The "Claims" section itself is NOT a separate chunk — its content is fully contained within its child articles. + +The alternative — create chunks at every header level (both Claims and its children) — was rejected because it produces overlapping content. A chunk for "Claims" would contain the text of Articles 15 and 16, and those articles would also have their own chunks. This duplication means: +- The same text appears in multiple chunks, inflating chunk count +- Retrieval could return overlapping content, wasting top_k slots +- Benchmark metrics become less meaningful (more chunks retrieved doesn't mean better coverage) + +Leaf-level produces the cleanest, most non-overlapping chunk set for fair comparison with other strategies. + +--- + +# ADR 0014: Strict Unidirectional Import Rules + +All domain directories (documents/, chunking/, benchmarking/, storage/) import only from core/. No domain imports from another domain. This is enforced as a project convention. + +The dependency graph is strictly: + +``` +core/ <- imports nothing from the project +documents/ -> imports from core/ +chunking/ -> imports from core/ +storage/ -> imports from core/ +benchmarking/ -> imports from core/ +``` + +If benchmarking needs to reference a query or document, it does so through a shared model defined in core/models.py, not by importing chunking/models.py or documents/models.py. + +The alternative — allowing cross-domain imports (e.g., benchmarking imports from chunking to call strategy functions directly) — was rejected because it creates circular dependency risk and makes it impossible to test, replace, or refactor one domain without affecting others. + +This rule is simple enough to enforce manually in code review. If the project grows, a linter rule (import-linter or similar) can be added. + +--- + +# ADR 0015: Direct API Usage — No LangChain, No LlamaIndex + +The project uses OpenAI API, Qdrant SDK, and FastAPI directly. No abstraction frameworks (LangChain, LlamaIndex) are used. + +The rationale: +- **Control**: Every API call, every prompt, every embedding is explicit and inspectable. No hidden abstractions making decisions on your behalf. +- **Debuggability**: When a chunk is wrong or an answer is bad, you can trace the exact prompt and parameters. Framework wrappers often obscure the actual API call. +- **Fairness**: Using raw OpenAI SDK means all strategies go through identical API paths. Framework abstractions sometimes add caching, batching, or prompt modification that could bias results. +- **Learning**: For a benchmarking tool, understanding exactly what's happening at each layer is valuable. Frameworks hide the mechanics you're trying to measure. +- **Dependency reduction**: Fewer packages to install, fewer version conflicts, fewer breaking changes from upstream framework updates. + +The cost is more boilerplate code for common patterns (API calls, retries, error handling). For a benchmarking tool with well-defined, non-repetitive API usage, this cost is minimal. + +The ChunkingStrategy interface is the project's own abstraction — simple, explicit, and sufficient. It does the one thing frameworks do (strategy polymorphism) without the hundreds of things you don't need.