chunking_strategies_evaluation/CONTEXT.md

Context — RAG Chunking Benchmarker Admin Dashboard

Language

Dashboard: A single-file React app (CDN-loaded, no build step) served by FastAPI at /app. Replaces Swagger as the primary UI for operating the RAG benchmarking platform. Avoid: Admin panel, web UI, frontend

Strategy: One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child. Semantic and semantic_parent_child determine chunk boundaries from meaning (adjacent-unit similarity), not from fixed sentence/paragraph counts. Avoid: Method, approach, technique

Semantic Boundary Detection: Using embeddings of consecutive units (sentences for semantic — Farsi-aware punctuation, with line/paragraph fallback when punctuation yields a single unit; paragraphs for semantic_parent_child) and cosine similarity against a threshold tied to the Boundary Embedding Model to decide where chunks start and end. Distinct from Corpus Embedding (finished-chunk vectors for Qdrant). If boundary embeddings are missing or mismatched, that Strategy fails — there is no fixed-count fallback. Avoid: Semantic embedding, chunk embedding (ambiguous — that often means storage vectors)

Experiment: A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics. Always records Corpus Embedding Model and Neighbor Expansion knobs. Records Boundary Embedding Model when semantic Strategies were part of processing/evaluation path for that corpus. Runs under different Boundary/Corpus settings must not be silently compared. Avoid: Run, trial, benchmark

Chunk Preview: The ability to visualize what a specific strategy produces for a given document — the raw text blocks, their sizes, and hierarchy. Avoid: Chunk inspection, chunk view

Tab: A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives. Avoid: Page, route, view

Retrieval Inspect: A full-page mode inside the Benchmarks Tab for auditing one Experiment’s retrieval: left rail lists questions (status for error / has Expansion Tree); main pane has Strategy picker (defaults to fixed_size when present), a side-by-side Generated Answer | Expected Answer strip with eval scores, and a vertical full-text Expansion Tree (previous → hit → next per top-k hit; neighbors empty/N/A for other Strategies). Opened from an Experiment; not a separate top-level Tab. Distinct from live Query and from truncated HTML report Expansion Tree blocks. Avoid: Query tab, chunk preview, report tree, expansion sample

Expected Answer: The reference answer for a benchmark question, supplied with the question set and used by evaluation (e.g. Answer Similarity). Shown in Retrieval Inspect beside the Generated Answer. Informal synonym “golden answer” is not product language. Avoid: Golden answer, ground truth, reference answer (prefer Expected Answer in UI and docs)

PDF Workspace: The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (.doc/.docx). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI. Avoid: PDF page, PDF mode, PDF dashboard

Text PDF: A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1. Avoid: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk)

Scanned PDF: A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1. Avoid: Image PDF, photo PDF, OCR PDF

Heading Reconstruction: Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown # / ## for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present. Avoid: Heading detection, structure recovery, outline parsing (too narrow)

Text-layer Gate: Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus. Avoid: OCR check, PDF validation, empty-page filter

Table Flattening: Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1. Avoid: Table extraction, table parsing, structured tables

Embedding Model: A registered model identity (Provider, dimension, optional task prefixes, default semantic_threshold). Used in one of two roles: Boundary or Corpus. Not an Experiment axis by itself — Strategies are compared under recorded Boundary+Corpus snapshots per Experiment. Avoid: Embedder, embedding provider (use Provider for that), vector model

Active Embedding Model: Legacy single global role (ADR-0019). Replaced by two Admin defaults — Default Boundary Embedding Model and Default Corpus Embedding Model — with per-operation overrides. Prefer Boundary / Corpus terms for new work. Avoid: Current model, selected embedder, runtime model

Boundary Embedding Model: The Embedding Model used only for Semantic Boundary Detection (semantic / semantic_parent_child). Does not write Qdrant vectors. Ignored when a Strategy has no boundary step. Admin stores a default; Process may override per run. Snapshotted at process start. Cuts always use this model’s semantic_threshold (never the Corpus model’s threshold when the two differ). Avoid: Semantic embedder, chunking model, similarity model

Corpus Embedding Model: The Embedding Model used to embed finished chunks into the Model Corpus and to embed queries for search. Storage and query always share this model. Determines which Model Corpus collections are read/written. Admin stores a default; Process, Query, and Experiment may override per run. Snapshotted for that operation. Avoid: Storage model, retrieval model, Qdrant model, Active Embedding Model

Provider: Where an Embedding Model runs: Cloud (e.g. OpenAI API) or Local (Ollama on the host). A Provider is a delivery channel, not a model identity. Avoid: Backend, engine, embedding source

Model Corpus: The Strategy collections whose vectors were produced by one Corpus Embedding Model. Collection names always include that model’s id ({strategy}__{model_id}_collection). Process, query, and Experiment only read/write the Model Corpus of the Corpus Embedding Model in force — other corpora remain untouched. Avoid: Collection set, vector partition, embedding space

Embedding Model Registry: The configured catalog of Embedding Models the operator may assign to Boundary or Corpus roles — each entry has a stable identity, Provider, vector dimension, and a default semantic_threshold (Admin may override per model in SQLite). Defaults: OpenAI 0.3, Nomic 0.6. Unlisted models are out of scope until added to the registry. Avoid: Model list, provider catalog, available embeddings

Neighbor Expansion: Query-time widening of retrieved context for the fixed_size Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (chunk_index). Neighbors are not re-ranked as independent hits. Counts (neighbor_prev / neighbor_next) are set per Query or Experiment like top_k; defaults are 0/0 (off). Missing neighbors at document edges are skipped. Knobs are ignored for non-fixed_size Strategies. Dashboard labels Experiments with a compact ±P/N badge (tooltip explains prev/next; muted when fixed_size was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time. Avoid: Context windowing, chunk padding, sliding window retrieval, adjacent chunk merge

Expansion Tree: The operator-facing grouping of Neighbor Expansion results, returned alongside the flat retrieved_chunks list: top-k hits in score order, each listing previous and next chunks appended for that hit (overlaps may appear under more than one hit). When expansion is off, the tree still lists the hits with empty neighbor arrays. Shown in the Query result UI, Experiment per-question detail, and both managerial and technical HTML report views. Distinct from the flat, deduped, document-ordered context sent to the LLM (each chunk id appears only once there). Avoid: Retrieved chunks list, neighbor list, expansion map

Benchmark Sweep: An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. (neighbor_prev, neighbor_next) = (0,0), (1,1), (2,2), (3,3) — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row. Avoid: Batch run, sequential benchmark, benchmark script, neighbor matrix

Strategy Candidate: A single comparable configuration for final Strategy selection under one Corpus Embedding Model: either fixed_size at one Neighbor Expansion level (±N), or semantic under one Boundary Embedding Model. Distinct from an Experiment (an Experiment is one doc’s run of one Candidate, or of several Strategies). Final selection compares Candidates, not raw multi-strategy Experiments. The Decision Board auto-discovers Candidates from single-strategy Experiments (newest per doc×Candidate cell), filtered by Corpus Embedding Model, with optional exclude of a bad Experiment. Avoid: Variant, configuration, setup, arm

Decision Board: A top-level Dashboard Tab for choosing between Strategy families after tuning: stage 1 picks the best Strategy Candidate within fixed_size and within semantic; stage 2 compares those two winners. Always accompanied by a per-document breakdown so aggregate rank cannot hide doc-level disagreement. Document universe for v1 is the fixed 10-doc evaluation set (same as the Neighbor Expansion Benchmark Sweep). Distinct from Compare (ad-hoc multi-Experiment side-by-side) and from Retrieval Inspect. Avoid: Final compare, strategy picker, results page, leaderboard, Decision mode

Architecture Decisions

#	Decision	Status
ADR-0001	Single-file React via CDN (no build step), served by FastAPI	Approved
ADR-0002	New /admin/ router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost)	Approved
ADR-0003	Top-tab navigation (Home, Documents, Query, Benchmarks, Decision, Admin) — not sidebar	Approved
ADR-0004	Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit.	Approved
ADR-0005	Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches.	Approved
ADR-0006	Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication.	Approved
ADR-0007	Admin router v1 endpoints: /admin/health, /admin/qdrant/collections (CRUD + wipe), /admin/chunks/{doc_id}, /admin/questions (CRUD + upload), /admin/cost-estimate. Config/SQLite/request-log deferred to v1.1.	Approved
ADR-0008	Custom dark theme: bg #111113/#1a1a1e/#222228, border #2a2a30, text #e4e4e7/#fafafa/#71717a, accent #eab308 (amber). Inter font from Google Fonts CDN. Continuity with existing amber/green report palette.	Approved
ADR-0009	File structure: dashboard HTML in src/static/index.html, admin backend in src/admin/ (routes.py, service.py). FastAPI mounts /app -> src/static via StaticFiles(html=True).	Approved
ADR-0010	Cant-do-yet implementation order: Admin panels first (chunk preview, questions mgmt, cost estimator), then cross-tab UX (experiment detail, query history, doc preview, progress), then cross-cutting (error handling).	Approved
ADR-0011	Admin tab uses collapsible sections (accordion pattern). Each panel is a card with a clickable header toggling display. Health starts expanded, others collapsed by default.	Approved
ADR-0012	Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default.	Approved
ADR-0013	Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations.	Approved
ADR-0014	Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables.	Approved
ADR-0016	Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md	Approved
ADR-0017	PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md	Approved
ADR-0018	Model Corpus via model-scoped collections; legacy unscoped → default cloud corpus. See docs/adr/0018-*.md	Superseded by ADR-0021 (naming)
ADR-0019	Active Embedding Model is global (Admin + registry), not an Experiment axis; provenance required. See docs/adr/0019-*.md	Superseded by ADR-0024
ADR-0020	Semantic strategies require Semantic Boundary Detection (fail hard, no fixed-count fallback). See docs/adr/0020-*.md	Approved
ADR-0021	Always scope Qdrant collection names by Embedding Model id (cloud and local). See docs/adr/0021-*.md	Approved
ADR-0022	Per-Embedding-Model semantic_threshold (Admin override; defaults OpenAI 0.3 / Nomic 0.6). See docs/adr/0022-*.md	Approved
ADR-0023	Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare ±P/N provenance + mismatch warning). See docs/adr/0023-*.md	Approved
ADR-0024	Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md	Implemented
ADR-0025	Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md	Approved
ADR-0026	Decision Board Tab: two-stage Strategy Candidate selection (fixed_size ±N vs semantic@Boundary) over the 10-doc set. See docs/adr/0026-*.md	Approved