chunking_strategies_evaluation/CONTEXT.md

# Context — RAG Chunking Benchmarker Admin Dashboard

## Language

**Dashboard**:
A single-file React app (CDN-loaded, no build step) served by FastAPI at `/app`. Replaces Swagger as the primary UI for operating the RAG benchmarking platform.
_Avoid_: Admin panel, web UI, frontend

**Strategy**:
One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child. Semantic and semantic_parent_child determine chunk boundaries from meaning (adjacent-unit similarity), not from fixed sentence/paragraph counts.
_Avoid_: Method, approach, technique

**Semantic Boundary Detection**:
Using embeddings of consecutive units (sentences for semantic — Farsi-aware punctuation, with line/paragraph fallback when punctuation yields a single unit; paragraphs for semantic_parent_child) and cosine similarity against a threshold tied to the Boundary Embedding Model to decide where chunks start and end. Distinct from Corpus Embedding (finished-chunk vectors for Qdrant). If boundary embeddings are missing or mismatched, that Strategy fails — there is no fixed-count fallback.
_Avoid_: Semantic embedding, chunk embedding (ambiguous — that often means storage vectors)

**Experiment**:
A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics. Always records Corpus Embedding Model and Neighbor Expansion knobs. Records Boundary Embedding Model when semantic Strategies were part of processing/evaluation path for that corpus. Runs under different Boundary/Corpus settings must not be silently compared.
_Avoid_: Run, trial, benchmark

**Chunk Preview**:
The ability to visualize what a specific strategy produces for a given document — the raw text blocks, their sizes, and hierarchy.
_Avoid_: Chunk inspection, chunk view

**Tab**:
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
_Avoid_: Page, route, view

**Retrieval Inspect**:
A full-page mode inside the Benchmarks Tab for auditing one Experiment’s retrieval: left rail lists questions (status for error / has Expansion Tree); main pane has Strategy picker (defaults to `fixed_size` when present), a side-by-side Generated Answer | Expected Answer strip with eval scores, and a vertical full-text Expansion Tree (previous → hit → next per top-k hit; neighbors empty/N/A for other Strategies). Opened from an Experiment; not a separate top-level Tab. Distinct from live Query and from truncated HTML report Expansion Tree blocks.
_Avoid_: Query tab, chunk preview, report tree, expansion sample

**Expected Answer**:
The reference answer for a benchmark question, supplied with the question set and used by evaluation (e.g. Answer Similarity). Shown in Retrieval Inspect beside the Generated Answer. Informal synonym “golden answer” is not product language.
_Avoid_: Golden answer, ground truth, reference answer (prefer Expected Answer in UI and docs)

**PDF Workspace**:
The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
_Avoid_: PDF page, PDF mode, PDF dashboard

**Text PDF**:
A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1.
_Avoid_: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk)

**Scanned PDF**:
A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1.
_Avoid_: Image PDF, photo PDF, OCR PDF

**Heading Reconstruction**:
Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown `#` / `##` for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present.
_Avoid_: Heading detection, structure recovery, outline parsing (too narrow)

**Text-layer Gate**:
Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus.
_Avoid_: OCR check, PDF validation, empty-page filter

**Table Flattening**:
Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
_Avoid_: Table extraction, table parsing, structured tables

**Embedding Model**:
A registered model identity (Provider, dimension, optional task prefixes, default `semantic_threshold`). Used in one of two roles: Boundary or Corpus. Not an Experiment axis by itself — Strategies are compared under recorded Boundary+Corpus snapshots per Experiment.
_Avoid_: Embedder, embedding provider (use Provider for that), vector model

**Active Embedding Model**:
Legacy single global role (ADR-0019). Replaced by two Admin defaults — Default Boundary Embedding Model and Default Corpus Embedding Model — with per-operation overrides. Prefer Boundary / Corpus terms for new work.
_Avoid_: Current model, selected embedder, runtime model

**Boundary Embedding Model**:
The Embedding Model used only for Semantic Boundary Detection (`semantic` / `semantic_parent_child`). Does not write Qdrant vectors. Ignored when a Strategy has no boundary step. Admin stores a default; Process may override per run. Snapshotted at process start. Cuts always use this model’s `semantic_threshold` (never the Corpus model’s threshold when the two differ).
_Avoid_: Semantic embedder, chunking model, similarity model

**Corpus Embedding Model**:
The Embedding Model used to embed finished chunks into the Model Corpus and to embed queries for search. Storage and query always share this model. Determines which Model Corpus collections are read/written. Admin stores a default; Process, Query, and Experiment may override per run. Snapshotted for that operation.
_Avoid_: Storage model, retrieval model, Qdrant model, Active Embedding Model

**Provider**:
Where an Embedding Model runs: Cloud (e.g. OpenAI API) or Local (Ollama on the host). A Provider is a delivery channel, not a model identity.
_Avoid_: Backend, engine, embedding source

**Model Corpus**:
The Strategy collections whose vectors were produced by one Corpus Embedding Model. Collection names always include that model’s id (`{strategy}__{model_id}_collection`). Process, query, and Experiment only read/write the Model Corpus of the Corpus Embedding Model in force — other corpora remain untouched.
_Avoid_: Collection set, vector partition, embedding space

**Embedding Model Registry**:
The configured catalog of Embedding Models the operator may assign to Boundary or Corpus roles — each entry has a stable identity, Provider, vector dimension, and a default `semantic_threshold` (Admin may override per model in SQLite). Defaults: OpenAI 0.3, Nomic 0.6. Unlisted models are out of scope until added to the registry.
_Avoid_: Model list, provider catalog, available embeddings

**Neighbor Expansion**:
Query-time widening of retrieved context for the `fixed_size` Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (`chunk_index`). Neighbors are not re-ranked as independent hits. Counts (`neighbor_prev` / `neighbor_next`) are set per Query or Experiment like `top_k`; defaults are `0`/`0` (off). Missing neighbors at document edges are skipped. Knobs are ignored for non-`fixed_size` Strategies. Dashboard labels Experiments with a compact `±P/N` badge (tooltip explains prev/next; muted when `fixed_size` was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time.
_Avoid_: Context windowing, chunk padding, sliding window retrieval, adjacent chunk merge

**Expansion Tree**:
The operator-facing grouping of Neighbor Expansion results, returned alongside the flat `retrieved_chunks` list: top-k hits in score order, each listing previous and next chunks appended for that hit (overlaps may appear under more than one hit). When expansion is off, the tree still lists the hits with empty neighbor arrays. Shown in the Query result UI, Experiment per-question detail, and both managerial and technical HTML report views. Distinct from the flat, deduped, document-ordered context sent to the LLM (each chunk id appears only once there).
_Avoid_: Retrieved chunks list, neighbor list, expansion map

**Benchmark Sweep**:
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix

## Architecture Decisions

| # | Decision | Status |
|---|----------|--------|
 ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
 ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
 ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved |
 ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
 ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
 ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
 ADR-0007 | Admin router v1 endpoints: /admin/health, /admin/qdrant/collections (CRUD + wipe), /admin/chunks/{doc_id}, /admin/questions (CRUD + upload), /admin/cost-estimate. Config/SQLite/request-log deferred to v1.1. | Approved |
 ADR-0008 | Custom dark theme: bg #111113/#1a1a1e/#222228, border #2a2a30, text #e4e4e7/#fafafa/#71717a, accent #eab308 (amber). Inter font from Google Fonts CDN. Continuity with existing amber/green report palette. | Approved |
 ADR-0009 | File structure: dashboard HTML in src/static/index.html, admin backend in src/admin/ (routes.py, service.py). FastAPI mounts /app -> src/static via StaticFiles(html=True). | Approved |
 ADR-0010 | Cant-do-yet implementation order: Admin panels first (chunk preview, questions mgmt, cost estimator), then cross-tab UX (experiment detail, query history, doc preview, progress), then cross-cutting (error handling). | Approved |
 ADR-0011 | Admin tab uses collapsible sections (accordion pattern). Each panel is a card with a clickable header toggling display. Health starts expanded, others collapsed by default. | Approved |
 ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved |
 ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved |
 ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
 ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
 ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |
 ADR-0018 | Model Corpus via model-scoped collections; legacy unscoped → default cloud corpus. See docs/adr/0018-*.md | Superseded by ADR-0021 (naming) |
 ADR-0019 | Active Embedding Model is global (Admin + registry), not an Experiment axis; provenance required. See docs/adr/0019-*.md | Superseded by ADR-0024 |
 ADR-0020 | Semantic strategies require Semantic Boundary Detection (fail hard, no fixed-count fallback). See docs/adr/0020-*.md | Approved |
 ADR-0021 | Always scope Qdrant collection names by Embedding Model id (cloud and local). See docs/adr/0021-*.md | Approved |
 ADR-0022 | Per-Embedding-Model `semantic_threshold` (Admin override; defaults OpenAI 0.3 / Nomic 0.6). See docs/adr/0022-*.md | Approved |
 ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
 ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
 ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |