docs: record embedding, neighbor, and inspect decisions

Co-authored-by: Cursor <cursoragent@cursor.com>

CONTEXT.md

@@ -7,11 +7,15 @@ A single-file React app (CDN-loaded, no build step) served by FastAPI at `/app`.
 _Avoid_: Admin panel, web UI, frontend
 
 **Strategy**:
-One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child.
+One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child. Semantic and semantic_parent_child determine chunk boundaries from meaning (adjacent-unit similarity), not from fixed sentence/paragraph counts.
 _Avoid_: Method, approach, technique
 
+**Semantic Boundary Detection**:
+Using embeddings of consecutive units (sentences for semantic — Farsi-aware punctuation, with line/paragraph fallback when punctuation yields a single unit; paragraphs for semantic_parent_child) and cosine similarity against a threshold tied to the Boundary Embedding Model to decide where chunks start and end. Distinct from Corpus Embedding (finished-chunk vectors for Qdrant). If boundary embeddings are missing or mismatched, that Strategy fails — there is no fixed-count fallback.
+_Avoid_: Semantic embedding, chunk embedding (ambiguous — that often means storage vectors)
+
 **Experiment**:
-A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics.
+A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics. Always records Corpus Embedding Model and Neighbor Expansion knobs. Records Boundary Embedding Model when semantic Strategies were part of processing/evaluation path for that corpus. Runs under different Boundary/Corpus settings must not be silently compared.
 _Avoid_: Run, trial, benchmark
 
 **Chunk Preview**:
@@ -22,6 +26,14 @@ _Avoid_: Chunk inspection, chunk view
 A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
 _Avoid_: Page, route, view
 
+**Retrieval Inspect**:
+A full-page mode inside the Benchmarks Tab for auditing one Experiment’s retrieval: left rail lists questions (status for error / has Expansion Tree); main pane has Strategy picker (defaults to `fixed_size` when present), a side-by-side Generated Answer | Expected Answer strip with eval scores, and a vertical full-text Expansion Tree (previous → hit → next per top-k hit; neighbors empty/N/A for other Strategies). Opened from an Experiment; not a separate top-level Tab. Distinct from live Query and from truncated HTML report Expansion Tree blocks.
+_Avoid_: Query tab, chunk preview, report tree, expansion sample
+
+**Expected Answer**:
+The reference answer for a benchmark question, supplied with the question set and used by evaluation (e.g. Answer Similarity). Shown in Retrieval Inspect beside the Generated Answer. Informal synonym “golden answer” is not product language.
+_Avoid_: Golden answer, ground truth, reference answer (prefer Expected Answer in UI and docs)
+
 **PDF Workspace**:
 The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
 _Avoid_: PDF page, PDF mode, PDF dashboard
@@ -46,6 +58,46 @@ _Avoid_: OCR check, PDF validation, empty-page filter
 Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
 _Avoid_: Table extraction, table parsing, structured tables
 
+**Embedding Model**:
+A registered model identity (Provider, dimension, optional task prefixes, default `semantic_threshold`). Used in one of two roles: Boundary or Corpus. Not an Experiment axis by itself — Strategies are compared under recorded Boundary+Corpus snapshots per Experiment.
+_Avoid_: Embedder, embedding provider (use Provider for that), vector model
+
+**Active Embedding Model**:
+Legacy single global role (ADR-0019). Replaced by two Admin defaults — Default Boundary Embedding Model and Default Corpus Embedding Model — with per-operation overrides. Prefer Boundary / Corpus terms for new work.
+_Avoid_: Current model, selected embedder, runtime model
+
+**Boundary Embedding Model**:
+The Embedding Model used only for Semantic Boundary Detection (`semantic` / `semantic_parent_child`). Does not write Qdrant vectors. Ignored when a Strategy has no boundary step. Admin stores a default; Process may override per run. Snapshotted at process start. Cuts always use this model’s `semantic_threshold` (never the Corpus model’s threshold when the two differ).
+_Avoid_: Semantic embedder, chunking model, similarity model
+
+**Corpus Embedding Model**:
+The Embedding Model used to embed finished chunks into the Model Corpus and to embed queries for search. Storage and query always share this model. Determines which Model Corpus collections are read/written. Admin stores a default; Process, Query, and Experiment may override per run. Snapshotted for that operation.
+_Avoid_: Storage model, retrieval model, Qdrant model, Active Embedding Model
+
+**Provider**:
+Where an Embedding Model runs: Cloud (e.g. OpenAI API) or Local (Ollama on the host). A Provider is a delivery channel, not a model identity.
+_Avoid_: Backend, engine, embedding source
+
+**Model Corpus**:
+The Strategy collections whose vectors were produced by one Corpus Embedding Model. Collection names always include that model’s id (`{strategy}__{model_id}_collection`). Process, query, and Experiment only read/write the Model Corpus of the Corpus Embedding Model in force — other corpora remain untouched.
+_Avoid_: Collection set, vector partition, embedding space
+
+**Embedding Model Registry**:
+The configured catalog of Embedding Models the operator may assign to Boundary or Corpus roles — each entry has a stable identity, Provider, vector dimension, and a default `semantic_threshold` (Admin may override per model in SQLite). Defaults: OpenAI 0.3, Nomic 0.6. Unlisted models are out of scope until added to the registry.
+_Avoid_: Model list, provider catalog, available embeddings
+
+**Neighbor Expansion**:
+Query-time widening of retrieved context for the `fixed_size` Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (`chunk_index`). Neighbors are not re-ranked as independent hits. Counts (`neighbor_prev` / `neighbor_next`) are set per Query or Experiment like `top_k`; defaults are `0`/`0` (off). Missing neighbors at document edges are skipped. Knobs are ignored for non-`fixed_size` Strategies. Dashboard labels Experiments with a compact `±P/N` badge (tooltip explains prev/next; muted when `fixed_size` was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time.
+_Avoid_: Context windowing, chunk padding, sliding window retrieval, adjacent chunk merge
+
+**Expansion Tree**:
+The operator-facing grouping of Neighbor Expansion results, returned alongside the flat `retrieved_chunks` list: top-k hits in score order, each listing previous and next chunks appended for that hit (overlaps may appear under more than one hit). When expansion is off, the tree still lists the hits with empty neighbor arrays. Shown in the Query result UI, Experiment per-question detail, and both managerial and technical HTML report views. Distinct from the flat, deduped, document-ordered context sent to the LLM (each chunk id appears only once there).
+_Avoid_: Retrieved chunks list, neighbor list, expansion map
+
+**Benchmark Sweep**:
+An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
+_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
+
 ## Architecture Decisions
 
 | # | Decision | Status |
@@ -66,3 +118,11 @@ _Avoid_: Table extraction, table parsing, structured tables
  ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
  ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
  ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |
+ ADR-0018 | Model Corpus via model-scoped collections; legacy unscoped → default cloud corpus. See docs/adr/0018-*.md | Superseded by ADR-0021 (naming) |
+ ADR-0019 | Active Embedding Model is global (Admin + registry), not an Experiment axis; provenance required. See docs/adr/0019-*.md | Superseded by ADR-0024 |
+ ADR-0020 | Semantic strategies require Semantic Boundary Detection (fail hard, no fixed-count fallback). See docs/adr/0020-*.md | Approved |
+ ADR-0021 | Always scope Qdrant collection names by Embedding Model id (cloud and local). See docs/adr/0021-*.md | Approved |
+ ADR-0022 | Per-Embedding-Model `semantic_threshold` (Admin override; defaults OpenAI 0.3 / Nomic 0.6). See docs/adr/0022-*.md | Approved |
+ ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
+ ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
+ ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |