docs: record embedding, neighbor, and inspect decisions

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/adr/0018-model-corpus-scoped-collections.md

@@ -0,0 +1,17 @@
+# Model Corpus via model-scoped collections
+
+Vectors for Cloud and Local Embedding Models must not share a Qdrant collection: dimensions and embedding spaces differ. Each Embedding Model therefore owns a **Model Corpus** — Strategy collections whose identity includes that model. Process, query, and Experiment only read/write the Active Embedding Model’s corpus; other corpora stay untouched. Pre-existing unscoped collections (e.g. `fixed_size_collection`) are migrated into the default cloud Embedding Model’s corpus so existing OpenAI work is not discarded.
+
+## Considered Options
+
+- **Model-scoped collections** — chosen; non-destructive switch between Providers/models; clear isolation
+- **Wipe-and-rebuild on switch** — simple naming, destructive and easy to forget
+- **Dimension-gated single set** — only allow same-dimension swaps; blocks most OpenAI ↔ Ollama use
+
+Legacy handling (original): **migrate unscoped → default cloud corpus**. **Superseded for naming by ADR-0021** — all models, including default cloud, use `{strategy}__{model_id}_collection`; delete leftover unscoped collections when rebuilding.
+
+## Consequences
+
+- Collection naming and vector size are driven by the Embedding Model Registry entry, not a global `1536` constant
+- Admin Qdrant list shows all corpora, labeled by Embedding Model, with the active corpus emphasized; wipe stays per named collection
+- Switching Active Embedding Model does not delete the other Model Corpus; operator must process documents again under the new model to populate it

docs/adr/0019-active-embedding-model-global.md

@@ -0,0 +1,22 @@
+# Active Embedding Model is global, not an Experiment axis
+
+**Status:** Superseded by [ADR-0024](0024-boundary-vs-corpus-embedding-models.md) — single Active Embedding Model replaced by **Boundary** + **Corpus** roles (query locked to Corpus).
+
+Original decision retained for history:
+
+The platform compares **Strategies** under controlled conditions. Embedding Model is a confounder, not a second experiment dimension: one **Active Embedding Model** applies to process, query, and new Experiments. The operator switches it from Admin among entries in a static **Embedding Model Registry** (Provider, stable id, vector dimension). Selection persists across restarts (config supplies the default only when unset). Every Experiment records which Embedding Model produced it; historical rows without provenance are treated as the default cloud model. A process, query, or Experiment **snapshots** the Active Embedding Model at start so a mid-flight Admin switch cannot mix models inside one operation.
+
+## Considered Options
+
+- **Global Active Embedding Model** — chosen; fair Strategy comparisons; A/B models via separate Experiments
+- **Embedding Model as Experiment axis** — Strategy × model cross-product; richer science, much heavier data model/UI
+- **Per-run choice with no Experiment coupling** — flexible, invites silent unfair comparisons
+
+Switcher: **Admin UI** (not env-only, not read-only status). Catalog: **static registry** (not live Ollama discovery, not free-form). Local Provider host: **`OLLAMA_BASE_URL` in config only**. Cost Estimator: **$0 embedding line when Active Embedding Model is Local**; LLM costs unchanged. This carve-out does not reopen general `/admin/config` (ADR-0007 deferred).
+
+## Consequences
+
+- Introduce Provider adapters (Cloud OpenAI, Local Ollama) behind one embed API; callers bind a registry entry for the operation
+- Dashboard Admin gains an Embedding Model switcher; Benchmarks/reports must show Experiment provenance
+- `docs/out-of-scope-v1.md` “no embedding model from dashboard” is superseded for this focused control only
+- Local Nomic registry entries use task prefixes (`search_document` / `search_query`); Cloud OpenAI entries do not

docs/adr/0020-semantic-boundary-detection-required.md

@@ -0,0 +1,16 @@
+# Semantic strategies require Semantic Boundary Detection
+
+`semantic` and `semantic_parent_child` must cut chunks from meaning: the orchestrator embeds consecutive units (sentences / paragraphs) with the Active Embedding Model snapshot, passes those vectors into `chunk()`, then embeds finished chunks for Qdrant. Fixed-count fallbacks are removed — missing or length-mismatched boundary embeddings fail that Strategy. This completes ADR 0012’s sentence-level design in the process path (and the parent/child analogue) so benchmarks cannot silently measure “every N units” under a semantic name.
+
+## Considered Options
+
+- **Wire both strategies + fail hard** — chosen; same Active model for boundaries and storage; accept double embed cost; re-process to replace old fake-semantic vectors
+- **Wire `semantic` only** — rejected; would leave parent/child on the same lie
+- **Keep fallback with warnings** — rejected; that is how the bug stayed hidden
+- **Always OpenAI for boundaries regardless of Active** — rejected; confounds model A/B experiments
+
+## Consequences
+
+- Process cost rises for these two Strategies (unit embeds + chunk embeds)
+- Prior Experiments / collections produced under the fallback are not true semantic — operator must re-process and re-benchmark
+- Strategy modules raise `ChunkingError` if boundary embeddings are absent or mismatched; orchestration is responsible for supplying them

docs/adr/0021-always-scope-collection-names-by-model.md

@@ -0,0 +1,14 @@
+# Always scope Qdrant collection names by Embedding Model id
+
+Every Model Corpus collection is named `{strategy}__{embedding_model_id}_collection`, including the default cloud model (`text-embedding-3-small`). This supersedes ADR-0018’s exception that left OpenAI on legacy unscoped names (`fixed_size_collection`). Uniform naming makes Admin labeling obvious for both Cloud and Local and avoids a special case that confused operators wiping/rebuilding corpora.
+
+## Considered Options
+
+- **Always include model id** — chosen; clear for cloud and local
+- **Keep legacy unscoped names for default cloud only** — ADR-0018; rejected going forward after a from-scratch rebuild
+
+## Consequences
+
+- Process/query create names like `fixed_size__text-embedding-3-small_collection` and `fixed_size__nomic-embed-text-v2-moe_collection`
+- Old unscoped collections are not written anymore; Admin still labels them as legacy default-cloud if they remain until deleted
+- Operators wiping for a clean slate should delete legacy collections before re-processing

docs/adr/0022-per-embedding-model-semantic-threshold.md

@@ -0,0 +1,22 @@
+# ADR-0022: Per-Embedding-Model semantic_threshold
+
+## Status
+
+Approved
+
+## Context
+
+Semantic Boundary Detection cuts when adjacent-unit cosine similarity falls below a threshold. Cloud OpenAI and local Nomic produce different similarity distributions for the same Farsi text: the global `SEMANTIC_THRESHOLD` (config, historically ~0.3) rarely triggers cuts under Nomic, collapsing `semantic` into a single chunk. Operators need a higher Nomic default without changing OpenAI behavior, and a way to tune without editing `.env` and restarting.
+
+## Decision
+
+- Each Embedding Model Registry entry has a `default_semantic_threshold` (OpenAI `text-embedding-3-small`: **0.3**; Nomic `nomic-embed-text-v2-moe`: **0.6`).
+- Admin may override the effective value per model id in SQLite (`semantic_threshold:{model_id}`).
+- Process snapshots the Active Embedding Model and uses that model’s effective threshold for `semantic` and `semantic_parent_child`.
+- Admin API: list includes `semantic_threshold` / `default_semantic_threshold`; `PUT /admin/embedding-models/{id}/semantic-threshold` persists overrides.
+- Changing the threshold does not rewrite existing corpora — re-process to apply.
+
+## Consequences
+
+- Global `SEMANTIC_THRESHOLD` remains a fallback only when a strategy is called without an explicit threshold (tests/legacy); production process path always passes the model-resolved value.
+- Operators must re-process after tuning; Experiments under different thresholds are not auto-invalidated.

docs/adr/0023-neighbor-expansion-fixed-size.md

@@ -0,0 +1,46 @@
+# ADR-0023: Neighbor Expansion for fixed_size retrieval
+
+## Status
+
+Accepted (implemented)
+
+## Context
+
+Fixed-size chunks cut text at token windows, so the single highest-scoring chunk often lacks the sentence before/after the answer. Operators want query-time widening: after top-k retrieval, also pull previous and next chunks in document order. Doing this for every Strategy, or reshaping the top-k budget, would muddy Experiment comparisons and hide whether gains come from chunking vs from extra context.
+
+Operators also need to **see** the relationship: all top-k hits and, for each hit, which upper/downer chunks were appended. A flat deduped list alone cannot show per-hit windows when neighbors overlap.
+
+## Decision
+
+### Retrieval behavior
+
+- **Neighbor Expansion** applies only to the `fixed_size` Strategy at query time (not at process/chunking time).
+- Every top-k hit is expanded by configurable `neighbor_prev` and `neighbor_next` counts (same document, by `chunk_index`).
+- Expansion is **additive**: neighbors are merged with top-k, deduped by chunk id, then sorted into **document order** for the LLM. Context may exceed `top_k`.
+- Knobs live on Query and Experiment requests like `top_k`, with config defaults **`0` / `0`** (opt-in). Non-`fixed_size` Strategies ignore the knobs. Missing neighbors at document edges are skipped.
+- Experiments must record `neighbor_prev` / `neighbor_next` for provenance.
+
+### Dual response shape
+
+- **`retrieved_chunks`**: flat list matching what the LLM saw (deduped, document order; entries labeled hit vs neighbor as needed for audit/eval).
+- **`expansion_tree`**: operator-facing grouping — top-k hits in **score order**, each with `neighbors_prev` / `neighbors_next`. When expansion is off (`0`/`0`), the tree still contains the hits with **empty** neighbor arrays.
+- Overlapping windows: the same chunk may appear under **more than one** hit in the Expansion Tree; it still appears **once** in `retrieved_chunks` / the LLM prompt.
+
+### Where the Expansion Tree is shown
+
+- Query result UI
+- Experiment per-question detail in the Dashboard
+- HTML report — **both** managerial and technical views
+
+### Experiment list & Compare provenance
+
+- Experiments list shows a compact **`±P/N`** Neighbors badge (tooltip: `neighbor_prev` / `neighbor_next`; muted when `fixed_size` was not in the run).
+- Compare view repeats the badge on each Experiment column/card.
+- Compare shows a **soft warning** when selected Experiments differ on Neighbor Expansion **or** Embedding Model — intentional A/B is allowed; do not treat mismatched fixed_size scores as identical setups.
+
+## Consequences
+
+- Fixed_size Experiments with expansion enabled may use more tokens than other Strategies at the same `top_k`; that asymmetry is intentional and must be visible in config/reports.
+- Hit-only retrieval metrics remain available by filtering labeled hits in the flat list; unfiltered eval reflects the expanded prompt.
+- UI and reports render the Expansion Tree; they must not assume the flat list alone can reconstruct per-hit appendages after dedupe.
+- Dashboard Compare must keep Neighbor Expansion (and Embedding Model) visible so operators can fairly compare `±0/0` vs `±1/1` vs larger windows.

docs/adr/0024-boundary-vs-corpus-embedding-models.md

@@ -0,0 +1,27 @@
+# ADR-0024: Boundary vs Corpus Embedding Model roles
+
+## Status
+
+Implemented. Supersedes ADR-0019’s single global Active Embedding Model for new work.
+
+## Context
+
+Embedding happens in three moments: (1) Semantic Boundary Detection for `semantic` / `semantic_parent_child`, (2) finished-chunk vectors into Qdrant, (3) query vectors for search. ADR-0019 forced one Active Embedding Model for all three so Strategy comparisons stayed fair. Operators now want to vary the model used for cuts independently from the model used for retrieval — but storage and query must remain the same vector space or RAG breaks.
+
+## Decision
+
+- Split into two roles from the Embedding Model Registry:
+  - **Boundary Embedding Model** — Cosine cuts only; ignored by Strategies without boundary detection. Uses that model’s `semantic_threshold`.
+  - **Corpus Embedding Model** — Finished-chunk storage **and** query embedding (always paired). Selects the Model Corpus (`{strategy}__{corpus_model_id}_collection`).
+- **Not** three independent knobs: Query may not diverge from Corpus.
+- Admin stores **Default Boundary** and **Default Corpus** (migrate legacy `active_embedding_model_id` → both defaults).
+- Per-operation overrides (Decision B): Process may set Boundary + Corpus; Query/Experiment set Corpus only (Boundary irrelevant at query time). Snapshot for the operation.
+- Process UI: show Boundary picker only when a semantic Strategy is selected.
+- Provenance: always record Corpus on Query/Experiment; record Boundary when semantic Strategies are in play. Compare warns when Corpus, Boundary (if present), or Neighbors differ.
+- Query/Benchmark Corpus choice should prefer models that have a usable Model Corpus for the target doc/strategy and warn when empty.
+
+## Consequences
+
+- ADR-0019’s single Active switcher is replaced; call sites that `snapshot_active_model()` must become Corpus (and Boundary where needed).
+- Cross-model science is allowed for boundary vs corpus; silent unfair Strategy×model mixes remain out of scope as an Experiment axis.
+- Implementation is live: Admin defaults, Process Boundary+Corpus overrides, Query/Experiment Corpus override, dual provenance, Compare warnings.

docs/adr/0025-retrieval-inspect.md

@@ -0,0 +1,24 @@
+# ADR-0025: Retrieval Inspect (Benchmarks full-page mode)
+
+## Status
+
+Accepted (implemented; Expected Answer strip amended)
+
+## Context
+
+Neighbor Expansion and Expansion Tree (ADR-0023) store per-question top-k hits with prev/next for Experiments, but operators cannot audit them well: Query is live-only, Experiment expand rows and HTML reports truncate or sample one question, and a new top-level Tab would stretch navigation. Operators need a dedicated, readable surface to choose a question and inspect retrieval in full detail. Operators also need the Expected Answer beside the Generated Answer to judge Similarity without leaving Inspect.
+
+## Decision
+
+- **Retrieval Inspect** is a **full-page mode inside the Benchmarks Tab** (not a new top-level Tab, not HTML-report-only).
+- Entry: **Inspect** action on an Experiment row (alongside Report).
+- Layout: **left question rail** (status for error / has Expansion Tree) + main pane with **Strategy picker** (default `fixed_size` when present), **side-by-side Generated | Expected Answer** strip with clear column headers (plain text, no word-diff), **full-width eval scores** under that strip, and a **vertical full-text Expansion Tree** (prev → hit → next per top-k hit; neighbors empty/N/A for other Strategies).
+- Expected column always renders; empty/missing `expected_answer` shows a muted “(no expected answer)”.
+- Visual language stays the existing Dashboard theme (ADR-0008); quality comes from hierarchy, spacing, and full text — not a separate brand.
+- Reads existing Experiment `per_question` payloads (`expansion_tree`, answers, `expected_answer`, scores); no new retrieval API required for v1.
+
+## Consequences
+
+- Benchmarks Tab gains a second view mode (list/run vs Inspect); state returns cleanly via Back.
+- HTML report Expansion Tree remains a summary; Inspect is the audit UX.
+- Earlier “no expected-answer compare panel” clause is superseded by the labeled side-by-side strip (not a diff engine).

docs/out-of-scope-v1.md

@@ -12,7 +12,9 @@ No built-in export of experiment results, query logs, or benchmark comparisons t
 
 ## Custom Chunking Parameters from the UI
 
-You cannot change `chunk_size`, `chunk_overlap`, `semantic_threshold`, or `semantic_min_chunk_size` from the dashboard. These are hardcoded in `src/core/config.py` with defaults (512 tokens, 50 overlap, 0.3 threshold). Changing them requires editing the config and restarting the server. The UI uses whatever the server has configured.
+You cannot change `chunk_size`, `chunk_overlap`, or `semantic_min_chunk_size` from the dashboard. Those remain in `src/core/config.py` (defaults: 512 tokens, 50 overlap, min chunk size from config). Changing them requires editing the config and restarting the server.
+
+**Exception:** per-Embedding-Model `semantic_threshold` is Admin-configurable (registry defaults OpenAI 0.3 / Nomic 0.6; overrides in SQLite). Re-process after changing it. Other Strategy knobs stay out of the UI.
 
 ## Streaming Responses
 
@@ -43,4 +45,4 @@ No full-text search or filtering within the dashboard. You cannot search for a s
 
 ## Configuration Management from the UI
 
-You cannot view or edit the server configuration (OpenAI model, temperature, max_tokens, embedding model, Qdrant URL, database path) from the dashboard. The `/admin/config` endpoint was deferred to V1.1. All configuration is managed via the `.env` file and `src/core/config.py`.
+You cannot view or edit arbitrary server configuration (LLM model, temperature, max_tokens, Qdrant URL, database path, Ollama base URL) from the dashboard. The general `/admin/config` endpoint was deferred to V1.1. **Exception (ADR-0019):** Active Embedding Model may be switched in Admin among entries in the static Embedding Model Registry. Ollama host URL remains env-only (`OLLAMA_BASE_URL`).