docs: record embedding, neighbor, and inspect decisions
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
64
CONTEXT.md
64
CONTEXT.md
@@ -7,11 +7,15 @@ A single-file React app (CDN-loaded, no build step) served by FastAPI at `/app`.
|
||||
_Avoid_: Admin panel, web UI, frontend
|
||||
|
||||
**Strategy**:
|
||||
One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child.
|
||||
One of the five chunking algorithms: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child. Semantic and semantic_parent_child determine chunk boundaries from meaning (adjacent-unit similarity), not from fixed sentence/paragraph counts.
|
||||
_Avoid_: Method, approach, technique
|
||||
|
||||
**Semantic Boundary Detection**:
|
||||
Using embeddings of consecutive units (sentences for semantic — Farsi-aware punctuation, with line/paragraph fallback when punctuation yields a single unit; paragraphs for semantic_parent_child) and cosine similarity against a threshold tied to the Boundary Embedding Model to decide where chunks start and end. Distinct from Corpus Embedding (finished-chunk vectors for Qdrant). If boundary embeddings are missing or mismatched, that Strategy fails — there is no fixed-count fallback.
|
||||
_Avoid_: Semantic embedding, chunk embedding (ambiguous — that often means storage vectors)
|
||||
|
||||
**Experiment**:
|
||||
A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics.
|
||||
A completed benchmark run — one document, N strategies, M questions, with per-question and aggregate metrics. Always records Corpus Embedding Model and Neighbor Expansion knobs. Records Boundary Embedding Model when semantic Strategies were part of processing/evaluation path for that corpus. Runs under different Boundary/Corpus settings must not be silently compared.
|
||||
_Avoid_: Run, trial, benchmark
|
||||
|
||||
**Chunk Preview**:
|
||||
@@ -22,6 +26,14 @@ _Avoid_: Chunk inspection, chunk view
|
||||
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
|
||||
_Avoid_: Page, route, view
|
||||
|
||||
**Retrieval Inspect**:
|
||||
A full-page mode inside the Benchmarks Tab for auditing one Experiment’s retrieval: left rail lists questions (status for error / has Expansion Tree); main pane has Strategy picker (defaults to `fixed_size` when present), a side-by-side Generated Answer | Expected Answer strip with eval scores, and a vertical full-text Expansion Tree (previous → hit → next per top-k hit; neighbors empty/N/A for other Strategies). Opened from an Experiment; not a separate top-level Tab. Distinct from live Query and from truncated HTML report Expansion Tree blocks.
|
||||
_Avoid_: Query tab, chunk preview, report tree, expansion sample
|
||||
|
||||
**Expected Answer**:
|
||||
The reference answer for a benchmark question, supplied with the question set and used by evaluation (e.g. Answer Similarity). Shown in Retrieval Inspect beside the Generated Answer. Informal synonym “golden answer” is not product language.
|
||||
_Avoid_: Golden answer, ground truth, reference answer (prefer Expected Answer in UI and docs)
|
||||
|
||||
**PDF Workspace**:
|
||||
The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
|
||||
_Avoid_: PDF page, PDF mode, PDF dashboard
|
||||
@@ -46,6 +58,46 @@ _Avoid_: OCR check, PDF validation, empty-page filter
|
||||
Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
|
||||
_Avoid_: Table extraction, table parsing, structured tables
|
||||
|
||||
**Embedding Model**:
|
||||
A registered model identity (Provider, dimension, optional task prefixes, default `semantic_threshold`). Used in one of two roles: Boundary or Corpus. Not an Experiment axis by itself — Strategies are compared under recorded Boundary+Corpus snapshots per Experiment.
|
||||
_Avoid_: Embedder, embedding provider (use Provider for that), vector model
|
||||
|
||||
**Active Embedding Model**:
|
||||
Legacy single global role (ADR-0019). Replaced by two Admin defaults — Default Boundary Embedding Model and Default Corpus Embedding Model — with per-operation overrides. Prefer Boundary / Corpus terms for new work.
|
||||
_Avoid_: Current model, selected embedder, runtime model
|
||||
|
||||
**Boundary Embedding Model**:
|
||||
The Embedding Model used only for Semantic Boundary Detection (`semantic` / `semantic_parent_child`). Does not write Qdrant vectors. Ignored when a Strategy has no boundary step. Admin stores a default; Process may override per run. Snapshotted at process start. Cuts always use this model’s `semantic_threshold` (never the Corpus model’s threshold when the two differ).
|
||||
_Avoid_: Semantic embedder, chunking model, similarity model
|
||||
|
||||
**Corpus Embedding Model**:
|
||||
The Embedding Model used to embed finished chunks into the Model Corpus and to embed queries for search. Storage and query always share this model. Determines which Model Corpus collections are read/written. Admin stores a default; Process, Query, and Experiment may override per run. Snapshotted for that operation.
|
||||
_Avoid_: Storage model, retrieval model, Qdrant model, Active Embedding Model
|
||||
|
||||
**Provider**:
|
||||
Where an Embedding Model runs: Cloud (e.g. OpenAI API) or Local (Ollama on the host). A Provider is a delivery channel, not a model identity.
|
||||
_Avoid_: Backend, engine, embedding source
|
||||
|
||||
**Model Corpus**:
|
||||
The Strategy collections whose vectors were produced by one Corpus Embedding Model. Collection names always include that model’s id (`{strategy}__{model_id}_collection`). Process, query, and Experiment only read/write the Model Corpus of the Corpus Embedding Model in force — other corpora remain untouched.
|
||||
_Avoid_: Collection set, vector partition, embedding space
|
||||
|
||||
**Embedding Model Registry**:
|
||||
The configured catalog of Embedding Models the operator may assign to Boundary or Corpus roles — each entry has a stable identity, Provider, vector dimension, and a default `semantic_threshold` (Admin may override per model in SQLite). Defaults: OpenAI 0.3, Nomic 0.6. Unlisted models are out of scope until added to the registry.
|
||||
_Avoid_: Model list, provider catalog, available embeddings
|
||||
|
||||
**Neighbor Expansion**:
|
||||
Query-time widening of retrieved context for the `fixed_size` Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (`chunk_index`). Neighbors are not re-ranked as independent hits. Counts (`neighbor_prev` / `neighbor_next`) are set per Query or Experiment like `top_k`; defaults are `0`/`0` (off). Missing neighbors at document edges are skipped. Knobs are ignored for non-`fixed_size` Strategies. Dashboard labels Experiments with a compact `±P/N` badge (tooltip explains prev/next; muted when `fixed_size` was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time.
|
||||
_Avoid_: Context windowing, chunk padding, sliding window retrieval, adjacent chunk merge
|
||||
|
||||
**Expansion Tree**:
|
||||
The operator-facing grouping of Neighbor Expansion results, returned alongside the flat `retrieved_chunks` list: top-k hits in score order, each listing previous and next chunks appended for that hit (overlaps may appear under more than one hit). When expansion is off, the tree still lists the hits with empty neighbor arrays. Shown in the Query result UI, Experiment per-question detail, and both managerial and technical HTML report views. Distinct from the flat, deduped, document-ordered context sent to the LLM (each chunk id appears only once there).
|
||||
_Avoid_: Retrieved chunks list, neighbor list, expansion map
|
||||
|
||||
**Benchmark Sweep**:
|
||||
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
|
||||
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
| # | Decision | Status |
|
||||
@@ -66,3 +118,11 @@ _Avoid_: Table extraction, table parsing, structured tables
|
||||
ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
|
||||
ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
|
||||
ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |
|
||||
ADR-0018 | Model Corpus via model-scoped collections; legacy unscoped → default cloud corpus. See docs/adr/0018-*.md | Superseded by ADR-0021 (naming) |
|
||||
ADR-0019 | Active Embedding Model is global (Admin + registry), not an Experiment axis; provenance required. See docs/adr/0019-*.md | Superseded by ADR-0024 |
|
||||
ADR-0020 | Semantic strategies require Semantic Boundary Detection (fail hard, no fixed-count fallback). See docs/adr/0020-*.md | Approved |
|
||||
ADR-0021 | Always scope Qdrant collection names by Embedding Model id (cloud and local). See docs/adr/0021-*.md | Approved |
|
||||
ADR-0022 | Per-Embedding-Model `semantic_threshold` (Admin override; defaults OpenAI 0.3 / Nomic 0.6). See docs/adr/0022-*.md | Approved |
|
||||
ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
|
||||
ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
|
||||
ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |
|
||||
|
||||
17
docs/adr/0018-model-corpus-scoped-collections.md
Normal file
17
docs/adr/0018-model-corpus-scoped-collections.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Model Corpus via model-scoped collections
|
||||
|
||||
Vectors for Cloud and Local Embedding Models must not share a Qdrant collection: dimensions and embedding spaces differ. Each Embedding Model therefore owns a **Model Corpus** — Strategy collections whose identity includes that model. Process, query, and Experiment only read/write the Active Embedding Model’s corpus; other corpora stay untouched. Pre-existing unscoped collections (e.g. `fixed_size_collection`) are migrated into the default cloud Embedding Model’s corpus so existing OpenAI work is not discarded.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Model-scoped collections** — chosen; non-destructive switch between Providers/models; clear isolation
|
||||
- **Wipe-and-rebuild on switch** — simple naming, destructive and easy to forget
|
||||
- **Dimension-gated single set** — only allow same-dimension swaps; blocks most OpenAI ↔ Ollama use
|
||||
|
||||
Legacy handling (original): **migrate unscoped → default cloud corpus**. **Superseded for naming by ADR-0021** — all models, including default cloud, use `{strategy}__{model_id}_collection`; delete leftover unscoped collections when rebuilding.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Collection naming and vector size are driven by the Embedding Model Registry entry, not a global `1536` constant
|
||||
- Admin Qdrant list shows all corpora, labeled by Embedding Model, with the active corpus emphasized; wipe stays per named collection
|
||||
- Switching Active Embedding Model does not delete the other Model Corpus; operator must process documents again under the new model to populate it
|
||||
22
docs/adr/0019-active-embedding-model-global.md
Normal file
22
docs/adr/0019-active-embedding-model-global.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Active Embedding Model is global, not an Experiment axis
|
||||
|
||||
**Status:** Superseded by [ADR-0024](0024-boundary-vs-corpus-embedding-models.md) — single Active Embedding Model replaced by **Boundary** + **Corpus** roles (query locked to Corpus).
|
||||
|
||||
Original decision retained for history:
|
||||
|
||||
The platform compares **Strategies** under controlled conditions. Embedding Model is a confounder, not a second experiment dimension: one **Active Embedding Model** applies to process, query, and new Experiments. The operator switches it from Admin among entries in a static **Embedding Model Registry** (Provider, stable id, vector dimension). Selection persists across restarts (config supplies the default only when unset). Every Experiment records which Embedding Model produced it; historical rows without provenance are treated as the default cloud model. A process, query, or Experiment **snapshots** the Active Embedding Model at start so a mid-flight Admin switch cannot mix models inside one operation.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Global Active Embedding Model** — chosen; fair Strategy comparisons; A/B models via separate Experiments
|
||||
- **Embedding Model as Experiment axis** — Strategy × model cross-product; richer science, much heavier data model/UI
|
||||
- **Per-run choice with no Experiment coupling** — flexible, invites silent unfair comparisons
|
||||
|
||||
Switcher: **Admin UI** (not env-only, not read-only status). Catalog: **static registry** (not live Ollama discovery, not free-form). Local Provider host: **`OLLAMA_BASE_URL` in config only**. Cost Estimator: **$0 embedding line when Active Embedding Model is Local**; LLM costs unchanged. This carve-out does not reopen general `/admin/config` (ADR-0007 deferred).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Introduce Provider adapters (Cloud OpenAI, Local Ollama) behind one embed API; callers bind a registry entry for the operation
|
||||
- Dashboard Admin gains an Embedding Model switcher; Benchmarks/reports must show Experiment provenance
|
||||
- `docs/out-of-scope-v1.md` “no embedding model from dashboard” is superseded for this focused control only
|
||||
- Local Nomic registry entries use task prefixes (`search_document` / `search_query`); Cloud OpenAI entries do not
|
||||
16
docs/adr/0020-semantic-boundary-detection-required.md
Normal file
16
docs/adr/0020-semantic-boundary-detection-required.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Semantic strategies require Semantic Boundary Detection
|
||||
|
||||
`semantic` and `semantic_parent_child` must cut chunks from meaning: the orchestrator embeds consecutive units (sentences / paragraphs) with the Active Embedding Model snapshot, passes those vectors into `chunk()`, then embeds finished chunks for Qdrant. Fixed-count fallbacks are removed — missing or length-mismatched boundary embeddings fail that Strategy. This completes ADR 0012’s sentence-level design in the process path (and the parent/child analogue) so benchmarks cannot silently measure “every N units” under a semantic name.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Wire both strategies + fail hard** — chosen; same Active model for boundaries and storage; accept double embed cost; re-process to replace old fake-semantic vectors
|
||||
- **Wire `semantic` only** — rejected; would leave parent/child on the same lie
|
||||
- **Keep fallback with warnings** — rejected; that is how the bug stayed hidden
|
||||
- **Always OpenAI for boundaries regardless of Active** — rejected; confounds model A/B experiments
|
||||
|
||||
## Consequences
|
||||
|
||||
- Process cost rises for these two Strategies (unit embeds + chunk embeds)
|
||||
- Prior Experiments / collections produced under the fallback are not true semantic — operator must re-process and re-benchmark
|
||||
- Strategy modules raise `ChunkingError` if boundary embeddings are absent or mismatched; orchestration is responsible for supplying them
|
||||
14
docs/adr/0021-always-scope-collection-names-by-model.md
Normal file
14
docs/adr/0021-always-scope-collection-names-by-model.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Always scope Qdrant collection names by Embedding Model id
|
||||
|
||||
Every Model Corpus collection is named `{strategy}__{embedding_model_id}_collection`, including the default cloud model (`text-embedding-3-small`). This supersedes ADR-0018’s exception that left OpenAI on legacy unscoped names (`fixed_size_collection`). Uniform naming makes Admin labeling obvious for both Cloud and Local and avoids a special case that confused operators wiping/rebuilding corpora.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Always include model id** — chosen; clear for cloud and local
|
||||
- **Keep legacy unscoped names for default cloud only** — ADR-0018; rejected going forward after a from-scratch rebuild
|
||||
|
||||
## Consequences
|
||||
|
||||
- Process/query create names like `fixed_size__text-embedding-3-small_collection` and `fixed_size__nomic-embed-text-v2-moe_collection`
|
||||
- Old unscoped collections are not written anymore; Admin still labels them as legacy default-cloud if they remain until deleted
|
||||
- Operators wiping for a clean slate should delete legacy collections before re-processing
|
||||
22
docs/adr/0022-per-embedding-model-semantic-threshold.md
Normal file
22
docs/adr/0022-per-embedding-model-semantic-threshold.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# ADR-0022: Per-Embedding-Model semantic_threshold
|
||||
|
||||
## Status
|
||||
|
||||
Approved
|
||||
|
||||
## Context
|
||||
|
||||
Semantic Boundary Detection cuts when adjacent-unit cosine similarity falls below a threshold. Cloud OpenAI and local Nomic produce different similarity distributions for the same Farsi text: the global `SEMANTIC_THRESHOLD` (config, historically ~0.3) rarely triggers cuts under Nomic, collapsing `semantic` into a single chunk. Operators need a higher Nomic default without changing OpenAI behavior, and a way to tune without editing `.env` and restarting.
|
||||
|
||||
## Decision
|
||||
|
||||
- Each Embedding Model Registry entry has a `default_semantic_threshold` (OpenAI `text-embedding-3-small`: **0.3**; Nomic `nomic-embed-text-v2-moe`: **0.6`).
|
||||
- Admin may override the effective value per model id in SQLite (`semantic_threshold:{model_id}`).
|
||||
- Process snapshots the Active Embedding Model and uses that model’s effective threshold for `semantic` and `semantic_parent_child`.
|
||||
- Admin API: list includes `semantic_threshold` / `default_semantic_threshold`; `PUT /admin/embedding-models/{id}/semantic-threshold` persists overrides.
|
||||
- Changing the threshold does not rewrite existing corpora — re-process to apply.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Global `SEMANTIC_THRESHOLD` remains a fallback only when a strategy is called without an explicit threshold (tests/legacy); production process path always passes the model-resolved value.
|
||||
- Operators must re-process after tuning; Experiments under different thresholds are not auto-invalidated.
|
||||
46
docs/adr/0023-neighbor-expansion-fixed-size.md
Normal file
46
docs/adr/0023-neighbor-expansion-fixed-size.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# ADR-0023: Neighbor Expansion for fixed_size retrieval
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (implemented)
|
||||
|
||||
## Context
|
||||
|
||||
Fixed-size chunks cut text at token windows, so the single highest-scoring chunk often lacks the sentence before/after the answer. Operators want query-time widening: after top-k retrieval, also pull previous and next chunks in document order. Doing this for every Strategy, or reshaping the top-k budget, would muddy Experiment comparisons and hide whether gains come from chunking vs from extra context.
|
||||
|
||||
Operators also need to **see** the relationship: all top-k hits and, for each hit, which upper/downer chunks were appended. A flat deduped list alone cannot show per-hit windows when neighbors overlap.
|
||||
|
||||
## Decision
|
||||
|
||||
### Retrieval behavior
|
||||
|
||||
- **Neighbor Expansion** applies only to the `fixed_size` Strategy at query time (not at process/chunking time).
|
||||
- Every top-k hit is expanded by configurable `neighbor_prev` and `neighbor_next` counts (same document, by `chunk_index`).
|
||||
- Expansion is **additive**: neighbors are merged with top-k, deduped by chunk id, then sorted into **document order** for the LLM. Context may exceed `top_k`.
|
||||
- Knobs live on Query and Experiment requests like `top_k`, with config defaults **`0` / `0`** (opt-in). Non-`fixed_size` Strategies ignore the knobs. Missing neighbors at document edges are skipped.
|
||||
- Experiments must record `neighbor_prev` / `neighbor_next` for provenance.
|
||||
|
||||
### Dual response shape
|
||||
|
||||
- **`retrieved_chunks`**: flat list matching what the LLM saw (deduped, document order; entries labeled hit vs neighbor as needed for audit/eval).
|
||||
- **`expansion_tree`**: operator-facing grouping — top-k hits in **score order**, each with `neighbors_prev` / `neighbors_next`. When expansion is off (`0`/`0`), the tree still contains the hits with **empty** neighbor arrays.
|
||||
- Overlapping windows: the same chunk may appear under **more than one** hit in the Expansion Tree; it still appears **once** in `retrieved_chunks` / the LLM prompt.
|
||||
|
||||
### Where the Expansion Tree is shown
|
||||
|
||||
- Query result UI
|
||||
- Experiment per-question detail in the Dashboard
|
||||
- HTML report — **both** managerial and technical views
|
||||
|
||||
### Experiment list & Compare provenance
|
||||
|
||||
- Experiments list shows a compact **`±P/N`** Neighbors badge (tooltip: `neighbor_prev` / `neighbor_next`; muted when `fixed_size` was not in the run).
|
||||
- Compare view repeats the badge on each Experiment column/card.
|
||||
- Compare shows a **soft warning** when selected Experiments differ on Neighbor Expansion **or** Embedding Model — intentional A/B is allowed; do not treat mismatched fixed_size scores as identical setups.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Fixed_size Experiments with expansion enabled may use more tokens than other Strategies at the same `top_k`; that asymmetry is intentional and must be visible in config/reports.
|
||||
- Hit-only retrieval metrics remain available by filtering labeled hits in the flat list; unfiltered eval reflects the expanded prompt.
|
||||
- UI and reports render the Expansion Tree; they must not assume the flat list alone can reconstruct per-hit appendages after dedupe.
|
||||
- Dashboard Compare must keep Neighbor Expansion (and Embedding Model) visible so operators can fairly compare `±0/0` vs `±1/1` vs larger windows.
|
||||
27
docs/adr/0024-boundary-vs-corpus-embedding-models.md
Normal file
27
docs/adr/0024-boundary-vs-corpus-embedding-models.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# ADR-0024: Boundary vs Corpus Embedding Model roles
|
||||
|
||||
## Status
|
||||
|
||||
Implemented. Supersedes ADR-0019’s single global Active Embedding Model for new work.
|
||||
|
||||
## Context
|
||||
|
||||
Embedding happens in three moments: (1) Semantic Boundary Detection for `semantic` / `semantic_parent_child`, (2) finished-chunk vectors into Qdrant, (3) query vectors for search. ADR-0019 forced one Active Embedding Model for all three so Strategy comparisons stayed fair. Operators now want to vary the model used for cuts independently from the model used for retrieval — but storage and query must remain the same vector space or RAG breaks.
|
||||
|
||||
## Decision
|
||||
|
||||
- Split into two roles from the Embedding Model Registry:
|
||||
- **Boundary Embedding Model** — Cosine cuts only; ignored by Strategies without boundary detection. Uses that model’s `semantic_threshold`.
|
||||
- **Corpus Embedding Model** — Finished-chunk storage **and** query embedding (always paired). Selects the Model Corpus (`{strategy}__{corpus_model_id}_collection`).
|
||||
- **Not** three independent knobs: Query may not diverge from Corpus.
|
||||
- Admin stores **Default Boundary** and **Default Corpus** (migrate legacy `active_embedding_model_id` → both defaults).
|
||||
- Per-operation overrides (Decision B): Process may set Boundary + Corpus; Query/Experiment set Corpus only (Boundary irrelevant at query time). Snapshot for the operation.
|
||||
- Process UI: show Boundary picker only when a semantic Strategy is selected.
|
||||
- Provenance: always record Corpus on Query/Experiment; record Boundary when semantic Strategies are in play. Compare warns when Corpus, Boundary (if present), or Neighbors differ.
|
||||
- Query/Benchmark Corpus choice should prefer models that have a usable Model Corpus for the target doc/strategy and warn when empty.
|
||||
|
||||
## Consequences
|
||||
|
||||
- ADR-0019’s single Active switcher is replaced; call sites that `snapshot_active_model()` must become Corpus (and Boundary where needed).
|
||||
- Cross-model science is allowed for boundary vs corpus; silent unfair Strategy×model mixes remain out of scope as an Experiment axis.
|
||||
- Implementation is live: Admin defaults, Process Boundary+Corpus overrides, Query/Experiment Corpus override, dual provenance, Compare warnings.
|
||||
24
docs/adr/0025-retrieval-inspect.md
Normal file
24
docs/adr/0025-retrieval-inspect.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# ADR-0025: Retrieval Inspect (Benchmarks full-page mode)
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (implemented; Expected Answer strip amended)
|
||||
|
||||
## Context
|
||||
|
||||
Neighbor Expansion and Expansion Tree (ADR-0023) store per-question top-k hits with prev/next for Experiments, but operators cannot audit them well: Query is live-only, Experiment expand rows and HTML reports truncate or sample one question, and a new top-level Tab would stretch navigation. Operators need a dedicated, readable surface to choose a question and inspect retrieval in full detail. Operators also need the Expected Answer beside the Generated Answer to judge Similarity without leaving Inspect.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Retrieval Inspect** is a **full-page mode inside the Benchmarks Tab** (not a new top-level Tab, not HTML-report-only).
|
||||
- Entry: **Inspect** action on an Experiment row (alongside Report).
|
||||
- Layout: **left question rail** (status for error / has Expansion Tree) + main pane with **Strategy picker** (default `fixed_size` when present), **side-by-side Generated | Expected Answer** strip with clear column headers (plain text, no word-diff), **full-width eval scores** under that strip, and a **vertical full-text Expansion Tree** (prev → hit → next per top-k hit; neighbors empty/N/A for other Strategies).
|
||||
- Expected column always renders; empty/missing `expected_answer` shows a muted “(no expected answer)”.
|
||||
- Visual language stays the existing Dashboard theme (ADR-0008); quality comes from hierarchy, spacing, and full text — not a separate brand.
|
||||
- Reads existing Experiment `per_question` payloads (`expansion_tree`, answers, `expected_answer`, scores); no new retrieval API required for v1.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Benchmarks Tab gains a second view mode (list/run vs Inspect); state returns cleanly via Back.
|
||||
- HTML report Expansion Tree remains a summary; Inspect is the audit UX.
|
||||
- Earlier “no expected-answer compare panel” clause is superseded by the labeled side-by-side strip (not a diff engine).
|
||||
@@ -12,7 +12,9 @@ No built-in export of experiment results, query logs, or benchmark comparisons t
|
||||
|
||||
## Custom Chunking Parameters from the UI
|
||||
|
||||
You cannot change `chunk_size`, `chunk_overlap`, `semantic_threshold`, or `semantic_min_chunk_size` from the dashboard. These are hardcoded in `src/core/config.py` with defaults (512 tokens, 50 overlap, 0.3 threshold). Changing them requires editing the config and restarting the server. The UI uses whatever the server has configured.
|
||||
You cannot change `chunk_size`, `chunk_overlap`, or `semantic_min_chunk_size` from the dashboard. Those remain in `src/core/config.py` (defaults: 512 tokens, 50 overlap, min chunk size from config). Changing them requires editing the config and restarting the server.
|
||||
|
||||
**Exception:** per-Embedding-Model `semantic_threshold` is Admin-configurable (registry defaults OpenAI 0.3 / Nomic 0.6; overrides in SQLite). Re-process after changing it. Other Strategy knobs stay out of the UI.
|
||||
|
||||
## Streaming Responses
|
||||
|
||||
@@ -43,4 +45,4 @@ No full-text search or filtering within the dashboard. You cannot search for a s
|
||||
|
||||
## Configuration Management from the UI
|
||||
|
||||
You cannot view or edit the server configuration (OpenAI model, temperature, max_tokens, embedding model, Qdrant URL, database path) from the dashboard. The `/admin/config` endpoint was deferred to V1.1. All configuration is managed via the `.env` file and `src/core/config.py`.
|
||||
You cannot view or edit arbitrary server configuration (LLM model, temperature, max_tokens, Qdrant URL, database path, Ollama base URL) from the dashboard. The general `/admin/config` endpoint was deferred to V1.1. **Exception (ADR-0019):** Active Embedding Model may be switched in Admin among entries in the static Embedding Model Registry. Ollama host URL remains env-only (`OLLAMA_BASE_URL`).
|
||||
|
||||
Reference in New Issue
Block a user