docs: record decision board language and adr-0026
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
11
CONTEXT.md
11
CONTEXT.md
@@ -98,13 +98,21 @@ _Avoid_: Retrieved chunks list, neighbor list, expansion map
|
|||||||
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
|
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
|
||||||
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||||
|
|
||||||
|
**Strategy Candidate**:
|
||||||
|
A single comparable configuration for final Strategy selection under one Corpus Embedding Model: either `fixed_size` at one Neighbor Expansion level (±N), or `semantic` under one Boundary Embedding Model. Distinct from an Experiment (an Experiment is one doc’s run of one Candidate, or of several Strategies). Final selection compares Candidates, not raw multi-strategy Experiments. The Decision Board auto-discovers Candidates from single-strategy Experiments (newest per doc×Candidate cell), filtered by Corpus Embedding Model, with optional exclude of a bad Experiment.
|
||||||
|
_Avoid_: Variant, configuration, setup, arm
|
||||||
|
|
||||||
|
**Decision Board**:
|
||||||
|
A top-level Dashboard Tab for choosing between Strategy families after tuning: stage 1 picks the best Strategy Candidate within `fixed_size` and within `semantic`; stage 2 compares those two winners. Always accompanied by a per-document breakdown so aggregate rank cannot hide doc-level disagreement. Document universe for v1 is the fixed 10-doc evaluation set (same as the Neighbor Expansion Benchmark Sweep). Distinct from Compare (ad-hoc multi-Experiment side-by-side) and from Retrieval Inspect.
|
||||||
|
_Avoid_: Final compare, strategy picker, results page, leaderboard, Decision mode
|
||||||
|
|
||||||
## Architecture Decisions
|
## Architecture Decisions
|
||||||
|
|
||||||
| # | Decision | Status |
|
| # | Decision | Status |
|
||||||
|---|----------|--------|
|
|---|----------|--------|
|
||||||
ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
|
ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
|
||||||
ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
|
ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
|
||||||
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved |
|
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Decision, Admin) — not sidebar | Approved |
|
||||||
ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
|
ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
|
||||||
ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
|
ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
|
||||||
ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
|
ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
|
||||||
@@ -126,3 +134,4 @@ _Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
|||||||
ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
|
ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
|
||||||
ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
|
ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
|
||||||
ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |
|
ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |
|
||||||
|
ADR-0026 | Decision Board Tab: two-stage Strategy Candidate selection (fixed_size ±N vs semantic@Boundary) over the 10-doc set. See docs/adr/0026-*.md | Approved |
|
||||||
|
|||||||
24
docs/adr/0026-decision-board-tab.md
Normal file
24
docs/adr/0026-decision-board-tab.md
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
# ADR-0026: Decision Board Tab for final Strategy selection
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (implemented)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
After Neighbor Expansion Sweeps and semantic Boundary variants, the operator’s real job is choosing between `fixed_size` and `semantic` — not ad-hoc Experiment Compare. Compare remains useful for arbitrary side-by-side diffs, but it does not encode two-stage tuning (best variant per family, then family showdown) or a fixed 10-doc evaluation universe.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
- Add a top-level Dashboard **Decision** Tab (Decision Board).
|
||||||
|
- **Strategy Candidates**: `fixed_size` at ±0…±3, and `semantic` under each Boundary Embedding Model present in data, under a selected Corpus Embedding Model.
|
||||||
|
- Auto-discover from **single-strategy** Experiments on the hardcoded 10-doc set; newest Experiment fills each doc×Candidate cell; optional exclude chips; incomplete cells are `—` and omitted from that Candidate’s mean/wins.
|
||||||
|
- Stage 1 ranks by **mean composite** (same weights as Experiments “Best”), shows win-count, allows manual override.
|
||||||
|
- Stage 2 duels the two family winners with full metrics + per-doc head-to-head; per-doc matrix lists all Candidates with duel winners highlighted.
|
||||||
|
- Aggregation is client-side from `GET /experiments?limit=…` (default list limit raised; max 500).
|
||||||
|
|
||||||
|
## Considered options
|
||||||
|
|
||||||
|
- Enhance Compare only — rejected (wrong job, crowded Benchmarks UX).
|
||||||
|
- Standalone HTML report only — rejected (weak for override / interactive decision).
|
||||||
|
- Manual pin of every cell — rejected for v1 (too heavy for ~60 cells).
|
||||||
Reference in New Issue
Block a user