Compare commits
4 Commits
e2c7fdc059
...
640691b8ab
| Author | SHA1 | Date | |
|---|---|---|---|
| 640691b8ab | |||
| 6261bc0a23 | |||
| 2ac69c9944 | |||
| 56b8d9401a |
11
CONTEXT.md
11
CONTEXT.md
@@ -98,13 +98,21 @@ _Avoid_: Retrieved chunks list, neighbor list, expansion map
|
||||
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
|
||||
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||
|
||||
**Strategy Candidate**:
|
||||
A single comparable configuration for final Strategy selection under one Corpus Embedding Model: either `fixed_size` at one Neighbor Expansion level (±N), or `semantic` under one Boundary Embedding Model. Distinct from an Experiment (an Experiment is one doc’s run of one Candidate, or of several Strategies). Final selection compares Candidates, not raw multi-strategy Experiments. The Decision Board auto-discovers Candidates from single-strategy Experiments (newest per doc×Candidate cell), filtered by Corpus Embedding Model, with optional exclude of a bad Experiment.
|
||||
_Avoid_: Variant, configuration, setup, arm
|
||||
|
||||
**Decision Board**:
|
||||
A top-level Dashboard Tab for choosing between Strategy families after tuning: stage 1 picks the best Strategy Candidate within `fixed_size` and within `semantic`; stage 2 compares those two winners. Always accompanied by a per-document breakdown so aggregate rank cannot hide doc-level disagreement. Document universe for v1 is the fixed 10-doc evaluation set (same as the Neighbor Expansion Benchmark Sweep). Distinct from Compare (ad-hoc multi-Experiment side-by-side) and from Retrieval Inspect.
|
||||
_Avoid_: Final compare, strategy picker, results page, leaderboard, Decision mode
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
| # | Decision | Status |
|
||||
|---|----------|--------|
|
||||
ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
|
||||
ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
|
||||
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved |
|
||||
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Decision, Admin) — not sidebar | Approved |
|
||||
ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
|
||||
ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
|
||||
ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
|
||||
@@ -126,3 +134,4 @@ _Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||
ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
|
||||
ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
|
||||
ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |
|
||||
ADR-0026 | Decision Board Tab: two-stage Strategy Candidate selection (fixed_size ±N vs semantic@Boundary) over the 10-doc set. See docs/adr/0026-*.md | Approved |
|
||||
|
||||
280
docs/HLD.md
Normal file
280
docs/HLD.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# High-Level Design (HLD)
|
||||
|
||||
RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
|
||||
|
||||
**Audience:** architects, tech leads, new engineers
|
||||
**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
|
||||
|
||||
Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
|
||||
|
||||
---
|
||||
|
||||
## 2. Goals and Non-Goals
|
||||
|
||||
### Goals
|
||||
|
||||
| Goal | Notes |
|
||||
|------|--------|
|
||||
| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
|
||||
| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
|
||||
| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
|
||||
| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
|
||||
| Failure isolation | One Strategy failing during process does not discard others |
|
||||
|
||||
### Non-Goals (v1)
|
||||
|
||||
| Out of scope | See |
|
||||
|--------------|-----|
|
||||
| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
|
||||
| Multi-tenant auth | Single-operator local tool |
|
||||
| Production-scale concurrency | SQLite + single process |
|
||||
| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
|
||||
|
||||
---
|
||||
|
||||
## 3. System Context
|
||||
|
||||
```
|
||||
┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐
|
||||
│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│
|
||||
│ (Browser) │ Dashboard /app │ │
|
||||
└──────────────┘ │ Documents · Chunking · Benchmarking │
|
||||
│ Admin · Static Dashboard │
|
||||
└───────────┬─────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────┼─────────────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
|
||||
│ SQLite │ │ Qdrant │ │ LLM / Embed │
|
||||
│ metadata, │ │ Model Corpus │ │ OpenAI Cloud │
|
||||
│ Experiments, │ │ vectors + │ │ + Ollama Local│
|
||||
│ app_settings │ │ chunk payload │ └────────────────┘
|
||||
└────────────────┘ └────────────────┘
|
||||
```
|
||||
|
||||
**External dependencies**
|
||||
|
||||
| System | Role |
|
||||
|--------|------|
|
||||
| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
|
||||
| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
|
||||
| Qdrant | Vector search per Strategy × Corpus Embedding Model |
|
||||
| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
|
||||
|
||||
---
|
||||
|
||||
## 4. Logical Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Presentation Layer │
|
||||
│ src/static/index.html — single-file React Dashboard (CDN, no build) │
|
||||
│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│ REST
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ API Layer (FastAPI) │
|
||||
│ /documents* /queries* /benchmarks* /experiments* /admin/* │
|
||||
│ /strategies /app (StaticFiles) │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ Domain Services │
|
||||
│ documents/service chunking/service query_service benchmark_service│
|
||||
│ evaluation admin/service report │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ Foundation │
|
||||
│ core/config · models · exceptions · dependencies │
|
||||
│ chunking/base · embedding · embedding_models · strategies/* │
|
||||
│ documents/parser · pdf_parser │
|
||||
│ storage/sqlite · storage/qdrant │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Major Components
|
||||
|
||||
| Component | Responsibility | Key modules |
|
||||
|-----------|----------------|-------------|
|
||||
| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
|
||||
| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
|
||||
| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
|
||||
| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
|
||||
| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
|
||||
| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
|
||||
| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
|
||||
| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
|
||||
|
||||
---
|
||||
|
||||
## 6. Domain Concepts (summary)
|
||||
|
||||
Full glossary: [CONTEXT.md](../CONTEXT.md).
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
|
||||
| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
|
||||
| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
|
||||
| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
|
||||
| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
|
||||
| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
|
||||
| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
|
||||
| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
|
||||
| **Decision Board** | Two-stage selection over the fixed evaluation document set |
|
||||
|
||||
---
|
||||
|
||||
## 7. End-to-End Flows
|
||||
|
||||
### 7.1 Ingest and process
|
||||
|
||||
```
|
||||
Upload (.docx/.doc/.pdf)
|
||||
→ Text-layer Gate (PDF) / parse DOCX
|
||||
→ Heading Reconstruction (PDF) → DocumentTree + markdown
|
||||
→ SQLite documents
|
||||
|
||||
Process(strategies, boundary?, corpus?)
|
||||
→ For each Strategy (isolated failures):
|
||||
Boundary embeds (semantic*) → Strategy.chunk()
|
||||
→ Corpus embeds → Qdrant Model Corpus upsert
|
||||
→ Update chunk_counts + last_* embedding provenance on document
|
||||
```
|
||||
|
||||
### 7.2 Query
|
||||
|
||||
```
|
||||
POST /queries
|
||||
→ Resolve Corpus Embedding Model
|
||||
→ Embed question (Corpus)
|
||||
→ Qdrant top-k (document filter)
|
||||
→ Neighbor Expansion if fixed_size (±P/N)
|
||||
→ Build context (parent fetch for semantic_parent_child)
|
||||
→ LLM answer → SQLite queries (+ expansion_tree)
|
||||
```
|
||||
|
||||
### 7.3 Experiment
|
||||
|
||||
```
|
||||
POST /benchmarks
|
||||
→ Load questions JSON
|
||||
→ For each question × Strategy: Query + evaluate_single
|
||||
→ Aggregate metrics → SQLite experiments
|
||||
→ Optional HTML report (managerial / technical)
|
||||
```
|
||||
|
||||
### 7.4 Decision
|
||||
|
||||
```
|
||||
Dashboard Decision Tab
|
||||
→ Load Experiments filtered by Corpus Embedding Model
|
||||
→ Discover Strategy Candidates (newest per doc × Candidate)
|
||||
→ Stage 1: best fixed_size ±N vs best semantic@Boundary
|
||||
→ Stage 2: compare winners + per-document breakdown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Data Architecture
|
||||
|
||||
### 8.1 Responsibility split
|
||||
|
||||
| Store | Owns |
|
||||
|-------|------|
|
||||
| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
|
||||
| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
|
||||
| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
|
||||
|
||||
### 8.2 Model Corpus naming
|
||||
|
||||
```
|
||||
{strategy}__{sanitized_model_id}_collection
|
||||
```
|
||||
|
||||
Example: `fixed_size__text-embedding-3-large_collection`
|
||||
|
||||
Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
|
||||
|
||||
### 8.3 Provenance
|
||||
|
||||
Experiments store:
|
||||
|
||||
- `embedding_model_id` / `embedding_provider` — Corpus
|
||||
- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
|
||||
- Neighbor Expansion levels in `benchmark_config`
|
||||
|
||||
Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
|
||||
|
||||
---
|
||||
|
||||
## 9. Deployment View
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Host process │
|
||||
│ uvicorn src.main:app │
|
||||
│ └── FastAPI + Dashboard static │
|
||||
│ │
|
||||
│ data/chunking_benchmark.db │
|
||||
│ .env (secrets + knobs) │
|
||||
└───────────────┬─────────────────────────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
▼ ▼
|
||||
Qdrant (:6333) OpenAI / Ollama
|
||||
```
|
||||
|
||||
Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
|
||||
|
||||
---
|
||||
|
||||
## 10. Cross-Cutting Concerns
|
||||
|
||||
| Concern | Approach |
|
||||
|---------|----------|
|
||||
| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
|
||||
| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
|
||||
| Logging | Request middleware + structured stage logs in query/benchmark |
|
||||
| CORS | Permissive (`*`) for local Dashboard development |
|
||||
| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
|
||||
| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
|
||||
|
||||
---
|
||||
|
||||
## 11. Key Architectural Decisions
|
||||
|
||||
| ADR | Decision |
|
||||
|-----|----------|
|
||||
| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
|
||||
| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
|
||||
| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
|
||||
| 0021 | Always scope Qdrant collections by Embedding Model id |
|
||||
| 0022 | Per-model `semantic_threshold` |
|
||||
| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
|
||||
| 0024 | Boundary vs Corpus Embedding Model roles |
|
||||
| 0025 | Retrieval Inspect in Benchmarks Tab |
|
||||
| 0026 | Decision Board two-stage Candidate selection |
|
||||
|
||||
---
|
||||
|
||||
## 12. Related Documents
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
|
||||
| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
|
||||
| [data-flow.md](data-flow.md) | Pipeline detail |
|
||||
| [api-reference.md](api-reference.md) | HTTP contracts |
|
||||
| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
|
||||
| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
|
||||
| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |
|
||||
571
docs/LLD.md
Normal file
571
docs/LLD.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# Low-Level Design (LLD)
|
||||
|
||||
RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.
|
||||
|
||||
**Audience:** implementers
|
||||
**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Package Map
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.py # App factory, middleware, router mount
|
||||
├── static/index.html # Dashboard (React + Babel CDN)
|
||||
├── core/
|
||||
│ ├── config.py # Settings (.env)
|
||||
│ ├── models.py # Chunk, DocumentTree, StrategyName
|
||||
│ ├── exceptions.py # Domain errors + handlers
|
||||
│ └── dependencies.py # OpenAI / Ollama / Qdrant singletons
|
||||
├── storage/
|
||||
│ ├── sqlite.py # documents, queries, experiments, app_settings
|
||||
│ └── qdrant.py # Model Corpus CRUD + search
|
||||
├── documents/
|
||||
│ ├── parser.py # DOCX / .doc (+ LibreOffice)
|
||||
│ ├── pdf_parser.py # Text PDF + Text-layer Gate
|
||||
│ ├── heading_heuristics.py # PDF Heading Reconstruction helpers
|
||||
│ ├── service.py # Upload / process / delete
|
||||
│ ├── routes.py
|
||||
│ └── models.py # ProcessRequest, DocumentResponse, …
|
||||
├── chunking/
|
||||
│ ├── base.py # ChunkingStrategy ABC, tokens, sentences
|
||||
│ ├── embedding.py # Boundary/Corpus resolve + embed_texts
|
||||
│ ├── embedding_models.py # Registry, thresholds, task prefixes
|
||||
│ ├── service.py # run_strategies orchestration
|
||||
│ └── strategies/
|
||||
│ ├── fixed_size.py
|
||||
│ ├── recursive.py
|
||||
│ ├── semantic.py
|
||||
│ ├── contextual_retrieval.py
|
||||
│ └── semantic_parent_child.py
|
||||
├── benchmarking/
|
||||
│ ├── query_service.py # Query + Neighbor Expansion
|
||||
│ ├── benchmark_service.py # Experiment runner + cost estimate
|
||||
│ ├── evaluation.py # LLM-as-Judge
|
||||
│ ├── report.py # HTML reports
|
||||
│ ├── routes.py
|
||||
│ └── models.py
|
||||
└── admin/
|
||||
├── routes.py # /admin/*
|
||||
└── service.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Application Bootstrap
|
||||
|
||||
**Entry:** `create_app()` in `src/main.py`
|
||||
|
||||
1. `init_db()` — create/migrate SQLite schema
|
||||
2. Register CORS + `RequestLoggingMiddleware`
|
||||
3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`
|
||||
4. Routers: documents, benchmarking, admin
|
||||
5. Mount `StaticFiles` at `/app` → `src/static/`
|
||||
|
||||
```python
|
||||
# Singletons (lru_cache)
|
||||
get_openai_client() # Cloud LLM + cloud embeds
|
||||
get_ollama_client() # OpenAI-compatible client → Ollama base URL
|
||||
get_qdrant_client()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
`Settings` (`src/core/config.py`) — selected fields:
|
||||
|
||||
| Field | Default | Use |
|
||||
|-------|---------|-----|
|
||||
| `openai_api_key` | required | Cloud Provider |
|
||||
| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
|
||||
| `llm_model` | `gpt-4o-mini` | Answer + judge |
|
||||
| `ollama_base_url` | host URL | Local embeds |
|
||||
| `qdrant_url` | `http://localhost:6333` | Vectors |
|
||||
| `top_k` | `5` | Retrieval |
|
||||
| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults |
|
||||
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
|
||||
| `semantic_threshold` | `0.3` | Fallback if model default missing |
|
||||
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
|
||||
| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
|
||||
| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Domain Models
|
||||
|
||||
### 4.1 StrategyName
|
||||
|
||||
```python
|
||||
class StrategyName(str, Enum):
|
||||
RECURSIVE = "recursive"
|
||||
FIXED_SIZE = "fixed_size"
|
||||
SEMANTIC = "semantic"
|
||||
CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
|
||||
SEMANTIC_PARENT_CHILD = "semantic_parent_child"
|
||||
```
|
||||
|
||||
### 4.2 Chunk (unified)
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `document_name` | str | Source filename |
|
||||
| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
|
||||
| `strategy_name` | StrategyName | |
|
||||
| `chunk_index` | int | Document order (used by Neighbor Expansion) |
|
||||
| `text` | str | Stored in Qdrant; used for LLM context |
|
||||
| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
|
||||
| `parent_id` | str \| None | Parent-child Strategy |
|
||||
| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |
|
||||
|
||||
`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).
|
||||
|
||||
### 4.3 DocumentTree
|
||||
|
||||
```
|
||||
DocumentTree
|
||||
└── root: DocumentTreeNode
|
||||
node_type: document | section | article | paragraph
|
||||
text, heading, heading_level, children[]
|
||||
```
|
||||
|
||||
Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage LLD
|
||||
|
||||
### 5.1 SQLite schema
|
||||
|
||||
**documents**
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|--------|------|---------|
|
||||
| `id` | TEXT PK | UUID hex |
|
||||
| `filename` | TEXT | Original name |
|
||||
| `parsed_text` | TEXT | Markdown for chunking |
|
||||
| `document_tree` | TEXT | JSON tree |
|
||||
| `chunk_counts` | TEXT | JSON `{strategy: count}` |
|
||||
| `last_corpus_embedding_model_id` | TEXT | Provenance |
|
||||
| `last_boundary_embedding_model_id` | TEXT | Provenance |
|
||||
| `created_at` | TEXT | ISO UTC |
|
||||
|
||||
**queries**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id`, `strategy_name` | Identity |
|
||||
| `question`, `answer` | Content |
|
||||
| `retrieved_chunks` | Flat LLM/eval list (JSON) |
|
||||
| `expansion_tree` | Per-hit neighbors (JSON) |
|
||||
| `latency_breakdown`, `token_usage` | Observability JSON |
|
||||
| `created_at` | |
|
||||
|
||||
**experiments**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id` | Identity |
|
||||
| `benchmark_config` | top_k, neighbor_prev/next, etc. |
|
||||
| `questions`, `per_question` | Inputs + per-cell results |
|
||||
| `aggregate_metrics` | Per-Strategy averages |
|
||||
| `strategies_used` | JSON list |
|
||||
| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
|
||||
| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
|
||||
| `created_at` | |
|
||||
|
||||
**app_settings** — key/value for:
|
||||
|
||||
- `corpus_embedding_model_id`, `boundary_embedding_model_id`
|
||||
- Legacy `active_embedding_model_id` (migrated into both roles)
|
||||
- `semantic_threshold:{model_id}` overrides
|
||||
|
||||
Connection: WAL mode, foreign keys ON, one connection per call (no pool).
|
||||
|
||||
### 5.2 Qdrant
|
||||
|
||||
**Collection name**
|
||||
|
||||
```python
|
||||
def collection_name(strategy, model_id) -> str:
|
||||
return f"{strategy}__{sanitize(model_id)}_collection"
|
||||
# sanitize: replace : and / with -
|
||||
```
|
||||
|
||||
**Point**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
|
||||
| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
|
||||
| `payload` | ChunkMetadata fields |
|
||||
|
||||
**Search**
|
||||
|
||||
- Cosine distance
|
||||
- Optional filter: `document_name == filename`
|
||||
- Returns payload + score
|
||||
|
||||
**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.
|
||||
|
||||
**Admin:** list/create/delete collections; wipe points.
|
||||
|
||||
---
|
||||
|
||||
## 6. Embedding Subsystem
|
||||
|
||||
### 6.1 Registry (`embedding_models.py`)
|
||||
|
||||
| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
|
||||
|----|----------|-----------|----------------------------|---------------|
|
||||
| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
|
||||
| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
|
||||
| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |
|
||||
|
||||
`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.
|
||||
|
||||
### 6.2 Role resolution (`embedding.py`)
|
||||
|
||||
```
|
||||
resolve_corpus_model(optional_id) → snapshot for process/query/experiment
|
||||
resolve_boundary_model(optional_id) → snapshot for semantic cuts
|
||||
|
||||
get_corpus_embedding_model() / get_boundary_embedding_model()
|
||||
→ app_settings → registry → DEFAULT_CLOUD_MODEL_ID
|
||||
```
|
||||
|
||||
Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.
|
||||
|
||||
### 6.3 embed_texts / embed_single
|
||||
|
||||
1. `apply_task_prefixes(texts, model, purpose)` if needed
|
||||
2. Batch: OpenAI 2048 / Ollama 64
|
||||
3. Route to `get_openai_client()` or `get_ollama_client()` by Provider
|
||||
4. Raise `EmbeddingError` on failure
|
||||
|
||||
**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).
|
||||
|
||||
---
|
||||
|
||||
## 7. Documents LLD
|
||||
|
||||
### 7.1 Upload
|
||||
|
||||
```
|
||||
upload_document(filename, bytes)
|
||||
→ temp file → parse_document(path)
|
||||
→ db.save_document(parsed_text=markdown, document_tree=JSON)
|
||||
```
|
||||
|
||||
**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).
|
||||
|
||||
**PDF path (`pdf_parser.py`):**
|
||||
|
||||
1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF
|
||||
2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)
|
||||
3. Table Flattening to sequential plain text
|
||||
4. Emit markdown `#` / `##` + DocumentTree
|
||||
|
||||
### 7.2 Process
|
||||
|
||||
```
|
||||
ProcessRequest:
|
||||
strategies: list[StrategyName] # default all five
|
||||
boundary_model_id: str | None
|
||||
corpus_model_id: str | None
|
||||
|
||||
process_document → chunking.service.run_strategies(...)
|
||||
→ ProcessResponse(completed[], failed[], corpus_*, boundary_*)
|
||||
```
|
||||
|
||||
### 7.3 Delete
|
||||
|
||||
Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Chunking LLD
|
||||
|
||||
### 8.1 Interface
|
||||
|
||||
```python
|
||||
class ChunkingStrategy(ABC):
|
||||
name: StrategyName
|
||||
|
||||
@abstractmethod
|
||||
def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
|
||||
...
|
||||
```
|
||||
|
||||
Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).
|
||||
|
||||
### 8.2 Orchestrator (`run_strategies`)
|
||||
|
||||
```
|
||||
corpus = resolve_corpus_model(corpus_model_id)
|
||||
boundary = resolve_boundary_model(...) if any(semantic*) else unused
|
||||
|
||||
for strategy in strategies:
|
||||
try:
|
||||
ensure_collection(strategy, corpus.id, corpus.dimension)
|
||||
chunks = _chunk_document(...) # inject Boundary embeds if needed
|
||||
texts = enriched_content or text
|
||||
embeddings = embed_texts(texts, model=corpus, purpose="document")
|
||||
upsert_chunks(chunks, embeddings, model_id=corpus.id)
|
||||
except → failed[] (others continue)
|
||||
|
||||
update_chunk_counts + update_process_embedding_provenance
|
||||
```
|
||||
|
||||
### 8.3 Strategy algorithms (summary)
|
||||
|
||||
| Strategy | Input units | Boundary | Output |
|
||||
|----------|-------------|----------|--------|
|
||||
| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
|
||||
| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
|
||||
| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
|
||||
| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
|
||||
| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |
|
||||
|
||||
Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.
|
||||
|
||||
**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.
|
||||
|
||||
Deep dive: [strategy-technical-details.md](strategy-technical-details.md).
|
||||
|
||||
---
|
||||
|
||||
## 9. Query LLD
|
||||
|
||||
### 9.1 `run_query` pipeline
|
||||
|
||||
```
|
||||
1. Load document (filename for filter)
|
||||
2. Resolve Corpus model
|
||||
3. embed_single(question, purpose="query")
|
||||
4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
|
||||
5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
|
||||
6. _build_context (parent scroll for semantic_parent_child)
|
||||
7. _generate_answer (settings.llm_model, temp 0)
|
||||
8. db.save_query(...)
|
||||
```
|
||||
|
||||
### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)
|
||||
|
||||
**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.
|
||||
|
||||
1. Build Expansion Tree nodes for each top-k hit (score order)
|
||||
2. Collect needed `chunk_index` values: hit±prev/next (skip < 0)
|
||||
3. Fetch missing indices via Qdrant
|
||||
4. Fill `neighbors_prev` / `neighbors_next` per hit
|
||||
5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`
|
||||
6. Neighbors have `role="neighbor"`, `score=None`
|
||||
|
||||
Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).
|
||||
|
||||
### 9.3 Parent-child context
|
||||
|
||||
For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.
|
||||
|
||||
---
|
||||
|
||||
## 10. Benchmark LLD
|
||||
|
||||
### 10.1 Questions format
|
||||
|
||||
```json
|
||||
{
|
||||
"questions": [
|
||||
{
|
||||
"id": "q1",
|
||||
"question": "...",
|
||||
"expected_answer": "...",
|
||||
"category": "...",
|
||||
"difficulty": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Load from `questions_file` path or inline `questions` list.
|
||||
|
||||
### 10.2 `run_benchmark`
|
||||
|
||||
```
|
||||
embedding_model = resolve_corpus_model(corpus_model_id)
|
||||
boundary_id = doc.last_boundary_embedding_model_id # if semantic in strategies
|
||||
|
||||
for question in questions:
|
||||
for strategy in strategies:
|
||||
result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
|
||||
scores = evaluate_single(question, retrieved_chunks, expected, generated)
|
||||
append per_question row
|
||||
|
||||
aggregate_metrics[strategy] = averages + hallucination_rate
|
||||
best_strategy = argmax (e.g. answer_similarity / composite — see service)
|
||||
save experiment with provenance + benchmark_config
|
||||
```
|
||||
|
||||
### 10.3 Evaluation (`evaluate_single`)
|
||||
|
||||
LLM returns JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"context_relevance": 1-10,
|
||||
"answer_similarity": 1-10,
|
||||
"faithfulness": 1-10,
|
||||
"hallucination": true|false,
|
||||
"reasoning": "..."
|
||||
}
|
||||
```
|
||||
|
||||
`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).
|
||||
|
||||
### 10.4 Reports
|
||||
|
||||
`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).
|
||||
|
||||
### 10.5 Cost estimate
|
||||
|
||||
`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.
|
||||
|
||||
---
|
||||
|
||||
## 11. Admin LLD
|
||||
|
||||
| Endpoint | Behavior |
|
||||
|----------|----------|
|
||||
| `GET /admin/health` | App + Qdrant + SQLite ping |
|
||||
| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
|
||||
| `PUT .../corpus` · `.../boundary` | Set role defaults |
|
||||
| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
|
||||
| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
|
||||
| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
|
||||
| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
|
||||
| `POST /admin/cost-estimate` | Same heuristic as dry-run |
|
||||
|
||||
Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).
|
||||
|
||||
---
|
||||
|
||||
## 12. Dashboard LLD (behavioral)
|
||||
|
||||
| Concern | Design |
|
||||
|---------|--------|
|
||||
| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
|
||||
| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
|
||||
| Theme | Dark `#111113`… + amber accent (ADR-0008) |
|
||||
| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
|
||||
| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
|
||||
| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
|
||||
| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |
|
||||
|
||||
No client router or global store — props from root state.
|
||||
|
||||
---
|
||||
|
||||
## 13. API Surface (concise)
|
||||
|
||||
| Method | Path | Service |
|
||||
|--------|------|---------|
|
||||
| GET/POST | `/documents` | list / upload |
|
||||
| POST | `/documents/{id}/process` | Strategies |
|
||||
| DELETE | `/documents/{id}` | delete |
|
||||
| GET | `/strategies` | catalog |
|
||||
| POST | `/queries` | query |
|
||||
| GET | `/queries/{id}` | history |
|
||||
| POST | `/benchmarks` | Experiment or dry_run |
|
||||
| GET | `/benchmarks/{id}` | Experiment detail |
|
||||
| GET | `/benchmarks/{id}/report` | HTML |
|
||||
| GET | `/experiments` | list (+ filters used by Decision Board) |
|
||||
| * | `/admin/*` | ops |
|
||||
| GET | `/app/` | Dashboard |
|
||||
|
||||
Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.
|
||||
|
||||
**Request highlights**
|
||||
|
||||
```python
|
||||
# Process
|
||||
boundary_model_id: Optional[str]
|
||||
corpus_model_id: Optional[str]
|
||||
|
||||
# Query / Benchmark
|
||||
top_k: int = 5
|
||||
neighbor_prev / neighbor_next: int = 0..5
|
||||
corpus_model_id: Optional[str]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Error Model
|
||||
|
||||
| Exception | Typical cause | HTTP |
|
||||
|-----------|---------------|------|
|
||||
| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
|
||||
| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
|
||||
| `EmbeddingError` | Provider/API failure | 400 |
|
||||
| `QdrantError` | Collection/upsert/search failure | 400 |
|
||||
| `QueryError` | Missing doc, LLM answer failure | 400 |
|
||||
| `BenchmarkError` | Bad questions file, eval failure | 400 |
|
||||
|
||||
Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.
|
||||
|
||||
---
|
||||
|
||||
## 15. Sequence Diagrams
|
||||
|
||||
### Process (one Strategy)
|
||||
|
||||
```
|
||||
Client → DocumentsAPI → DocumentService → ChunkingService
|
||||
ChunkingService → resolve Boundary/Corpus
|
||||
ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
|
||||
ChunkingService → embed_texts (Corpus)
|
||||
ChunkingService → Qdrant.upsert
|
||||
ChunkingService → SQLite.update counts + provenance
|
||||
Client ← ProcessResponse
|
||||
```
|
||||
|
||||
### Query with Neighbor Expansion
|
||||
|
||||
```
|
||||
Client → QueryAPI → run_query
|
||||
→ embed_single (Corpus)
|
||||
→ Qdrant.search top-k
|
||||
→ apply_neighbor_expansion → get_chunks_by_indices
|
||||
→ _build_context → OpenAI chat
|
||||
→ SQLite.save_query
|
||||
Client ← QueryResponse (retrieved_chunks + expansion_tree)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 16. Extension Points
|
||||
|
||||
| Extension | Steps |
|
||||
|-----------|-------|
|
||||
| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
|
||||
| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
|
||||
| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
|
||||
| New Admin op | Prefer `/admin` only when domain routers lack the capability |
|
||||
|
||||
---
|
||||
|
||||
## 17. Scripts
|
||||
|
||||
`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).
|
||||
|
||||
---
|
||||
|
||||
## 18. Related Documents
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| [HLD.md](HLD.md) | System context and component view |
|
||||
| [data-flow.md](data-flow.md) | Narrative pipelines |
|
||||
| [configuration.md](configuration.md) | Env knobs |
|
||||
| [adr/](adr/) | Decision records |
|
||||
| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |
|
||||
@@ -6,12 +6,12 @@ Complete documentation for the RAG Chunking Benchmarker.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **New to the project?** Start with [Architecture Overview](architecture.md)
|
||||
2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
|
||||
3. **Need to use the API?** Check [API Reference](api-reference.md)
|
||||
4. **Configuring the system?** See [Configuration Guide](configuration.md)
|
||||
5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
|
||||
6. **Curious about data flow?** See [Data Flow](data-flow.md)
|
||||
1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail)
|
||||
2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md)
|
||||
3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
|
||||
4. **Need to use the API?** Check [API Reference](api-reference.md)
|
||||
5. **Configuring the system?** See [Configuration Guide](configuration.md)
|
||||
6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
|
||||
|
||||
---
|
||||
|
||||
@@ -19,13 +19,17 @@ Complete documentation for the RAG Chunking Benchmarker.
|
||||
|
||||
| File | Purpose | Audience |
|
||||
|------|---------|----------|
|
||||
| [architecture.md](architecture.md) | System structure and design | New team members |
|
||||
| [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members |
|
||||
| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers |
|
||||
| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members |
|
||||
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
|
||||
| [api-reference.md](api-reference.md) | All endpoints documented | Developers |
|
||||
| [configuration.md](configuration.md) | Settings and environment variables | DevOps |
|
||||
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
|
||||
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
|
||||
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
|
||||
| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators |
|
||||
| [adr/](adr/) | Architectural Decision Records | Everyone |
|
||||
| [phases.md](phases.md) | Implementation phases | Project managers |
|
||||
| [tasks.md](tasks.md) | Task tracking | Developers |
|
||||
|
||||
|
||||
24
docs/adr/0026-decision-board-tab.md
Normal file
24
docs/adr/0026-decision-board-tab.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# ADR-0026: Decision Board Tab for final Strategy selection
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (implemented)
|
||||
|
||||
## Context
|
||||
|
||||
After Neighbor Expansion Sweeps and semantic Boundary variants, the operator’s real job is choosing between `fixed_size` and `semantic` — not ad-hoc Experiment Compare. Compare remains useful for arbitrary side-by-side diffs, but it does not encode two-stage tuning (best variant per family, then family showdown) or a fixed 10-doc evaluation universe.
|
||||
|
||||
## Decision
|
||||
|
||||
- Add a top-level Dashboard **Decision** Tab (Decision Board).
|
||||
- **Strategy Candidates**: `fixed_size` at ±0…±3, and `semantic` under each Boundary Embedding Model present in data, under a selected Corpus Embedding Model.
|
||||
- Auto-discover from **single-strategy** Experiments on the hardcoded 10-doc set; newest Experiment fills each doc×Candidate cell; optional exclude chips; incomplete cells are `—` and omitted from that Candidate’s mean/wins.
|
||||
- Stage 1 ranks by **mean composite** (same weights as Experiments “Best”), shows win-count, allows manual override.
|
||||
- Stage 2 duels the two family winners with full metrics + per-doc head-to-head; per-doc matrix lists all Candidates with duel winners highlighted.
|
||||
- Aggregation is client-side from `GET /experiments?limit=…` (default list limit raised; max 500).
|
||||
|
||||
## Considered options
|
||||
|
||||
- Enhance Compare only — rejected (wrong job, crowded Benchmarks UX).
|
||||
- Standalone HTML report only — rejected (weak for override / interactive decision).
|
||||
- Manual pin of every cell — rejected for v1 (too heavy for ~60 cells).
|
||||
458
docs/assets/decision/generate_charts.py
Normal file
458
docs/assets/decision/generate_charts.py
Normal file
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate SVG charts for the final chunking-strategy decision report.
|
||||
|
||||
Reads Experiment aggregates from SQLite (Decision Board rules) and writes
|
||||
SVG files next to this script. Re-run after new Experiments if needed:
|
||||
|
||||
.venv/bin/python docs/assets/decision/generate_charts.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
DB = ROOT / "data" / "chunking_benchmark.db"
|
||||
OUT = Path(__file__).resolve().parent
|
||||
|
||||
DECISION_DOCS = [
|
||||
"bazresi.docx",
|
||||
"customer1.docx",
|
||||
"fire.docx",
|
||||
"general-havades-individuals.doc",
|
||||
"havades.docx",
|
||||
"life-time-individual.docx",
|
||||
"moavenin.docx",
|
||||
"Refah.docx",
|
||||
"website.docx",
|
||||
"lifetime-compensation.docx",
|
||||
]
|
||||
SHORT = {
|
||||
"bazresi.docx": "bazresi",
|
||||
"customer1.docx": "customer1",
|
||||
"fire.docx": "fire",
|
||||
"general-havades-individuals.doc": "havades-ind",
|
||||
"havades.docx": "havades",
|
||||
"life-time-individual.docx": "lifetime-ind",
|
||||
"moavenin.docx": "moavenin",
|
||||
"Refah.docx": "Refah",
|
||||
"website.docx": "website",
|
||||
"lifetime-compensation.docx": "lifetime-comp",
|
||||
}
|
||||
CORPUS = "text-embedding-3-large"
|
||||
|
||||
TEAL_FILL = "#14b8a6"
|
||||
AMBER_FILL = "#f59e0b"
|
||||
INK = "#111827"
|
||||
MUTED = "#6b7280"
|
||||
GRID = "#e5e7eb"
|
||||
BG = "#ffffff"
|
||||
WIN = "#047857"
|
||||
|
||||
|
||||
def esc(s: str) -> str:
|
||||
return (
|
||||
str(s)
|
||||
.replace("&", "&")
|
||||
.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.replace('"', """)
|
||||
)
|
||||
|
||||
|
||||
def composite(m: dict | None) -> float | None:
|
||||
if not m:
|
||||
return None
|
||||
return (
|
||||
(m.get("avg_context_relevance") or 0) * 0.3
|
||||
+ (m.get("avg_answer_similarity") or 0) * 0.4
|
||||
+ (m.get("avg_faithfulness") or 0) * 0.3
|
||||
) * (1 - (m.get("hallucination_rate") or 0))
|
||||
|
||||
|
||||
def load_cells() -> dict[tuple[str, str], dict]:
|
||||
conn = sqlite3.connect(DB)
|
||||
conn.row_factory = sqlite3.Row
|
||||
docs = {r["id"]: r["filename"] for r in conn.execute("SELECT id, filename FROM documents")}
|
||||
rows = []
|
||||
for e in conn.execute(
|
||||
"""
|
||||
SELECT id, document_id, strategies_used, aggregate_metrics, benchmark_config,
|
||||
embedding_model_id, boundary_embedding_model_id, created_at, questions
|
||||
FROM experiments ORDER BY created_at DESC
|
||||
"""
|
||||
):
|
||||
fn = docs.get(e["document_id"])
|
||||
strats = json.loads(e["strategies_used"] or "[]")
|
||||
agg = json.loads(e["aggregate_metrics"] or "{}")
|
||||
cfg = json.loads(e["benchmark_config"] or "{}")
|
||||
corpus = e["embedding_model_id"] or cfg.get("corpus_embedding_model_id")
|
||||
prev = int(cfg.get("neighbor_prev") or 0)
|
||||
nxt = int(cfg.get("neighbor_next") or 0)
|
||||
bound = e["boundary_embedding_model_id"] or cfg.get("boundary_embedding_model_id")
|
||||
if fn not in DECISION_DOCS or corpus != CORPUS or len(strats) != 1:
|
||||
continue
|
||||
strat = strats[0]
|
||||
if strat == "fixed_size":
|
||||
if prev != nxt or prev not in (0, 1, 2, 3):
|
||||
continue
|
||||
cid = f"fixed_size:±{prev}"
|
||||
family = "fixed_size"
|
||||
elif strat == "semantic":
|
||||
if not bound:
|
||||
continue
|
||||
cid = f"semantic:{bound}"
|
||||
family = "semantic"
|
||||
else:
|
||||
continue
|
||||
rows.append(
|
||||
{
|
||||
"fn": fn,
|
||||
"cid": cid,
|
||||
"family": family,
|
||||
"agg": agg,
|
||||
"metrics": agg.get(family) or {},
|
||||
"id": e["id"],
|
||||
}
|
||||
)
|
||||
conn.close()
|
||||
cells: dict[tuple[str, str], dict] = {}
|
||||
for r in rows:
|
||||
key = (r["fn"], r["cid"])
|
||||
if key not in cells:
|
||||
cells[key] = r
|
||||
return cells
|
||||
|
||||
|
||||
def mean(xs: list[float]) -> float | None:
|
||||
return sum(xs) / len(xs) if xs else None
|
||||
|
||||
|
||||
def svg_wrap(w: int, h: int, body: str, title: str) -> str:
|
||||
return f'''<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="{w}" height="{h}" viewBox="0 0 {w} {h}" role="img" aria-label="{esc(title)}">
|
||||
<title>{esc(title)}</title>
|
||||
<rect width="{w}" height="{h}" fill="{BG}"/>
|
||||
{body}
|
||||
</svg>
|
||||
'''
|
||||
|
||||
|
||||
def chart_stage1(cells: dict) -> None:
|
||||
cands = [
|
||||
("fixed_size:±3", "fixed_size ±3", TEAL_FILL, True),
|
||||
("fixed_size:±2", "fixed_size ±2", TEAL_FILL, False),
|
||||
("fixed_size:±1", "fixed_size ±1", TEAL_FILL, False),
|
||||
("semantic:text-embedding-3-large", "semantic @ large", AMBER_FILL, False),
|
||||
("fixed_size:±0", "fixed_size ±0", "#99f6e4", False),
|
||||
("semantic:nomic-embed-text-v2-moe", "semantic @ nomic", "#fcd34d", False),
|
||||
]
|
||||
values = []
|
||||
for cid, label, color, winner in cands:
|
||||
scores = []
|
||||
for fn in DECISION_DOCS:
|
||||
r = cells.get((fn, cid))
|
||||
if r:
|
||||
sc = composite(r["metrics"])
|
||||
if sc is not None:
|
||||
scores.append(sc)
|
||||
values.append((cid, label, color, winner, mean(scores) or 0))
|
||||
|
||||
w, h = 820, 420
|
||||
left, right, top, bottom = 210, 40, 56, 48
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
vmin, vmax = 8.4, 9.2
|
||||
bar_h = plot_h / len(values) * 0.62
|
||||
gap = plot_h / len(values)
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="32" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 1 — mean composite (10 documents)</text>',
|
||||
f'<text x="24" y="50" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Higher is better. Official ranking used by Decision Board. Winner: fixed_size ±3.</text>',
|
||||
]
|
||||
# grid
|
||||
for tick in [8.5, 8.6, 8.7, 8.8, 8.9, 9.0, 9.1, 9.2]:
|
||||
x = left + (tick - vmin) / (vmax - vmin) * plot_w
|
||||
parts.append(f'<line x1="{x:.1f}" y1="{top}" x2="{x:.1f}" y2="{h - bottom}" stroke="{GRID}" stroke-width="1"/>')
|
||||
parts.append(
|
||||
f'<text x="{x:.1f}" y="{h - 18}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">{tick:.1f}</text>'
|
||||
)
|
||||
|
||||
for i, (_, label, color, winner, val) in enumerate(values):
|
||||
y = top + i * gap + (gap - bar_h) / 2
|
||||
bw = (val - vmin) / (vmax - vmin) * plot_w
|
||||
stroke = WIN if winner else "none"
|
||||
sw = 2 if winner else 0
|
||||
parts.append(
|
||||
f'<rect x="{left}" y="{y:.1f}" width="{max(bw, 0):.1f}" height="{bar_h:.1f}" rx="4" fill="{color}" stroke="{stroke}" stroke-width="{sw}"/>'
|
||||
)
|
||||
weight = "700" if winner else "500"
|
||||
parts.append(
|
||||
f'<text x="{left - 10}" y="{y + bar_h * 0.68:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="{weight}" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{left + bw + 8:.1f}" y="{y + bar_h * 0.68:.1f}" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="{INK}">{val:.3f}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage1-mean-composite.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 1 mean composite"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_stage2(cells: dict) -> None:
|
||||
fs_id = "fixed_size:±3"
|
||||
sem_id = "semantic:text-embedding-3-large"
|
||||
pairs = []
|
||||
for fn in DECISION_DOCS:
|
||||
a = composite(cells[(fn, fs_id)]["metrics"])
|
||||
b = composite(cells[(fn, sem_id)]["metrics"])
|
||||
pairs.append((SHORT[fn], a, b))
|
||||
|
||||
w, h = 920, 460
|
||||
left, right, top, bottom = 52, 24, 64, 88
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(pairs)
|
||||
slot = plot_w / n
|
||||
bar_w = slot * 0.32
|
||||
vmin, vmax = 5.8, 10.0
|
||||
|
||||
def y_of(v: float) -> float:
|
||||
return top + (1 - (v - vmin) / (vmax - vmin)) * plot_h
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 2 — per document (fixed_size ±3 vs semantic @ large)</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Mean composite still favors fixed_size. Semantic wins 6 of 10 docs, but by smaller margins except website/customer1.</text>',
|
||||
f'<rect x="620" y="14" width="12" height="12" rx="2" fill="{TEAL_FILL}"/>',
|
||||
f'<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">fixed_size ±3</text>',
|
||||
f'<rect x="760" y="14" width="12" height="12" rx="2" fill="{AMBER_FILL}"/>',
|
||||
f'<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">semantic @ large</text>',
|
||||
]
|
||||
for tick in [6, 7, 8, 9, 10]:
|
||||
y = y_of(tick)
|
||||
parts.append(f'<line x1="{left}" y1="{y:.1f}" x2="{w - right}" y2="{y:.1f}" stroke="{GRID}" stroke-width="1"/>')
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + 4:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">{tick}</text>'
|
||||
)
|
||||
|
||||
for i, (label, a, b) in enumerate(pairs):
|
||||
cx = left + i * slot + slot / 2
|
||||
xa = cx - bar_w - 3
|
||||
xb = cx + 3
|
||||
ha = plot_h - (y_of(a) - top)
|
||||
hb = plot_h - (y_of(b) - top)
|
||||
parts.append(f'<rect x="{xa:.1f}" y="{y_of(a):.1f}" width="{bar_w:.1f}" height="{ha:.1f}" rx="3" fill="{TEAL_FILL}"/>')
|
||||
parts.append(f'<rect x="{xb:.1f}" y="{y_of(b):.1f}" width="{bar_w:.1f}" height="{hb:.1f}" rx="3" fill="{AMBER_FILL}"/>')
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 36}" text-anchor="end" transform="rotate(-32 {cx:.1f} {h - 36})" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-per-document.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 per document"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_delta(cells: dict) -> None:
|
||||
fs_id = "fixed_size:±3"
|
||||
sem_id = "semantic:text-embedding-3-large"
|
||||
deltas = []
|
||||
for fn in DECISION_DOCS:
|
||||
a = composite(cells[(fn, fs_id)]["metrics"])
|
||||
b = composite(cells[(fn, sem_id)]["metrics"])
|
||||
deltas.append((SHORT[fn], a - b))
|
||||
deltas.sort(key=lambda x: x[1], reverse=True)
|
||||
|
||||
w, h = 820, 440
|
||||
left, right, top, bottom = 120, 56, 56, 36
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(deltas)
|
||||
gap = plot_h / n
|
||||
bar_h = gap * 0.62
|
||||
max_abs = max(abs(d) for _, d in deltas)
|
||||
# fire is 2.58, others < 1.2 — use 2.8
|
||||
max_abs = 2.8
|
||||
zero_x = left + plot_w / 2
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Margin: fixed_size ±3 minus semantic @ large</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Teal = fixed_size wins the document. Amber = semantic wins. Mean ranking is driven by large teal bars (especially fire).</text>',
|
||||
f'<line x1="{zero_x:.1f}" y1="{top}" x2="{zero_x:.1f}" y2="{h - bottom}" stroke="{INK}" stroke-width="1.2"/>',
|
||||
f'<text x="{left}" y="{h - 12}" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">semantic better</text>',
|
||||
f'<text x="{w - right}" y="{h - 12}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">fixed_size better</text>',
|
||||
]
|
||||
for i, (label, d) in enumerate(deltas):
|
||||
y = top + i * gap + (gap - bar_h) / 2
|
||||
bw = abs(d) / max_abs * (plot_w / 2)
|
||||
if d >= 0:
|
||||
x = zero_x
|
||||
color = TEAL_FILL
|
||||
tx = x + bw + 6
|
||||
anchor = "start"
|
||||
else:
|
||||
x = zero_x - bw
|
||||
color = AMBER_FILL
|
||||
tx = x - 6
|
||||
anchor = "end"
|
||||
parts.append(f'<rect x="{x:.1f}" y="{y:.1f}" width="{bw:.1f}" height="{bar_h:.1f}" rx="3" fill="{color}"/>')
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + bar_h * 0.7:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{tx:.1f}" y="{y + bar_h * 0.7:.1f}" text-anchor="{anchor}" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="{INK}">{d:+.2f}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-margins.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 margins"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_heatmap(cells: dict) -> None:
|
||||
cands = [
|
||||
("fixed_size:±0", "±0"),
|
||||
("fixed_size:±1", "±1"),
|
||||
("fixed_size:±2", "±2"),
|
||||
("fixed_size:±3", "±3"),
|
||||
("semantic:text-embedding-3-large", "sem@large"),
|
||||
("semantic:nomic-embed-text-v2-moe", "sem@nomic"),
|
||||
]
|
||||
scores: list[list[float | None]] = []
|
||||
for fn in DECISION_DOCS:
|
||||
row = []
|
||||
for cid, _ in cands:
|
||||
r = cells.get((fn, cid))
|
||||
row.append(composite(r["metrics"]) if r else None)
|
||||
scores.append(row)
|
||||
|
||||
cell_w, cell_h = 78, 32
|
||||
left, top = 128, 72
|
||||
w = left + cell_w * len(cands) + 24
|
||||
h = top + cell_h * len(DECISION_DOCS) + 36
|
||||
|
||||
def color_for(s: float | None) -> tuple[str, str]:
|
||||
if s is None:
|
||||
return "#f3f4f6", MUTED
|
||||
# 6.2 .. 9.8
|
||||
t = max(0.0, min(1.0, (s - 7.0) / (9.8 - 7.0)))
|
||||
# pale rose -> amber -> teal
|
||||
if t < 0.5:
|
||||
u = t * 2
|
||||
r = int(251 + (20 - 251) * 0)
|
||||
# interpolate rose 251,113,133 -> amber 245,158,11
|
||||
rr = int(251 + (245 - 251) * u)
|
||||
gg = int(113 + (158 - 113) * u)
|
||||
bb = int(133 + (11 - 133) * u)
|
||||
else:
|
||||
u = (t - 0.5) * 2
|
||||
rr = int(245 + (13 - 245) * u)
|
||||
gg = int(158 + (148 - 158) * u)
|
||||
bb = int(11 + (136 - 11) * u)
|
||||
bg = f"#{rr:02x}{gg:02x}{bb:02x}"
|
||||
fg = "#111827" if t < 0.72 else "#f9fafb"
|
||||
return bg, fg
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Candidate heatmap — composite by document</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Darker teal = stronger. Semantic @ large collapses on fire; ±1–±3 stay high across the set.</text>',
|
||||
]
|
||||
for j, (_, lab) in enumerate(cands):
|
||||
x = left + j * cell_w + cell_w / 2
|
||||
parts.append(
|
||||
f'<text x="{x:.1f}" y="{top - 10}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{esc(lab)}</text>'
|
||||
)
|
||||
for i, fn in enumerate(DECISION_DOCS):
|
||||
y = top + i * cell_h
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + cell_h * 0.65:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{INK}">{esc(SHORT[fn])}</text>'
|
||||
)
|
||||
for j, val in enumerate(scores[i]):
|
||||
x = left + j * cell_w
|
||||
bg, fg = color_for(val)
|
||||
parts.append(f'<rect x="{x}" y="{y}" width="{cell_w - 3}" height="{cell_h - 3}" rx="4" fill="{bg}"/>')
|
||||
txt = "—" if val is None else f"{val:.2f}"
|
||||
parts.append(
|
||||
f'<text x="{x + (cell_w - 3) / 2:.1f}" y="{y + cell_h * 0.62:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{fg}">{txt}</text>'
|
||||
)
|
||||
|
||||
(OUT / "heatmap-candidates.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Candidate heatmap"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_metrics(cells: dict) -> None:
|
||||
"""Grouped bars: four judge metrics for the two stage-2 winners."""
|
||||
def bag(cid: str, family: str) -> dict[str, float]:
|
||||
keys = [
|
||||
"avg_context_relevance",
|
||||
"avg_answer_similarity",
|
||||
"avg_faithfulness",
|
||||
"hallucination_rate",
|
||||
]
|
||||
acc = {k: [] for k in keys}
|
||||
for fn in DECISION_DOCS:
|
||||
m = cells[(fn, cid)]["metrics"]
|
||||
for k in keys:
|
||||
acc[k].append(m[k])
|
||||
return {k: sum(v) / len(v) for k, v in acc.items()}
|
||||
|
||||
fs = bag("fixed_size:±3", "fixed_size")
|
||||
sem = bag("semantic:text-embedding-3-large", "semantic")
|
||||
labels = [
|
||||
("Context relevance", fs["avg_context_relevance"], sem["avg_context_relevance"], False),
|
||||
("Answer similarity", fs["avg_answer_similarity"], sem["avg_answer_similarity"], False),
|
||||
("Faithfulness", fs["avg_faithfulness"], sem["avg_faithfulness"], False),
|
||||
("Hallucination % (lower better)", fs["hallucination_rate"] * 100, sem["hallucination_rate"] * 100, True),
|
||||
]
|
||||
|
||||
w, h = 820, 380
|
||||
left, right, top, bottom = 52, 24, 64, 48
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(labels)
|
||||
slot = plot_w / n
|
||||
bar_w = slot * 0.28
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 2 winners — mean judge metrics</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">fixed_size ±3 leads on all four metrics, including lower hallucination.</text>',
|
||||
f'<rect x="620" y="14" width="12" height="12" rx="2" fill="{TEAL_FILL}"/>',
|
||||
f'<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">fixed_size ±3</text>',
|
||||
f'<rect x="760" y="14" width="12" height="12" rx="2" fill="{AMBER_FILL}"/>',
|
||||
f'<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">semantic @ large</text>',
|
||||
]
|
||||
|
||||
for i, (lab, a, b, is_pct) in enumerate(labels):
|
||||
vmax = 10 if not is_pct else max(a, b) * 1.35
|
||||
cx = left + i * slot + slot / 2
|
||||
xa = cx - bar_w - 3
|
||||
xb = cx + 3
|
||||
|
||||
def bar(x: float, val: float, color: str) -> str:
|
||||
bh = val / vmax * plot_h
|
||||
y = top + plot_h - bh
|
||||
return f'<rect x="{x:.1f}" y="{y:.1f}" width="{bar_w:.1f}" height="{bh:.1f}" rx="3" fill="{color}"/>'
|
||||
|
||||
parts.append(bar(xa, a, TEAL_FILL))
|
||||
parts.append(bar(xb, b, AMBER_FILL))
|
||||
fmt = (lambda v: f"{v:.1f}%") if is_pct else (lambda v: f"{v:.2f}")
|
||||
ya = top + plot_h - a / vmax * plot_h - 6
|
||||
yb = top + plot_h - b / vmax * plot_h - 6
|
||||
parts.append(
|
||||
f'<text x="{xa + bar_w / 2:.1f}" y="{ya:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{fmt(a)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{xb + bar_w / 2:.1f}" y="{yb:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{fmt(b)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 16}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{esc(lab)}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-metrics.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 mean metrics"), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cells = load_cells()
|
||||
chart_stage1(cells)
|
||||
chart_stage2(cells)
|
||||
chart_delta(cells)
|
||||
chart_heatmap(cells)
|
||||
chart_metrics(cells)
|
||||
print(f"Wrote SVGs in {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
143
docs/assets/decision/heatmap-candidates.svg
Normal file
143
docs/assets/decision/heatmap-candidates.svg
Normal file
@@ -0,0 +1,143 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="620" height="428" viewBox="0 0 620 428" role="img" aria-label="Candidate heatmap">
|
||||
<title>Candidate heatmap</title>
|
||||
<rect width="620" height="428" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Candidate heatmap — composite by document</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Darker teal = stronger. Semantic @ large collapses on fire; ±1–±3 stay high across the set.</text>
|
||||
<text x="167.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±0</text>
|
||||
<text x="245.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±1</text>
|
||||
<text x="323.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±2</text>
|
||||
<text x="401.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±3</text>
|
||||
<text x="479.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">sem@large</text>
|
||||
<text x="557.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">sem@nomic</text>
|
||||
<text x="120" y="92.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">bazresi</text>
|
||||
<rect x="128" y="72" width="75" height="29" rx="4" fill="#e19d15"/>
|
||||
<text x="165.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.52</text>
|
||||
<rect x="206" y="72" width="75" height="29" rx="4" fill="#58975f"/>
|
||||
<text x="243.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.34</text>
|
||||
<rect x="284" y="72" width="75" height="29" rx="4" fill="#58975f"/>
|
||||
<text x="321.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.34</text>
|
||||
<rect x="362" y="72" width="75" height="29" rx="4" fill="#549761"/>
|
||||
<text x="399.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.37</text>
|
||||
<rect x="440" y="72" width="75" height="29" rx="4" fill="#4e9664"/>
|
||||
<text x="477.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.40</text>
|
||||
<rect x="518" y="72" width="75" height="29" rx="4" fill="#7c984b"/>
|
||||
<text x="555.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.12</text>
|
||||
<text x="120" y="124.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">customer1</text>
|
||||
<rect x="128" y="104" width="75" height="29" rx="4" fill="#9f9a39"/>
|
||||
<text x="165.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.92</text>
|
||||
<rect x="206" y="104" width="75" height="29" rx="4" fill="#96993e"/>
|
||||
<text x="243.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.97</text>
|
||||
<rect x="284" y="104" width="75" height="29" rx="4" fill="#97993d"/>
|
||||
<text x="321.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.96</text>
|
||||
<rect x="362" y="104" width="75" height="29" rx="4" fill="#ec9d0f"/>
|
||||
<text x="399.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.45</text>
|
||||
<rect x="440" y="104" width="75" height="29" rx="4" fill="#96993e"/>
|
||||
<text x="477.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.97</text>
|
||||
<rect x="518" y="104" width="75" height="29" rx="4" fill="#f88255"/>
|
||||
<text x="555.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.54</text>
|
||||
<text x="120" y="156.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">fire</text>
|
||||
<rect x="128" y="136" width="75" height="29" rx="4" fill="#f69426"/>
|
||||
<text x="165.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.09</text>
|
||||
<rect x="206" y="136" width="75" height="29" rx="4" fill="#f59c10"/>
|
||||
<text x="243.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.34</text>
|
||||
<rect x="284" y="136" width="75" height="29" rx="4" fill="#f68f33"/>
|
||||
<text x="321.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.93</text>
|
||||
<rect x="362" y="136" width="75" height="29" rx="4" fill="#b19b2f"/>
|
||||
<text x="399.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.81</text>
|
||||
<rect x="440" y="136" width="75" height="29" rx="4" fill="#fb7185"/>
|
||||
<text x="477.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">6.23</text>
|
||||
<rect x="518" y="136" width="75" height="29" rx="4" fill="#fa7674"/>
|
||||
<text x="555.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.19</text>
|
||||
<text x="120" y="188.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades-ind</text>
|
||||
<rect x="128" y="168" width="75" height="29" rx="4" fill="#f78b3c"/>
|
||||
<text x="165.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.83</text>
|
||||
<rect x="206" y="168" width="75" height="29" rx="4" fill="#c99c22"/>
|
||||
<text x="243.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.66</text>
|
||||
<rect x="284" y="168" width="75" height="29" rx="4" fill="#7e984a"/>
|
||||
<text x="321.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.12</text>
|
||||
<rect x="362" y="168" width="75" height="29" rx="4" fill="#6f9852"/>
|
||||
<text x="399.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.21</text>
|
||||
<rect x="440" y="168" width="75" height="29" rx="4" fill="#d99c19"/>
|
||||
<text x="477.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.56</text>
|
||||
<rect x="518" y="168" width="75" height="29" rx="4" fill="#e09d16"/>
|
||||
<text x="555.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.53</text>
|
||||
<text x="120" y="220.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades</text>
|
||||
<rect x="128" y="200" width="75" height="29" rx="4" fill="#f59b12"/>
|
||||
<text x="165.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.31</text>
|
||||
<rect x="206" y="200" width="75" height="29" rx="4" fill="#ef9d0d"/>
|
||||
<text x="243.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.43</text>
|
||||
<rect x="284" y="200" width="75" height="29" rx="4" fill="#f49d0b"/>
|
||||
<text x="321.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.40</text>
|
||||
<rect x="362" y="200" width="75" height="29" rx="4" fill="#e79d12"/>
|
||||
<text x="399.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.48</text>
|
||||
<rect x="440" y="200" width="75" height="29" rx="4" fill="#f69030"/>
|
||||
<text x="477.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.97</text>
|
||||
<rect x="518" y="200" width="75" height="29" rx="4" fill="#f29d0c"/>
|
||||
<text x="555.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.42</text>
|
||||
<text x="120" y="252.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-ind</text>
|
||||
<rect x="128" y="232" width="75" height="29" rx="4" fill="#6d9854"/>
|
||||
<text x="165.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.22</text>
|
||||
<rect x="206" y="232" width="75" height="29" rx="4" fill="#4b9666"/>
|
||||
<text x="243.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.42</text>
|
||||
<rect x="284" y="232" width="75" height="29" rx="4" fill="#489667"/>
|
||||
<text x="321.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.44</text>
|
||||
<rect x="362" y="232" width="75" height="29" rx="4" fill="#479668"/>
|
||||
<text x="399.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.45</text>
|
||||
<rect x="440" y="232" width="75" height="29" rx="4" fill="#3d966d"/>
|
||||
<text x="477.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.50</text>
|
||||
<rect x="518" y="232" width="75" height="29" rx="4" fill="#4e9664"/>
|
||||
<text x="555.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.41</text>
|
||||
<text x="120" y="284.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">moavenin</text>
|
||||
<rect x="128" y="264" width="75" height="29" rx="4" fill="#879945"/>
|
||||
<text x="165.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.06</text>
|
||||
<rect x="206" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="243.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="284" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="321.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="362" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="399.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="440" y="264" width="75" height="29" rx="4" fill="#d09c1e"/>
|
||||
<text x="477.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.62</text>
|
||||
<rect x="518" y="264" width="75" height="29" rx="4" fill="#f59a14"/>
|
||||
<text x="555.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.29</text>
|
||||
<text x="120" y="316.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">Refah</text>
|
||||
<rect x="128" y="296" width="75" height="29" rx="4" fill="#459669"/>
|
||||
<text x="165.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.46</text>
|
||||
<rect x="206" y="296" width="75" height="29" rx="4" fill="#1c947f"/>
|
||||
<text x="243.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.71</text>
|
||||
<rect x="284" y="296" width="75" height="29" rx="4" fill="#1e947e"/>
|
||||
<text x="321.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.70</text>
|
||||
<rect x="362" y="296" width="75" height="29" rx="4" fill="#20947d"/>
|
||||
<text x="399.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.68</text>
|
||||
<rect x="440" y="296" width="75" height="29" rx="4" fill="#0d9488"/>
|
||||
<text x="477.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.81</text>
|
||||
<rect x="518" y="296" width="75" height="29" rx="4" fill="#349572"/>
|
||||
<text x="555.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.56</text>
|
||||
<text x="120" y="348.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">website</text>
|
||||
<rect x="128" y="328" width="75" height="29" rx="4" fill="#f8864b"/>
|
||||
<text x="165.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.67</text>
|
||||
<rect x="206" y="328" width="75" height="29" rx="4" fill="#f78e35"/>
|
||||
<text x="243.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.91</text>
|
||||
<rect x="284" y="328" width="75" height="29" rx="4" fill="#e29d15"/>
|
||||
<text x="321.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.51</text>
|
||||
<rect x="362" y="328" width="75" height="29" rx="4" fill="#ea9d10"/>
|
||||
<text x="399.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.47</text>
|
||||
<rect x="440" y="328" width="75" height="29" rx="4" fill="#879945"/>
|
||||
<text x="477.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.06</text>
|
||||
<rect x="518" y="328" width="75" height="29" rx="4" fill="#e59d13"/>
|
||||
<text x="555.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.49</text>
|
||||
<text x="120" y="380.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-comp</text>
|
||||
<rect x="128" y="360" width="75" height="29" rx="4" fill="#93993f"/>
|
||||
<text x="165.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.99</text>
|
||||
<rect x="206" y="360" width="75" height="29" rx="4" fill="#bd9b28"/>
|
||||
<text x="243.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.73</text>
|
||||
<rect x="284" y="360" width="75" height="29" rx="4" fill="#bd9b28"/>
|
||||
<text x="321.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.73</text>
|
||||
<rect x="362" y="360" width="75" height="29" rx="4" fill="#b99b2b"/>
|
||||
<text x="399.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.76</text>
|
||||
<rect x="440" y="360" width="75" height="29" rx="4" fill="#7c984c"/>
|
||||
<text x="477.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.13</text>
|
||||
<rect x="518" y="360" width="75" height="29" rx="4" fill="#98993d"/>
|
||||
<text x="555.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.96</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 16 KiB |
41
docs/assets/decision/stage1-mean-composite.svg
Normal file
41
docs/assets/decision/stage1-mean-composite.svg
Normal file
@@ -0,0 +1,41 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="420" viewBox="0 0 820 420" role="img" aria-label="Stage 1 mean composite">
|
||||
<title>Stage 1 mean composite</title>
|
||||
<rect width="820" height="420" fill="#ffffff"/>
|
||||
<text x="24" y="32" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 1 — mean composite (10 documents)</text>
|
||||
<text x="24" y="50" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Higher is better. Official ranking used by Decision Board. Winner: fixed_size ±3.</text>
|
||||
<line x1="281.2" y1="56" x2="281.2" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="281.2" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.5</text>
|
||||
<line x1="352.5" y1="56" x2="352.5" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="352.5" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.6</text>
|
||||
<line x1="423.7" y1="56" x2="423.7" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="423.7" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.7</text>
|
||||
<line x1="495.0" y1="56" x2="495.0" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="495.0" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.8</text>
|
||||
<line x1="566.3" y1="56" x2="566.3" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="566.3" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.9</text>
|
||||
<line x1="637.5" y1="56" x2="637.5" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="637.5" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.0</text>
|
||||
<line x1="708.8" y1="56" x2="708.8" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="708.8" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.1</text>
|
||||
<line x1="780.0" y1="56" x2="780.0" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="780.0" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.2</text>
|
||||
<rect x="210" y="66.0" width="456.2" height="32.7" rx="4" fill="#14b8a6" stroke="#047857" stroke-width="2"/>
|
||||
<text x="200" y="88.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="700" fill="#111827">fixed_size ±3</text>
|
||||
<text x="674.2" y="88.2" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">9.040</text>
|
||||
<rect x="210" y="118.7" width="418.4" height="32.7" rx="4" fill="#14b8a6" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="140.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±2</text>
|
||||
<text x="636.4" y="140.9" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.987</text>
|
||||
<rect x="210" y="171.3" width="374.8" height="32.7" rx="4" fill="#14b8a6" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="193.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±1</text>
|
||||
<text x="592.8" y="193.5" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.926</text>
|
||||
<rect x="210" y="224.0" width="232.8" height="32.7" rx="4" fill="#f59e0b" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="246.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">semantic @ large</text>
|
||||
<text x="450.8" y="246.2" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.727</text>
|
||||
<rect x="210" y="276.7" width="146.7" height="32.7" rx="4" fill="#99f6e4" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="298.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±0</text>
|
||||
<text x="364.7" y="298.9" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.606</text>
|
||||
<rect x="210" y="329.3" width="107.4" height="32.7" rx="4" fill="#fcd34d" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="351.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">semantic @ nomic</text>
|
||||
<text x="325.4" y="351.5" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.551</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
40
docs/assets/decision/stage2-margins.svg
Normal file
40
docs/assets/decision/stage2-margins.svg
Normal file
@@ -0,0 +1,40 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="440" viewBox="0 0 820 440" role="img" aria-label="Stage 2 margins">
|
||||
<title>Stage 2 margins</title>
|
||||
<rect width="820" height="440" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Margin: fixed_size ±3 minus semantic @ large</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Teal = fixed_size wins the document. Amber = semantic wins. Mean ranking is driven by large teal bars (especially fire).</text>
|
||||
<line x1="442.0" y1="56" x2="442.0" y2="404" stroke="#111827" stroke-width="1.2"/>
|
||||
<text x="120" y="428" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">semantic better</text>
|
||||
<text x="764" y="428" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">fixed_size better</text>
|
||||
<rect x="442.0" y="62.6" width="296.9" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="77.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fire</text>
|
||||
<text x="744.9" y="77.7" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+2.58</text>
|
||||
<rect x="442.0" y="97.4" width="128.1" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="112.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">moavenin</text>
|
||||
<text x="576.1" y="112.5" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+1.11</text>
|
||||
<rect x="442.0" y="132.2" width="73.7" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="147.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades-ind</text>
|
||||
<text x="521.7" y="147.3" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+0.64</text>
|
||||
<rect x="442.0" y="167.0" width="59.1" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="182.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades</text>
|
||||
<text x="507.1" y="182.1" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+0.51</text>
|
||||
<rect x="437.5" y="201.8" width="4.5" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="216.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">bazresi</text>
|
||||
<text x="431.5" y="216.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.04</text>
|
||||
<rect x="435.1" y="236.6" width="6.9" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="251.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-ind</text>
|
||||
<text x="429.1" y="251.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.06</text>
|
||||
<rect x="426.5" y="271.4" width="15.5" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="286.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Refah</text>
|
||||
<text x="420.5" y="286.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.13</text>
|
||||
<rect x="399.7" y="306.2" width="42.3" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="321.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-comp</text>
|
||||
<text x="393.7" y="321.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.37</text>
|
||||
<rect x="382.0" y="341.0" width="60.0" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="356.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">customer1</text>
|
||||
<text x="376.0" y="356.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.52</text>
|
||||
<rect x="373.9" y="375.8" width="68.1" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="390.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">website</text>
|
||||
<text x="367.9" y="390.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.59</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.4 KiB |
31
docs/assets/decision/stage2-metrics.svg
Normal file
31
docs/assets/decision/stage2-metrics.svg
Normal file
@@ -0,0 +1,31 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="380" viewBox="0 0 820 380" role="img" aria-label="Stage 2 mean metrics">
|
||||
<title>Stage 2 mean metrics</title>
|
||||
<rect width="820" height="380" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 2 winners — mean judge metrics</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">fixed_size ±3 leads on all four metrics, including lower hallucination.</text>
|
||||
<rect x="620" y="14" width="12" height="12" rx="2" fill="#14b8a6"/>
|
||||
<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fixed_size ±3</text>
|
||||
<rect x="760" y="14" width="12" height="12" rx="2" fill="#f59e0b"/>
|
||||
<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">semantic @ large</text>
|
||||
<rect x="89.9" y="80.0" width="52.1" height="252.0" rx="3" fill="#14b8a6"/>
|
||||
<rect x="148.0" y="80.7" width="52.1" height="251.3" rx="3" fill="#f59e0b"/>
|
||||
<text x="116.0" y="74.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.40</text>
|
||||
<text x="174.0" y="74.7" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.38</text>
|
||||
<text x="145.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Context relevance</text>
|
||||
<rect x="275.9" y="92.1" width="52.1" height="239.9" rx="3" fill="#14b8a6"/>
|
||||
<rect x="334.0" y="95.3" width="52.1" height="236.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="302.0" y="86.1" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.95</text>
|
||||
<text x="360.0" y="89.3" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.83</text>
|
||||
<text x="331.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Answer similarity</text>
|
||||
<rect x="461.9" y="78.2" width="52.1" height="253.8" rx="3" fill="#14b8a6"/>
|
||||
<rect x="520.0" y="82.7" width="52.1" height="249.3" rx="3" fill="#f59e0b"/>
|
||||
<text x="488.0" y="72.2" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.47</text>
|
||||
<text x="546.0" y="76.7" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.30</text>
|
||||
<text x="517.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Faithfulness</text>
|
||||
<rect x="647.9" y="241.0" width="52.1" height="91.0" rx="3" fill="#14b8a6"/>
|
||||
<rect x="706.0" y="133.5" width="52.1" height="198.5" rx="3" fill="#f59e0b"/>
|
||||
<text x="674.0" y="235.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">2.2%</text>
|
||||
<text x="732.0" y="127.5" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">4.8%</text>
|
||||
<text x="703.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Hallucination % (lower better)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.3 KiB |
51
docs/assets/decision/stage2-per-document.svg
Normal file
51
docs/assets/decision/stage2-per-document.svg
Normal file
@@ -0,0 +1,51 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="920" height="460" viewBox="0 0 920 460" role="img" aria-label="Stage 2 per document">
|
||||
<title>Stage 2 per document</title>
|
||||
<rect width="920" height="460" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 2 — per document (fixed_size ±3 vs semantic @ large)</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Mean composite still favors fixed_size. Semantic wins 6 of 10 docs, but by smaller margins except website/customer1.</text>
|
||||
<rect x="620" y="14" width="12" height="12" rx="2" fill="#14b8a6"/>
|
||||
<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fixed_size ±3</text>
|
||||
<rect x="760" y="14" width="12" height="12" rx="2" fill="#f59e0b"/>
|
||||
<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">semantic @ large</text>
|
||||
<line x1="52" y1="357.3" x2="896" y2="357.3" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="361.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">6</text>
|
||||
<line x1="52" y1="284.0" x2="896" y2="284.0" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="288.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">7</text>
|
||||
<line x1="52" y1="210.7" x2="896" y2="210.7" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="214.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8</text>
|
||||
<line x1="52" y1="137.3" x2="896" y2="137.3" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="141.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9</text>
|
||||
<line x1="52" y1="64.0" x2="896" y2="64.0" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="68.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">10</text>
|
||||
<rect x="64.2" y="110.5" width="27.0" height="261.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="97.2" y="107.6" width="27.0" height="264.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="94.2" y="424" text-anchor="end" transform="rotate(-32 94.2 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">bazresi</text>
|
||||
<rect x="148.6" y="177.6" width="27.0" height="194.4" rx="3" fill="#14b8a6"/>
|
||||
<rect x="181.6" y="139.3" width="27.0" height="232.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="178.6" y="424" text-anchor="end" transform="rotate(-32 178.6 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">customer1</text>
|
||||
<rect x="233.0" y="151.3" width="27.0" height="220.7" rx="3" fill="#14b8a6"/>
|
||||
<rect x="266.0" y="340.6" width="27.0" height="31.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="263.0" y="424" text-anchor="end" transform="rotate(-32 263.0 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">fire</text>
|
||||
<rect x="317.4" y="122.3" width="27.0" height="249.7" rx="3" fill="#14b8a6"/>
|
||||
<rect x="350.4" y="169.3" width="27.0" height="202.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="347.4" y="424" text-anchor="end" transform="rotate(-32 347.4 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades-ind</text>
|
||||
<rect x="401.8" y="175.2" width="27.0" height="196.8" rx="3" fill="#14b8a6"/>
|
||||
<rect x="434.8" y="212.9" width="27.0" height="159.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="431.8" y="424" text-anchor="end" transform="rotate(-32 431.8 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades</text>
|
||||
<rect x="486.2" y="104.7" width="27.0" height="267.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="519.2" y="100.3" width="27.0" height="271.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="516.2" y="424" text-anchor="end" transform="rotate(-32 516.2 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-ind</text>
|
||||
<rect x="570.6" y="83.4" width="27.0" height="288.6" rx="3" fill="#14b8a6"/>
|
||||
<rect x="603.6" y="165.1" width="27.0" height="206.9" rx="3" fill="#f59e0b"/>
|
||||
<text x="600.6" y="424" text-anchor="end" transform="rotate(-32 600.6 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">moavenin</text>
|
||||
<rect x="655.0" y="87.5" width="27.0" height="284.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="688.0" y="77.6" width="27.0" height="294.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="685.0" y="424" text-anchor="end" transform="rotate(-32 685.0 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">Refah</text>
|
||||
<rect x="739.4" y="176.5" width="27.0" height="195.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="772.4" y="133.1" width="27.0" height="238.9" rx="3" fill="#f59e0b"/>
|
||||
<text x="769.4" y="424" text-anchor="end" transform="rotate(-32 769.4 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">website</text>
|
||||
<rect x="823.8" y="154.9" width="27.0" height="217.1" rx="3" fill="#14b8a6"/>
|
||||
<rect x="856.8" y="127.9" width="27.0" height="244.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="853.8" y="424" text-anchor="end" transform="rotate(-32 853.8 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-comp</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.1 KiB |
149
docs/final-chunking-strategy-decision.md
Normal file
149
docs/final-chunking-strategy-decision.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# Chunking Strategy — Final Decision
|
||||
|
||||
**Date:** 17 August 2026
|
||||
**Decision:** Adopt **`fixed_size`** as the chunking Strategy family.
|
||||
**Not decided here:** Neighbor Expansion level **±N** (follow-up).
|
||||
**Corpus Embedding Model:** `text-embedding-3-large`
|
||||
**Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe)
|
||||
**Questions:** 201 per Candidate (nomic semantic: 195)
|
||||
|
||||
**One-line close:** **`fixed_size` wins** on official mean composite (**9.040** vs **8.727**). Pick **±N later**, but **not ±0**.
|
||||
|
||||
Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board:
|
||||
|
||||
```text
|
||||
(0.3 × context relevance + 0.4 × answer similarity + 0.3 × faithfulness)
|
||||
× (1 − hallucination rate)
|
||||
```
|
||||
|
||||
Regenerate figures after new Experiments:
|
||||
|
||||
```bash
|
||||
.venv/bin/python docs/assets/decision/generate_charts.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Why `fixed_size` is the winner
|
||||
|
||||
Decision Board ranks by **mean composite**, not by “how many documents won.” Stage 1 auto-picks **`fixed_size ±3`** vs **`semantic @ text-embedding-3-large`**. Stage 2 mean composite is **+0.313** for `fixed_size`.
|
||||
|
||||
|
||||
|
||||
| Candidate | Mean composite | Context | Similarity | Faithfulness | Hallucination | Docs |
|
||||
|-----------|----------------|---------|------------|--------------|---------------|------|
|
||||
| **fixed_size ±3** | **9.040** | 9.40 | 8.95 | 9.47 | 2.2% | 10/10 |
|
||||
| fixed_size ±2 | 8.987 | 9.43 | 8.93 | 9.44 | 2.7% | 10/10 |
|
||||
| fixed_size ±1 | 8.926 | 9.38 | 8.89 | 9.42 | 3.0% | 10/10 |
|
||||
| **semantic @ text-embedding-3-large** | **8.727** | 9.38 | 8.83 | 9.30 | 4.8% | 10/10 |
|
||||
| fixed_size ±0 | 8.606 | 9.18 | 8.79 | 9.25 | 4.9% | 10/10 |
|
||||
| semantic @ nomic-embed-text-v2-moe | 8.551 | 9.19 | 8.65 | 9.14 | 4.7% | 10/10 |
|
||||
|
||||
`fixed_size ±1`, `±2`, and `±3` all beat the best semantic Candidate on **mean**. **`±0` (no Neighbor Expansion) does not** (8.606 vs 8.727). Shipping `fixed_size` without expansion would weaken this family decision.
|
||||
|
||||
---
|
||||
|
||||
## 2. Judge metrics (stage 2 winners)
|
||||
|
||||
`fixed_size ±3` leads on every judge metric, including **lower hallucination**.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Per-document showdown
|
||||
|
||||
Mean ranking still favors `fixed_size`. **Head-to-head at ±3: semantic 6 / `fixed_size` 4.** Semantic’s six wins are mostly **small**; `fixed_size`’s four wins are **larger**, especially **fire**.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
| Document | Questions | fixed_size ±3 | semantic @ large | Winner |
|
||||
|----------|-----------|---------------|------------------|--------|
|
||||
| fire.docx | 20 | **8.810** | 6.228 | **fixed_size** (+2.58) |
|
||||
| moavenin.docx | 20 | **9.735** | 8.621 | **fixed_size** (+1.11) |
|
||||
| general-havades-individuals.doc | 20 | **9.205** | 8.564 | **fixed_size** (+0.64) |
|
||||
| havades.docx | 20 | **8.483** | 7.969 | **fixed_size** (+0.51) |
|
||||
| bazresi.docx | 16 | 9.366 | **9.405** | semantic (−0.04) |
|
||||
| life-time-individual.docx | 20 | 9.445 | **9.505** | semantic (−0.06) |
|
||||
| Refah.docx | 20 | 9.680 | **9.815** | semantic (−0.14) |
|
||||
| lifetime-compensation.docx | 15 | 8.761 | **9.129** | semantic (−0.37) |
|
||||
| customer1.docx | 20 | 8.451 | **8.973** | semantic (−0.52) |
|
||||
| website.docx | 30 | 8.466 | **9.058** | semantic (−0.59) |
|
||||
|
||||
Win-count would pick semantic. **Official product ranking (mean composite) picks `fixed_size`.** This memo follows the Decision Board rule (ADR-0026).
|
||||
|
||||
---
|
||||
|
||||
## 4. Full Candidate heatmap
|
||||
|
||||
Every cell is a newest single-strategy Experiment under `text-embedding-3-large`. Semantic @ large **collapses on fire**; `fixed_size ±1…±3` stay high across the set.
|
||||
|
||||
|
||||
|
||||
Best `fixed_size` ±N **by document** (does not change the family call):
|
||||
|
||||
| ±N | Documents where it is the best fixed_size cell |
|
||||
|----|------------------------------------------------|
|
||||
| ±3 | bazresi, fire, general-havades-individuals, havades, life-time-individual (5) |
|
||||
| ±1 | customer1, moavenin, Refah (3) |
|
||||
| ±2 | website (1) |
|
||||
| ±0 | lifetime-compensation (1) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Method
|
||||
|
||||
- Corpus Embedding Model locked to **`text-embedding-3-large`**
|
||||
- Newest **single-strategy** Experiment per document × Candidate (Decision Board cells)
|
||||
- Retrieval `top_k = 5`
|
||||
- Stage 1: best `fixed_size` ±N vs best `semantic@Boundary`
|
||||
- Stage 2: mean composite + per-document breakdown
|
||||
- Experiments dated **9–10 August 2026**
|
||||
- `logs/neighbor_sweep.log` recorded mid-run connection errors on some units; **SQLite now has a full 10×4 `fixed_size` grid** — treat the database as source of truth
|
||||
|
||||
---
|
||||
|
||||
## 6. Excluded strategies
|
||||
|
||||
No comparable **10-doc, single-strategy, `text-embedding-3-large`** Experiments exist for the three Strategies below. They are not Decision Board Candidates (ADR-0026).
|
||||
|
||||
| Strategy | Why it is not the winner |
|
||||
|----------|--------------------------|
|
||||
| **recursive** | Not on the Decision Board grid. Informal PDF / `text-embedding-3-small` multi-strategy runs (not comparable to this close-out) were mixed; recursive **does not get Neighbor Expansion** (expansion is `fixed_size` only). |
|
||||
| **contextual_retrieval** | Extra LLM call per chunk at process time. Same informal PDF/`small` runs scored **below** recursive and `fixed_size`. |
|
||||
| **semantic_parent_child** | Boundary-detection cost plus parent fetch at query. Informal PDF/`small` composites were **much worse** (~1.8–2.8). Fail-hard if Boundary embeds mismatch (ADR-0020). |
|
||||
|
||||
One leftover five-strategy Experiment on `customer1` under large is **invalid** (four Strategies scored 0 / errors) and was ignored.
|
||||
|
||||
---
|
||||
|
||||
## 7. Binding decision
|
||||
|
||||
**Use `fixed_size` for chunking** under **`text-embedding-3-large`.**
|
||||
|
||||
**Do not use `semantic` as the default Strategy** on this evaluation universe.
|
||||
|
||||
**Follow-up:** choose Neighbor Expansion **±N** among **±1, ±2, ±3**. Stage 1 auto-pick is **±3**. **±0 is not recommended** if this family decision is to remain valid.
|
||||
|
||||
Production config change is **out of scope** for this record.
|
||||
|
||||
---
|
||||
|
||||
## 8. Evidence (stage 2 cells)
|
||||
|
||||
| Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id |
|
||||
|----------|-------------------------------|----------------------------------|
|
||||
| bazresi.docx | `6fe08750ee324130b3d46d8b3bc95280` | `6263c4f9e63347cea9eb19f4d36309c0` |
|
||||
| customer1.docx | `8c4d888d367b4c0299b2d19dd9654f80` | `0c4572e80902408796f0dd11776e8e55` |
|
||||
| fire.docx | `230a3cf57ae74b679ecde83cfeb9b6f5` | `414162efc314406db082ba8537c2423f` |
|
||||
| general-havades-individuals.doc | `7ad892cfcd6143109f77a7808c8dfbd9` | `14fa4f872373446c8b042fc8e84e6983` |
|
||||
| havades.docx | `04bd45d836a64c3da2e68bb747c0c9b8` | `232238b4d3304243b82b02326ef64617` |
|
||||
| life-time-individual.docx | `d682706f54824469b235c099bf5a3d3c` | `bc7d9f1fbe3540c097a5f8cc1a1c44bb` |
|
||||
| moavenin.docx | `63f81cf0a632418aab2f93884518ed71` | `4e01960b5db14e81975a4fbc86c4e10c` |
|
||||
| Refah.docx | `6304dba39f3f4c05935195c588a36a64` | `b5c630c45615451a93d97d80dee361df` |
|
||||
| website.docx | `fb9d47750ca74761a3c82c57aa51030b` | `53256897cfb74dd7a6508e750ff1ba63` |
|
||||
| lifetime-compensation.docx | `99736ffbc6da455b8b3ca3a0fb8e5ef7` | `e51c2b57e7364dfa97e5429f615d6e4e` |
|
||||
|
||||
HTML reports: `GET /benchmarks/{id}/report`. Dashboard: Decision Tab, Corpus = `text-embedding-3-large`.
|
||||
@@ -399,6 +399,11 @@ def get_experiment(experiment_id: str) -> dict[str, Any] | None:
|
||||
return db.get_experiment(experiment_id)
|
||||
|
||||
|
||||
def list_experiments(document_id: str | None = None) -> dict[str, Any]:
|
||||
def list_experiments(
|
||||
document_id: str | None = None,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 200,
|
||||
) -> dict[str, Any]:
|
||||
"""List experiments, optionally filtered by document."""
|
||||
return db.list_experiments(document_id=document_id)
|
||||
return db.list_experiments(document_id=document_id, offset=offset, limit=limit)
|
||||
|
||||
@@ -8,7 +8,7 @@ Endpoints:
|
||||
GET /experiments List all experiments
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter, Query
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
from src.core.exceptions import BenchmarkError, QueryError
|
||||
@@ -185,9 +185,15 @@ async def get_benchmark(experiment_id: str):
|
||||
|
||||
|
||||
@router.get("/experiments")
|
||||
async def list_experiments(document_id: str | None = None):
|
||||
async def list_experiments(
|
||||
document_id: str | None = None,
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(200, ge=1, le=500),
|
||||
):
|
||||
"""List all experiments, optionally filtered by document."""
|
||||
result = benchmark_service.list_experiments(document_id=document_id)
|
||||
result = benchmark_service.list_experiments(
|
||||
document_id=document_id, offset=offset, limit=limit
|
||||
)
|
||||
# Enrich with document filenames and best_strategy
|
||||
from src.storage import sqlite as db
|
||||
for item in result.get("items", []):
|
||||
|
||||
@@ -434,6 +434,128 @@
|
||||
.cmp-exp-2 { background: rgba(99,102,241,0.15); color: #818cf8; }
|
||||
.cmp-exp-3 { background: rgba(244,63,94,0.15); color: #fb7185; }
|
||||
.cmp-divider { height: 1px; background: var(--border); margin: 20px 0; }
|
||||
/* ── Decision Board ─────────────────────────────────────── */
|
||||
.decision-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
|
||||
@media (max-width: 960px) { .decision-grid { grid-template-columns: 1fr; } }
|
||||
.decision-cand {
|
||||
border: 1px solid var(--border); border-radius: var(--radius);
|
||||
padding: 12px 14px; margin-bottom: 8px; cursor: pointer;
|
||||
background: var(--bg-base); transition: border-color 0.15s, background 0.15s;
|
||||
}
|
||||
.decision-cand:hover { border-color: #3a3a42; }
|
||||
.decision-cand.selected {
|
||||
border-color: var(--accent); background: rgba(234,179,8,0.08);
|
||||
}
|
||||
.decision-cand .cand-title { font-weight: 600; color: var(--text-heading); margin-bottom: 6px; }
|
||||
.decision-metrics { display: flex; flex-wrap: wrap; gap: 8px 14px; font-size: 12px; color: var(--text-muted); }
|
||||
.decision-metrics strong { color: var(--text-body); font-variant-numeric: tabular-nums; }
|
||||
.decision-legend {
|
||||
display: flex; flex-wrap: wrap; gap: 10px 16px; align-items: center;
|
||||
margin-bottom: 14px; padding: 12px 14px;
|
||||
background: rgba(15, 20, 25, 0.55);
|
||||
border: 1px solid rgba(255,255,255,0.06);
|
||||
border-radius: 10px;
|
||||
font-size: 12px; color: var(--text-muted);
|
||||
}
|
||||
.decision-legend-item { display: inline-flex; align-items: center; gap: 7px; }
|
||||
.decision-legend-swatch {
|
||||
width: 22px; height: 14px; border-radius: 4px;
|
||||
border: 1px solid rgba(255,255,255,0.08);
|
||||
box-shadow: inset 3px 0 0 var(--swatch-accent, transparent);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.decision-matrix-wrap {
|
||||
overflow-x: auto;
|
||||
border: 1px solid rgba(255,255,255,0.06);
|
||||
border-radius: 12px;
|
||||
background: rgba(15, 20, 25, 0.35);
|
||||
}
|
||||
.decision-matrix {
|
||||
width: 100%; border-collapse: separate; border-spacing: 0;
|
||||
font-size: 13px; margin: 0;
|
||||
}
|
||||
.decision-matrix th,
|
||||
.decision-matrix td {
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid rgba(255,255,255,0.05);
|
||||
border-right: 1px solid rgba(255,255,255,0.04);
|
||||
text-align: center;
|
||||
vertical-align: middle;
|
||||
}
|
||||
.decision-matrix th:last-child,
|
||||
.decision-matrix td:last-child { border-right: none; }
|
||||
.decision-matrix tbody tr:last-child th,
|
||||
.decision-matrix tbody tr:last-child td { border-bottom: none; }
|
||||
.decision-matrix thead th {
|
||||
position: sticky; top: 0; z-index: 2;
|
||||
background: rgba(21, 32, 51, 0.96);
|
||||
backdrop-filter: blur(8px);
|
||||
color: #94A3B8;
|
||||
font-size: 11px; font-weight: 600;
|
||||
letter-spacing: 0.03em;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.decision-matrix .sticky-col {
|
||||
position: sticky; left: 0; z-index: 1;
|
||||
text-align: left; font-weight: 500; white-space: nowrap;
|
||||
background: #1A2332; color: var(--text-heading);
|
||||
min-width: 160px; max-width: 220px;
|
||||
}
|
||||
.decision-matrix thead .sticky-col { z-index: 3; background: rgba(21, 32, 51, 0.96); }
|
||||
.decision-matrix tr.mean-row td,
|
||||
.decision-matrix tr.mean-row .sticky-col {
|
||||
border-top: 1px solid rgba(255,255,255,0.1);
|
||||
background: rgba(15, 20, 25, 0.65);
|
||||
font-weight: 700;
|
||||
}
|
||||
.decision-matrix tr.wins-row td,
|
||||
.decision-matrix tr.wins-row .sticky-col {
|
||||
background: rgba(15, 20, 25, 0.45);
|
||||
color: var(--text-muted);
|
||||
font-weight: 600;
|
||||
}
|
||||
.decision-matrix th.col-duel {
|
||||
color: #FBBF24;
|
||||
}
|
||||
.decision-heat {
|
||||
font-variant-numeric: tabular-nums;
|
||||
font-size: 12.5px; font-weight: 600;
|
||||
min-width: 76px;
|
||||
letter-spacing: 0.01em;
|
||||
transition: filter 0.15s ease;
|
||||
}
|
||||
.decision-heat:hover { filter: brightness(1.12); }
|
||||
.decision-heat .best-mark,
|
||||
.decision-legend .best-mark,
|
||||
.best-mark {
|
||||
display: inline-block;
|
||||
font-size: 9px; font-weight: 700; line-height: 1;
|
||||
margin-right: 5px; padding: 2px 5px;
|
||||
border-radius: 999px; vertical-align: middle;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
.decision-heat .best-mark.fs,
|
||||
.decision-legend .best-mark.fs,
|
||||
.best-mark.fs {
|
||||
background: rgba(56, 189, 248, 0.15);
|
||||
color: #7DD3FC;
|
||||
border: 1px solid rgba(56, 189, 248, 0.28);
|
||||
}
|
||||
.decision-heat .best-mark.sem,
|
||||
.decision-legend .best-mark.sem,
|
||||
.best-mark.sem {
|
||||
background: rgba(167, 139, 250, 0.15);
|
||||
color: #C4B5FD;
|
||||
border: 1px solid rgba(167, 139, 250, 0.28);
|
||||
}
|
||||
.decision-duel {
|
||||
display: grid; grid-template-columns: 1fr auto 1fr; gap: 16px; align-items: stretch;
|
||||
}
|
||||
@media (max-width: 800px) { .decision-duel { grid-template-columns: 1fr; } }
|
||||
.decision-vs {
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-weight: 700; color: var(--text-muted); font-size: 18px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
@@ -554,7 +676,7 @@ function HomeTab() {
|
||||
useEffect(() => {
|
||||
Promise.all([
|
||||
api('/documents').catch(() => ({ total: 0 })),
|
||||
api('/experiments').catch(() => []),
|
||||
api('/experiments?limit=500').catch(() => []),
|
||||
]).then(([docs, exps]) => {
|
||||
setStats({ docs: docs.total || 0, experiments: (exps?.items || []).length });
|
||||
setLoading(false);
|
||||
@@ -589,6 +711,7 @@ function HomeTab() {
|
||||
React.createElement('li', null, 'Process it with chunking strategies'),
|
||||
React.createElement('li', null, 'Ask a question in ', React.createElement('b', null, 'Query'), ' (or inside PDF Workspace)'),
|
||||
React.createElement('li', null, 'Run a full benchmark in ', React.createElement('b', null, 'Benchmarks'), ' / PDF Workspace'),
|
||||
React.createElement('li', null, 'Pick a final Strategy on the ', React.createElement('b', null, 'Decision'), ' board'),
|
||||
React.createElement('li', null, 'Check system health in the ', React.createElement('b', null, 'Admin'), ' tab')
|
||||
)
|
||||
)
|
||||
@@ -1660,7 +1783,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatF
|
||||
const [inspectLoadingId, setInspectLoadingId] = useState(null);
|
||||
|
||||
const fetchExperiments = useCallback(() => {
|
||||
api('/experiments').then(d => setExperiments(d?.items || (Array.isArray(d) ? d : []))).catch(() => {});
|
||||
api('/experiments?limit=500').then(d => setExperiments(d?.items || (Array.isArray(d) ? d : []))).catch(() => {});
|
||||
}, []);
|
||||
|
||||
useEffect(() => { fetchExperiments(); }, [fetchExperiments]);
|
||||
@@ -2267,6 +2390,648 @@ function PdfWorkspaceTab({ documents, setDocuments, strategies, addToast, questi
|
||||
);
|
||||
}
|
||||
|
||||
// -- Tab: Decision Board (ADR-0026) --------------------------
|
||||
const DECISION_DOC_FILENAMES = [
|
||||
'bazresi.docx',
|
||||
'customer1.docx',
|
||||
'fire.docx',
|
||||
'general-havades-individuals.doc',
|
||||
'havades.docx',
|
||||
'life-time-individual.docx',
|
||||
'moavenin.docx',
|
||||
'Refah.docx',
|
||||
'website.docx',
|
||||
'lifetime-compensation.docx',
|
||||
];
|
||||
|
||||
function decisionComposite(m) {
|
||||
if (!m) return null;
|
||||
return ((m.avg_context_relevance || 0) * 0.3
|
||||
+ (m.avg_answer_similarity || 0) * 0.4
|
||||
+ (m.avg_faithfulness || 0) * 0.3) * (1 - (m.hallucination_rate || 0));
|
||||
}
|
||||
|
||||
function decisionFmt(v, digits) {
|
||||
if (v == null || Number.isNaN(v)) return '—';
|
||||
return Number(v).toFixed(digits == null ? 2 : digits);
|
||||
}
|
||||
|
||||
function decisionPct(v) {
|
||||
if (v == null || Number.isNaN(v)) return '—';
|
||||
return `${(Number(v) * 100).toFixed(0)}%`;
|
||||
}
|
||||
|
||||
function decisionHeatBin(score) {
|
||||
if (score == null || Number.isNaN(score)) {
|
||||
return {
|
||||
key: 'missing', label: '—',
|
||||
bg: 'transparent', fg: 'var(--text-muted)', accent: 'transparent',
|
||||
};
|
||||
}
|
||||
const s = Number(score);
|
||||
// Modern dark-dashboard scale: translucent wash + luminous text + left accent rail
|
||||
// (no solid primary blocks — avoids Windows-98 / crayon look)
|
||||
if (s < 8.0) {
|
||||
return {
|
||||
key: 'lt80', label: '<8.0',
|
||||
bg: 'rgba(244, 63, 94, 0.14)', fg: '#FB7185', accent: '#F43F5E',
|
||||
};
|
||||
}
|
||||
if (s < 8.5) {
|
||||
return {
|
||||
key: '80_85', label: '8.0–8.5',
|
||||
bg: 'rgba(251, 146, 60, 0.13)', fg: '#FB923C', accent: '#F97316',
|
||||
};
|
||||
}
|
||||
if (s < 9.0) {
|
||||
return {
|
||||
key: '85_90', label: '8.5–9.0',
|
||||
bg: 'rgba(250, 204, 21, 0.14)', fg: '#FDE047', accent: '#EAB308',
|
||||
};
|
||||
}
|
||||
if (s < 9.5) {
|
||||
return {
|
||||
key: '90_95', label: '9.0–9.5',
|
||||
bg: 'rgba(45, 212, 191, 0.13)', fg: '#2DD4BF', accent: '#14B8A6',
|
||||
};
|
||||
}
|
||||
return {
|
||||
key: 'ge95', label: '≥9.5',
|
||||
bg: 'rgba(52, 211, 153, 0.16)', fg: '#34D399', accent: '#10B981',
|
||||
};
|
||||
}
|
||||
|
||||
function decisionHeatStyle(score) {
|
||||
const bin = decisionHeatBin(score);
|
||||
const style = { background: bin.bg, color: bin.fg };
|
||||
if (bin.accent && bin.accent !== 'transparent') {
|
||||
style.boxShadow = `inset 3px 0 0 ${bin.accent}`;
|
||||
}
|
||||
return style;
|
||||
}
|
||||
|
||||
/** Per-row (or mean-row) ids of best fixed_size and best semantic Candidate. */
|
||||
function decisionFamilyBestIds(candidates, scoreOf) {
|
||||
let bestFs = null, bestFsScore = -Infinity;
|
||||
let bestSem = null, bestSemScore = -Infinity;
|
||||
(candidates || []).forEach(c => {
|
||||
const s = scoreOf(c);
|
||||
if (s == null || Number.isNaN(s)) return;
|
||||
if (c.family === 'fixed_size' && s > bestFsScore) {
|
||||
bestFsScore = s;
|
||||
bestFs = c.id;
|
||||
}
|
||||
if (c.family === 'semantic' && s > bestSemScore) {
|
||||
bestSemScore = s;
|
||||
bestSem = c.id;
|
||||
}
|
||||
});
|
||||
return { fixed: bestFs, semantic: bestSem };
|
||||
}
|
||||
|
||||
const DECISION_HEAT_LEGEND = [
|
||||
decisionHeatBin(null),
|
||||
decisionHeatBin(7.9),
|
||||
decisionHeatBin(8.2),
|
||||
decisionHeatBin(8.7),
|
||||
decisionHeatBin(9.2),
|
||||
decisionHeatBin(9.6),
|
||||
];
|
||||
|
||||
function DecisionHeatLegend() {
|
||||
return React.createElement('div', { className: 'decision-legend', role: 'list', 'aria-label': 'Composite score color guide' },
|
||||
React.createElement('span', { style: { fontWeight: 600, color: 'var(--text-body)' } }, 'Score guide'),
|
||||
DECISION_HEAT_LEGEND.map(bin =>
|
||||
React.createElement('span', { key: bin.key, className: 'decision-legend-item', role: 'listitem' },
|
||||
React.createElement('span', {
|
||||
className: 'decision-legend-swatch',
|
||||
style: {
|
||||
background: bin.bg === 'transparent' ? 'rgba(255,255,255,0.04)' : bin.bg,
|
||||
boxShadow: bin.accent && bin.accent !== 'transparent' ? `inset 3px 0 0 ${bin.accent}` : undefined,
|
||||
},
|
||||
'aria-hidden': true,
|
||||
}),
|
||||
bin.label
|
||||
)
|
||||
),
|
||||
React.createElement('span', {
|
||||
style: { width: 1, height: 14, background: 'rgba(255,255,255,0.1)', margin: '0 4px' },
|
||||
'aria-hidden': true,
|
||||
}),
|
||||
React.createElement('span', { className: 'decision-legend-item' },
|
||||
React.createElement('span', { className: 'best-mark fs' }, 'F'),
|
||||
'best fixed_size'
|
||||
),
|
||||
React.createElement('span', { className: 'decision-legend-item' },
|
||||
React.createElement('span', { className: 'best-mark sem' }, 'S'),
|
||||
'best semantic'
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function meanOf(nums) {
|
||||
const vals = nums.filter(v => v != null && !Number.isNaN(v));
|
||||
if (!vals.length) return null;
|
||||
return vals.reduce((a, b) => a + b, 0) / vals.length;
|
||||
}
|
||||
|
||||
function buildDecisionBoard(experiments, corpusId, excludedIds) {
|
||||
const excluded = new Set(excludedIds || []);
|
||||
const pool = (experiments || []).filter(e => {
|
||||
if (excluded.has(e.id)) return false;
|
||||
const fn = e.document_filename || '';
|
||||
if (!DECISION_DOC_FILENAMES.includes(fn)) return false;
|
||||
if (corpusIdOf(e) !== corpusId) return false;
|
||||
const strats = e.strategies_used || [];
|
||||
return strats.length === 1;
|
||||
});
|
||||
|
||||
// Newest first assumed from API; keep first hit per cell
|
||||
const cellMap = {}; // key: `${fn}||${candId}` -> exp
|
||||
const fsLevels = new Set([0, 1, 2, 3]);
|
||||
const semBounds = new Set();
|
||||
|
||||
pool.forEach(e => {
|
||||
const strat = (e.strategies_used || [])[0];
|
||||
const fn = e.document_filename;
|
||||
if (strat === 'fixed_size') {
|
||||
const { prev, next } = neighborCounts(e);
|
||||
if (prev !== next || !fsLevels.has(prev)) return;
|
||||
const candId = `fixed_size:±${prev}`;
|
||||
const key = `${fn}||${candId}`;
|
||||
if (!cellMap[key]) cellMap[key] = e;
|
||||
} else if (strat === 'semantic') {
|
||||
const b = boundaryIdOf(e);
|
||||
if (!b) return;
|
||||
semBounds.add(b);
|
||||
const candId = `semantic:${b}`;
|
||||
const key = `${fn}||${candId}`;
|
||||
if (!cellMap[key]) cellMap[key] = e;
|
||||
}
|
||||
});
|
||||
|
||||
const fixedCands = [0, 1, 2, 3].map(n => ({
|
||||
id: `fixed_size:±${n}`,
|
||||
family: 'fixed_size',
|
||||
label: `fixed_size ±${n}/${n}`,
|
||||
short: `±${n}`,
|
||||
level: n,
|
||||
}));
|
||||
const semanticCands = [...semBounds].sort().map(b => ({
|
||||
id: `semantic:${b}`,
|
||||
family: 'semantic',
|
||||
label: `semantic @ ${b}`,
|
||||
short: b,
|
||||
boundary: b,
|
||||
}));
|
||||
|
||||
function metricsFor(exp, family) {
|
||||
if (!exp) return null;
|
||||
return (exp.aggregate_metrics || {})[family] || null;
|
||||
}
|
||||
|
||||
function summarize(cands) {
|
||||
return cands.map(c => {
|
||||
const perDoc = {};
|
||||
const composites = [];
|
||||
const metricBags = {
|
||||
avg_context_relevance: [],
|
||||
avg_answer_similarity: [],
|
||||
avg_faithfulness: [],
|
||||
hallucination_rate: [],
|
||||
};
|
||||
let filled = 0;
|
||||
DECISION_DOC_FILENAMES.forEach(fn => {
|
||||
const exp = cellMap[`${fn}||${c.id}`];
|
||||
const m = metricsFor(exp, c.family);
|
||||
const score = decisionComposite(m);
|
||||
perDoc[fn] = { exp, metrics: m, composite: score };
|
||||
if (score != null) {
|
||||
filled += 1;
|
||||
composites.push(score);
|
||||
Object.keys(metricBags).forEach(k => {
|
||||
if (m && m[k] != null) metricBags[k].push(m[k]);
|
||||
});
|
||||
}
|
||||
});
|
||||
return {
|
||||
...c,
|
||||
perDoc,
|
||||
filled,
|
||||
totalDocs: DECISION_DOC_FILENAMES.length,
|
||||
meanComposite: meanOf(composites),
|
||||
meanMetrics: {
|
||||
avg_context_relevance: meanOf(metricBags.avg_context_relevance),
|
||||
avg_answer_similarity: meanOf(metricBags.avg_answer_similarity),
|
||||
avg_faithfulness: meanOf(metricBags.avg_faithfulness),
|
||||
hallucination_rate: meanOf(metricBags.hallucination_rate),
|
||||
},
|
||||
wins: 0,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
const fixedSummaries = summarize(fixedCands);
|
||||
const semanticSummaries = summarize(semanticCands);
|
||||
|
||||
function assignWins(summaries) {
|
||||
DECISION_DOC_FILENAMES.forEach(fn => {
|
||||
let best = null;
|
||||
let bestScore = -Infinity;
|
||||
summaries.forEach(s => {
|
||||
const sc = s.perDoc[fn]?.composite;
|
||||
if (sc == null) return;
|
||||
if (sc > bestScore) {
|
||||
bestScore = sc;
|
||||
best = s;
|
||||
}
|
||||
});
|
||||
if (best) best.wins += 1;
|
||||
});
|
||||
}
|
||||
assignWins(fixedSummaries);
|
||||
assignWins(semanticSummaries);
|
||||
|
||||
function autoPick(summaries) {
|
||||
if (!summaries.length) return null;
|
||||
return [...summaries].sort((a, b) => {
|
||||
const ma = a.meanComposite == null ? -1 : a.meanComposite;
|
||||
const mb = b.meanComposite == null ? -1 : b.meanComposite;
|
||||
if (mb !== ma) return mb - ma;
|
||||
if (b.wins !== a.wins) return b.wins - a.wins;
|
||||
return a.id.localeCompare(b.id);
|
||||
})[0];
|
||||
}
|
||||
|
||||
return {
|
||||
cellMap,
|
||||
fixedSummaries,
|
||||
semanticSummaries,
|
||||
autoFixed: autoPick(fixedSummaries),
|
||||
autoSemantic: autoPick(semanticSummaries),
|
||||
allCandidates: [...fixedSummaries, ...semanticSummaries],
|
||||
};
|
||||
}
|
||||
|
||||
function DecisionTab({ addToast }) {
|
||||
const [experiments, setExperiments] = useState([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [corpusId, setCorpusId] = useState('text-embedding-3-large');
|
||||
const [embModels, setEmbModels] = useState([]);
|
||||
const [excludedIds, setExcludedIds] = useState([]);
|
||||
const [overrideFixed, setOverrideFixed] = useState(null);
|
||||
const [overrideSemantic, setOverrideSemantic] = useState(null);
|
||||
|
||||
const refresh = useCallback(() => {
|
||||
setLoading(true);
|
||||
Promise.all([
|
||||
api('/experiments?limit=500'),
|
||||
api('/admin/embedding-models'),
|
||||
]).then(([ex, emb]) => {
|
||||
setExperiments(ex?.items || []);
|
||||
setEmbModels(emb?.models || []);
|
||||
if (emb?.corpus_id) {
|
||||
setCorpusId(prev => prev || emb.corpus_id);
|
||||
}
|
||||
}).catch(() => addToast('Failed to load Decision Board data', 'error'))
|
||||
.finally(() => setLoading(false));
|
||||
}, [addToast]);
|
||||
|
||||
useEffect(() => { refresh(); }, [refresh]);
|
||||
|
||||
const board = React.useMemo(
|
||||
() => buildDecisionBoard(experiments, corpusId, excludedIds),
|
||||
[experiments, corpusId, excludedIds]
|
||||
);
|
||||
|
||||
const winnerFixed = (overrideFixed && board.fixedSummaries.find(c => c.id === overrideFixed))
|
||||
|| board.autoFixed;
|
||||
const winnerSemantic = (overrideSemantic && board.semanticSummaries.find(c => c.id === overrideSemantic))
|
||||
|| board.autoSemantic;
|
||||
|
||||
const excludeId = (id) => {
|
||||
if (!id) return;
|
||||
setExcludedIds(prev => prev.includes(id) ? prev : [...prev, id]);
|
||||
};
|
||||
const unexcludeId = (id) => setExcludedIds(prev => prev.filter(x => x !== id));
|
||||
|
||||
const fsFilled = board.fixedSummaries.reduce((a, c) => a + c.filled, 0);
|
||||
const fsTotal = board.fixedSummaries.length * DECISION_DOC_FILENAMES.length;
|
||||
const semFilled = board.semanticSummaries.reduce((a, c) => a + c.filled, 0);
|
||||
const semTotal = board.semanticSummaries.length * DECISION_DOC_FILENAMES.length;
|
||||
|
||||
// Stage-2 head-to-head
|
||||
let duel = null;
|
||||
if (winnerFixed && winnerSemantic) {
|
||||
let fsWins = 0, semWins = 0, ties = 0;
|
||||
const perDoc = DECISION_DOC_FILENAMES.map(fn => {
|
||||
const a = winnerFixed.perDoc[fn]?.composite;
|
||||
const b = winnerSemantic.perDoc[fn]?.composite;
|
||||
let winner = '—';
|
||||
if (a != null && b != null) {
|
||||
if (a > b) { fsWins += 1; winner = 'fixed'; }
|
||||
else if (b > a) { semWins += 1; winner = 'semantic'; }
|
||||
else { ties += 1; winner = 'tie'; }
|
||||
}
|
||||
return { fn, a, b, winner };
|
||||
});
|
||||
const recommend = (winnerFixed.meanComposite || 0) >= (winnerSemantic.meanComposite || 0)
|
||||
? winnerFixed : winnerSemantic;
|
||||
// Prefer win-count if mean close? stick to mean primary with wins as display
|
||||
duel = { fsWins, semWins, ties, perDoc, recommend };
|
||||
}
|
||||
|
||||
const renderCandCard = (c, selected, onSelect, familyAutoId) =>
|
||||
React.createElement('div', {
|
||||
key: c.id,
|
||||
className: `decision-cand${selected ? ' selected' : ''}`,
|
||||
onClick: () => onSelect(selected ? null : (c.id === familyAutoId ? null : c.id)),
|
||||
title: 'Click to override family winner; click selected again to return to auto',
|
||||
},
|
||||
React.createElement('div', { className: 'cand-title', style: { display: 'flex', justifyContent: 'space-between', gap: 8 } },
|
||||
React.createElement('span', null,
|
||||
React.createElement('input', {
|
||||
type: 'radio',
|
||||
checked: !!selected,
|
||||
readOnly: true,
|
||||
style: { marginRight: 8 },
|
||||
}),
|
||||
c.label,
|
||||
c.id === familyAutoId
|
||||
? React.createElement('span', { className: 'badge badge-accent', style: { marginLeft: 8 } }, 'auto')
|
||||
: null
|
||||
),
|
||||
React.createElement('span', { style: { color: 'var(--accent)', fontVariantNumeric: 'tabular-nums' } },
|
||||
decisionFmt(c.meanComposite))
|
||||
),
|
||||
React.createElement('div', { className: 'decision-metrics' },
|
||||
React.createElement('span', null, 'Wins ', React.createElement('strong', null, `${c.wins}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Coverage ', React.createElement('strong', null, `${c.filled}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Context ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_context_relevance))),
|
||||
React.createElement('span', null, 'Similarity ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_answer_similarity))),
|
||||
React.createElement('span', null, 'Faithfulness ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_faithfulness))),
|
||||
React.createElement('span', null, 'Halluc. ', React.createElement('strong', null, decisionPct(c.meanMetrics.hallucination_rate)))
|
||||
)
|
||||
);
|
||||
|
||||
const duelPanel = (c, side) => {
|
||||
if (!c) {
|
||||
return React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'text-muted' }, `No ${side} winner yet`));
|
||||
}
|
||||
const isRec = duel && duel.recommend && duel.recommend.id === c.id;
|
||||
return React.createElement('div', {
|
||||
className: 'card',
|
||||
style: {
|
||||
margin: 0,
|
||||
borderColor: isRec ? 'var(--accent)' : undefined,
|
||||
boxShadow: isRec ? '0 0 0 1px rgba(234,179,8,0.35)' : undefined,
|
||||
},
|
||||
},
|
||||
React.createElement('div', { className: 'flex-between mb-2' },
|
||||
React.createElement('div', { className: 'card-title mb-0' }, c.label),
|
||||
isRec ? React.createElement('span', { className: 'badge badge-success' }, 'Recommended') : null
|
||||
),
|
||||
React.createElement('div', { style: { fontSize: 28, fontWeight: 700, color: 'var(--accent)', marginBottom: 8 } },
|
||||
decisionFmt(c.meanComposite)),
|
||||
React.createElement('div', { className: 'decision-metrics', style: { marginBottom: 8 } },
|
||||
React.createElement('span', null, 'Doc wins vs other ', React.createElement('strong', null,
|
||||
side === 'fixed' ? (duel ? duel.fsWins : '—') : (duel ? duel.semWins : '—'))),
|
||||
React.createElement('span', null, 'Family wins ', React.createElement('strong', null, `${c.wins}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Coverage ', React.createElement('strong', null, `${c.filled}/${c.totalDocs}`))
|
||||
),
|
||||
React.createElement('table', { className: 'cmp-table' },
|
||||
React.createElement('tbody', null,
|
||||
[['Context Relevance', c.meanMetrics.avg_context_relevance, false],
|
||||
['Answer Similarity', c.meanMetrics.avg_answer_similarity, false],
|
||||
['Faithfulness', c.meanMetrics.avg_faithfulness, false],
|
||||
['Hallucination Rate', c.meanMetrics.hallucination_rate, true]].map(([label, val, isPct]) =>
|
||||
React.createElement('tr', { key: label },
|
||||
React.createElement('td', null, label),
|
||||
React.createElement('td', { style: { fontWeight: 600 } }, isPct ? decisionPct(val) : decisionFmt(val))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
};
|
||||
|
||||
return React.createElement('div', null,
|
||||
React.createElement('div', { className: 'flex-between mb-2' },
|
||||
React.createElement('h1', { style: { color: 'var(--text-heading)', margin: 0, fontSize: '22px' } }, 'Decision Board'),
|
||||
React.createElement('button', { className: 'btn btn-secondary btn-sm', onClick: refresh, disabled: loading },
|
||||
loading ? React.createElement('span', { className: 'spinner' }) : '↻ Refresh')
|
||||
),
|
||||
React.createElement('p', { className: 'text-sm text-muted', style: { marginBottom: 16, maxWidth: 720 } },
|
||||
'Two-stage final selection: pick the best fixed_size Neighbor level and best semantic Boundary, then compare those winners across the 10-doc evaluation set. Mean composite ranks stage 1; win-counts are shown; click a Candidate to override.'
|
||||
),
|
||||
|
||||
// Header controls
|
||||
React.createElement('div', { className: 'card mb-4' },
|
||||
React.createElement('div', { className: 'row mb-0', style: { alignItems: 'flex-end' } },
|
||||
React.createElement(EmbeddingModelSelect, {
|
||||
label: 'Corpus filter',
|
||||
value: corpusId,
|
||||
onChange: (v) => { setCorpusId(v); setOverrideFixed(null); setOverrideSemantic(null); },
|
||||
models: embModels,
|
||||
role: 'corpus',
|
||||
title: 'Only single-strategy Experiments under this Corpus Embedding Model',
|
||||
style: { maxWidth: 360 },
|
||||
}),
|
||||
React.createElement('div', { className: 'col' },
|
||||
React.createElement('div', { className: 'text-sm text-muted' }, 'Coverage'),
|
||||
React.createElement('div', { style: { fontWeight: 600, color: 'var(--text-heading)', marginTop: 6 } },
|
||||
`fixed_size ${fsFilled}/${fsTotal || 40} · semantic ${semFilled}/${semTotal || '—'}`)
|
||||
)
|
||||
),
|
||||
excludedIds.length > 0 && React.createElement('div', { style: { marginTop: 12 } },
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-1' }, 'Excluded Experiments (next-newest fills the cell)'),
|
||||
React.createElement('div', { style: { display: 'flex', flexWrap: 'wrap', gap: 6 } },
|
||||
excludedIds.map(id =>
|
||||
React.createElement('button', {
|
||||
key: id,
|
||||
className: 'btn btn-secondary btn-sm',
|
||||
onClick: () => unexcludeId(id),
|
||||
title: 'Click to restore',
|
||||
}, `✕ ${id.substring(0, 8)}…`)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
|
||||
loading && React.createElement('div', { style: { textAlign: 'center', padding: 24 } },
|
||||
React.createElement('span', { className: 'spinner' })),
|
||||
|
||||
!loading && React.createElement(React.Fragment, null,
|
||||
// Stage 1
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Stage 1 — Best Candidate per family'),
|
||||
React.createElement('div', { className: 'decision-grid mb-4' },
|
||||
React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'card-title' }, 'fixed_size (Neighbor Expansion)'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
'Auto winner: ', board.autoFixed ? board.autoFixed.label : '—',
|
||||
overrideFixed ? ' · override active' : ''),
|
||||
board.fixedSummaries.map(c => renderCandCard(
|
||||
c,
|
||||
winnerFixed && winnerFixed.id === c.id,
|
||||
setOverrideFixed,
|
||||
board.autoFixed?.id
|
||||
))
|
||||
),
|
||||
React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'card-title' }, 'semantic (Boundary Embedding Model)'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
'Auto winner: ', board.autoSemantic ? board.autoSemantic.label : '—',
|
||||
overrideSemantic ? ' · override active' : ''),
|
||||
board.semanticSummaries.length === 0
|
||||
? React.createElement('div', { className: 'empty-state' }, 'No semantic single-strategy Experiments for this Corpus')
|
||||
: board.semanticSummaries.map(c => renderCandCard(
|
||||
c,
|
||||
winnerSemantic && winnerSemantic.id === c.id,
|
||||
setOverrideSemantic,
|
||||
board.autoSemantic?.id
|
||||
))
|
||||
)
|
||||
),
|
||||
|
||||
// Stage 2
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Stage 2 — Family showdown'),
|
||||
React.createElement('div', { className: 'decision-duel mb-4' },
|
||||
duelPanel(winnerFixed, 'fixed'),
|
||||
React.createElement('div', { className: 'decision-vs' }, 'vs'),
|
||||
duelPanel(winnerSemantic, 'semantic')
|
||||
),
|
||||
duel && React.createElement('div', { className: 'card mb-4' },
|
||||
React.createElement('div', { className: 'card-title' }, 'Head-to-head by document'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
`fixed_size wins ${duel.fsWins} · semantic wins ${duel.semWins} · ties ${duel.ties}`),
|
||||
React.createElement('table', { className: 'cmp-table' },
|
||||
React.createElement('thead', null,
|
||||
React.createElement('tr', null,
|
||||
React.createElement('th', null, 'Document'),
|
||||
React.createElement('th', null, winnerFixed?.short || 'fixed'),
|
||||
React.createElement('th', null, winnerSemantic?.short || 'semantic'),
|
||||
React.createElement('th', null, 'Winner')
|
||||
)
|
||||
),
|
||||
React.createElement('tbody', null,
|
||||
duel.perDoc.map(row =>
|
||||
React.createElement('tr', { key: row.fn },
|
||||
React.createElement('td', null, row.fn),
|
||||
React.createElement('td', { style: { textAlign: 'center', fontVariantNumeric: 'tabular-nums' } }, decisionFmt(row.a)),
|
||||
React.createElement('td', { style: { textAlign: 'center', fontVariantNumeric: 'tabular-nums' } }, decisionFmt(row.b)),
|
||||
React.createElement('td', null,
|
||||
row.winner === 'fixed' ? React.createElement('span', { className: 'badge badge-accent' }, 'fixed_size')
|
||||
: row.winner === 'semantic' ? React.createElement('span', { className: 'badge badge-success' }, 'semantic')
|
||||
: row.winner === 'tie' ? React.createElement('span', { className: 'badge' }, 'tie')
|
||||
: React.createElement('span', { className: 'text-muted' }, '—')
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
|
||||
// Matrix
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Per-document matrix'),
|
||||
React.createElement('div', { className: 'card', style: { paddingBottom: 14 } },
|
||||
React.createElement(DecisionHeatLegend),
|
||||
React.createElement('div', { className: 'decision-matrix-wrap' },
|
||||
React.createElement('table', { className: 'decision-matrix' },
|
||||
React.createElement('thead', null,
|
||||
React.createElement('tr', null,
|
||||
React.createElement('th', { className: 'sticky-col' }, 'Document'),
|
||||
board.allCandidates.map(c =>
|
||||
React.createElement('th', {
|
||||
key: c.id,
|
||||
className: (winnerFixed && c.id === winnerFixed.id) || (winnerSemantic && c.id === winnerSemantic.id)
|
||||
? 'col-duel' : undefined,
|
||||
title: c.label,
|
||||
}, c.family === 'fixed_size' ? c.short : (c.short || '').substring(0, 14))
|
||||
)
|
||||
)
|
||||
),
|
||||
React.createElement('tbody', null,
|
||||
DECISION_DOC_FILENAMES.map(fn => {
|
||||
const rowBest = decisionFamilyBestIds(board.allCandidates, c => c.perDoc[fn]?.composite);
|
||||
return React.createElement('tr', { key: fn },
|
||||
React.createElement('td', { className: 'sticky-col' }, fn),
|
||||
board.allCandidates.map(c => {
|
||||
const cell = c.perDoc[fn];
|
||||
const score = cell?.composite;
|
||||
const exp = cell?.exp;
|
||||
const mark = c.id === rowBest.fixed ? 'fs' : (c.id === rowBest.semantic ? 'sem' : null);
|
||||
return React.createElement('td', {
|
||||
key: c.id,
|
||||
className: 'decision-heat',
|
||||
style: { ...decisionHeatStyle(score), cursor: exp ? 'pointer' : 'default' },
|
||||
title: exp
|
||||
? `${c.label} · ${fn}\ncomposite=${decisionFmt(score)}${mark ? `\nbest ${mark === 'fs' ? 'fixed_size' : 'semantic'} in row` : ''}\nid=${exp.id}\nClick: report · Shift+click: exclude`
|
||||
: `${c.label} · ${fn}: missing`,
|
||||
onClick: (e) => {
|
||||
if (!exp) return;
|
||||
if (e.shiftKey) {
|
||||
excludeId(exp.id);
|
||||
addToast(`Excluded ${exp.id.substring(0, 8)}…`, 'success');
|
||||
return;
|
||||
}
|
||||
window.open(`/benchmarks/${exp.id}/report`, '_blank');
|
||||
},
|
||||
},
|
||||
mark
|
||||
? React.createElement('span', {
|
||||
className: `best-mark ${mark}`,
|
||||
'aria-label': mark === 'fs' ? 'Best fixed_size in row' : 'Best semantic in row',
|
||||
}, mark === 'fs' ? 'F' : 'S')
|
||||
: null,
|
||||
decisionFmt(score)
|
||||
);
|
||||
})
|
||||
);
|
||||
}),
|
||||
(() => {
|
||||
const meanBest = decisionFamilyBestIds(board.allCandidates, c => c.meanComposite);
|
||||
return React.createElement('tr', { key: '_mean', className: 'mean-row' },
|
||||
React.createElement('td', { className: 'sticky-col' }, 'Mean'),
|
||||
board.allCandidates.map(c => {
|
||||
const mark = c.id === meanBest.fixed ? 'fs' : (c.id === meanBest.semantic ? 'sem' : null);
|
||||
return React.createElement('td', {
|
||||
key: c.id,
|
||||
className: 'decision-heat',
|
||||
style: decisionHeatStyle(c.meanComposite),
|
||||
},
|
||||
mark
|
||||
? React.createElement('span', {
|
||||
className: `best-mark ${mark}`,
|
||||
'aria-label': mark === 'fs' ? 'Best fixed_size mean' : 'Best semantic mean',
|
||||
}, mark === 'fs' ? 'F' : 'S')
|
||||
: null,
|
||||
decisionFmt(c.meanComposite)
|
||||
);
|
||||
})
|
||||
);
|
||||
})(),
|
||||
React.createElement('tr', { key: '_wins', className: 'wins-row' },
|
||||
React.createElement('td', { className: 'sticky-col' }, 'Wins'),
|
||||
board.allCandidates.map(c =>
|
||||
React.createElement('td', { key: c.id }, `${c.wins}/${c.totalDocs}`)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
React.createElement('div', { className: 'text-sm text-muted', style: { marginTop: 12 } },
|
||||
'Click a cell to open its Experiment report. Shift+click to exclude. ',
|
||||
React.createElement('span', { className: 'best-mark fs' }, 'F'),
|
||||
' / ',
|
||||
React.createElement('span', { className: 'best-mark sem' }, 'S'),
|
||||
' mark the best score in each family per row.'
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// -- Tab: Admin ---------------------------------------------
|
||||
function AdminTab({ addToast, documents, strategies, setActiveTab, setQuestionsFile }) {
|
||||
// Collapse state for each section
|
||||
@@ -2804,6 +3569,7 @@ function App() {
|
||||
{ id: 'pdf', label: 'PDF' },
|
||||
{ id: 'query', label: 'Query' },
|
||||
{ id: 'benchmarks', label: 'Benchmarks' },
|
||||
{ id: 'decision', label: 'Decision' },
|
||||
{ id: 'admin', label: 'Admin' },
|
||||
];
|
||||
|
||||
@@ -2818,6 +3584,7 @@ function App() {
|
||||
documents, strategies, addToast, formatFilter: 'word' });
|
||||
case 'benchmarks': return React.createElement(BenchmarksTab, {
|
||||
documents, strategies, addToast, questionsFile, formatFilter: 'word' });
|
||||
case 'decision': return React.createElement(DecisionTab, { addToast });
|
||||
case 'admin': return React.createElement(AdminTab, {
|
||||
addToast, documents, strategies, setActiveTab, setQuestionsFile });
|
||||
default: return null;
|
||||
|
||||
@@ -408,9 +408,11 @@ def get_experiment(experiment_id: str) -> dict[str, Any] | None:
|
||||
|
||||
|
||||
def list_experiments(
|
||||
*, document_id: str | None = None, offset: int = 0, limit: int = 50
|
||||
*, document_id: str | None = None, offset: int = 0, limit: int = 200
|
||||
) -> dict[str, Any]:
|
||||
"""List experiments, optionally filtered by document."""
|
||||
limit = max(1, min(int(limit), 500))
|
||||
offset = max(0, int(offset))
|
||||
conn = _connect()
|
||||
try:
|
||||
if document_id:
|
||||
|
||||
Reference in New Issue
Block a user