Compare commits
9 Commits
e2c7fdc059
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| e1ca9f4127 | |||
| 3f9677f6aa | |||
| 47e4846270 | |||
| dc76ca67b8 | |||
| d5ccec2f2c | |||
| 640691b8ab | |||
| 6261bc0a23 | |||
| 2ac69c9944 | |||
| 56b8d9401a |
@@ -14,9 +14,9 @@ OLLAMA_BASE_URL=http://192.168.10.10:11435
|
||||
|
||||
# Retrieval defaults
|
||||
TOP_K=5
|
||||
# Neighbor Expansion for fixed_size only (ADR-0023); 0 = off
|
||||
NEIGHBOR_PREV=0
|
||||
NEIGHBOR_NEXT=0
|
||||
# Neighbor Expansion for fixed_size only (ADR-0023); 3/3 = stabilized default after benchmark
|
||||
NEIGHBOR_PREV=3
|
||||
NEIGHBOR_NEXT=3
|
||||
|
||||
# LLM generation parameters
|
||||
TEMPERATURE=0.0
|
||||
|
||||
13
CONTEXT.md
13
CONTEXT.md
@@ -87,7 +87,7 @@ The configured catalog of Embedding Models the operator may assign to Boundary o
|
||||
_Avoid_: Model list, provider catalog, available embeddings
|
||||
|
||||
**Neighbor Expansion**:
|
||||
Query-time widening of retrieved context for the `fixed_size` Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (`chunk_index`). Neighbors are not re-ranked as independent hits. Counts (`neighbor_prev` / `neighbor_next`) are set per Query or Experiment like `top_k`; defaults are `0`/`0` (off). Missing neighbors at document edges are skipped. Knobs are ignored for non-`fixed_size` Strategies. Dashboard labels Experiments with a compact `±P/N` badge (tooltip explains prev/next; muted when `fixed_size` was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time.
|
||||
Query-time widening of retrieved context for the `fixed_size` Strategy only: for every top-k hit, also include a configurable number of previous and next chunks in document order within the same document. Neighbors are added on top of the top-k set (context may grow beyond k). The same chunk id appears at most once in the LLM context. Final LLM context is sorted by document order (`chunk_index`). Neighbors are not re-ranked as independent hits. Counts (`neighbor_prev` / `neighbor_next`) are set per Query or Experiment like `top_k`; benchmarker defaults are `3`/`3` (symmetric ±3 after Strategy finalization); `0`/`0` turns expansion off. Missing neighbors at document edges are skipped. Knobs are ignored for non-`fixed_size` Strategies. Dashboard labels Experiments with a compact `±P/N` badge (tooltip explains prev/next; muted when `fixed_size` was not in the run). Experiments list and Compare show these knobs; Compare warns when selected Experiments differ on Neighbor Expansion, Corpus Embedding Model, or Boundary Embedding Model (when recorded) so operators do not misread cross-run rankings. Distinct from how a Strategy cuts text at process time.
|
||||
_Avoid_: Context windowing, chunk padding, sliding window retrieval, adjacent chunk merge
|
||||
|
||||
**Expansion Tree**:
|
||||
@@ -98,13 +98,21 @@ _Avoid_: Retrieved chunks list, neighbor list, expansion map
|
||||
An ordered batch of Experiments that hold Strategy and Corpus Embedding Model fixed while stepping Neighbor Expansion through symmetric levels (±0, ±1, ±2, ±3) — i.e. `(neighbor_prev, neighbor_next)` = `(0,0)`, `(1,1)`, `(2,2)`, `(3,3)` — across one or more documents. Each level is its own Experiment; the Sweep is the sequence, not a single stored row.
|
||||
_Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||
|
||||
**Strategy Candidate**:
|
||||
A single comparable configuration for final Strategy selection under one Corpus Embedding Model: either `fixed_size` at one Neighbor Expansion level (±N), or `semantic` under one Boundary Embedding Model. Distinct from an Experiment (an Experiment is one doc’s run of one Candidate, or of several Strategies). Final selection compares Candidates, not raw multi-strategy Experiments. The Decision Board auto-discovers Candidates from single-strategy Experiments (newest per doc×Candidate cell), filtered by Corpus Embedding Model, with optional exclude of a bad Experiment.
|
||||
_Avoid_: Variant, configuration, setup, arm
|
||||
|
||||
**Decision Board**:
|
||||
A top-level Dashboard Tab for choosing between Strategy families after tuning: stage 1 picks the best Strategy Candidate within `fixed_size` and within `semantic`; stage 2 compares those two winners. Always accompanied by a per-document breakdown so aggregate rank cannot hide doc-level disagreement. Document universe for v1 is the fixed 10-doc evaluation set (same as the Neighbor Expansion Benchmark Sweep). Distinct from Compare (ad-hoc multi-Experiment side-by-side) and from Retrieval Inspect.
|
||||
_Avoid_: Final compare, strategy picker, results page, leaderboard, Decision mode
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
| # | Decision | Status |
|
||||
|---|----------|--------|
|
||||
ADR-0001 | Single-file React via CDN (no build step), served by FastAPI | Approved |
|
||||
ADR-0002 | New `/admin/` router for dashboard-specific backend ops (health, Qdrant CRUD, chunk preview, questions, cost) | Approved |
|
||||
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Admin) — not sidebar | Approved |
|
||||
ADR-0003 | Top-tab navigation (Home, Documents, Query, Benchmarks, Decision, Admin) — not sidebar | Approved |
|
||||
ADR-0004 | Babel-in-browser JSX: single index.html with inline <script type="text/babel">, React+ReactDOM+Babel from CDN. Zero build tooling, one file to edit. | Approved |
|
||||
ADR-0005 | Persistent tabs with useState on root App. No routing library, no state library — 5-7 shared state values passed as props. Tab components stay mounted, state survives tab switches. | Approved |
|
||||
ADR-0006 | Dashboard calls existing REST endpoints for documents/queries/benchmarks. New /admin/* router ONLY fills gaps: Qdrant CRUD, health, chunk preview, questions dataset, cost estimation. No endpoint duplication. | Approved |
|
||||
@@ -126,3 +134,4 @@ _Avoid_: Batch run, sequential benchmark, benchmark script, neighbor matrix
|
||||
ADR-0023 | Neighbor Expansion for fixed_size (+ Expansion Tree; list/Compare `±P/N` provenance + mismatch warning). See docs/adr/0023-*.md | Approved |
|
||||
ADR-0024 | Boundary vs Corpus Embedding Model roles (Admin defaults + per-op overrides; query locked to Corpus). See docs/adr/0024-*.md | Implemented |
|
||||
ADR-0025 | Retrieval Inspect: Benchmarks full-page mode (question rail + full-text Expansion Tree). See docs/adr/0025-*.md | Approved |
|
||||
ADR-0026 | Decision Board Tab: two-stage Strategy Candidate selection (fixed_size ±N vs semantic@Boundary) over the 10-doc set. See docs/adr/0026-*.md | Approved |
|
||||
|
||||
280
docs/HLD.md
Normal file
280
docs/HLD.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# High-Level Design (HLD)
|
||||
|
||||
RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
|
||||
|
||||
**Audience:** architects, tech leads, new engineers
|
||||
**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
|
||||
|
||||
Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
|
||||
|
||||
---
|
||||
|
||||
## 2. Goals and Non-Goals
|
||||
|
||||
### Goals
|
||||
|
||||
| Goal | Notes |
|
||||
|------|--------|
|
||||
| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
|
||||
| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
|
||||
| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
|
||||
| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
|
||||
| Failure isolation | One Strategy failing during process does not discard others |
|
||||
|
||||
### Non-Goals (v1)
|
||||
|
||||
| Out of scope | See |
|
||||
|--------------|-----|
|
||||
| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
|
||||
| Multi-tenant auth | Single-operator local tool |
|
||||
| Production-scale concurrency | SQLite + single process |
|
||||
| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
|
||||
|
||||
---
|
||||
|
||||
## 3. System Context
|
||||
|
||||
```
|
||||
┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐
|
||||
│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│
|
||||
│ (Browser) │ Dashboard /app │ │
|
||||
└──────────────┘ │ Documents · Chunking · Benchmarking │
|
||||
│ Admin · Static Dashboard │
|
||||
└───────────┬─────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────┼─────────────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
|
||||
│ SQLite │ │ Qdrant │ │ LLM / Embed │
|
||||
│ metadata, │ │ Model Corpus │ │ OpenAI Cloud │
|
||||
│ Experiments, │ │ vectors + │ │ + Ollama Local│
|
||||
│ app_settings │ │ chunk payload │ └────────────────┘
|
||||
└────────────────┘ └────────────────┘
|
||||
```
|
||||
|
||||
**External dependencies**
|
||||
|
||||
| System | Role |
|
||||
|--------|------|
|
||||
| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
|
||||
| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
|
||||
| Qdrant | Vector search per Strategy × Corpus Embedding Model |
|
||||
| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
|
||||
|
||||
---
|
||||
|
||||
## 4. Logical Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Presentation Layer │
|
||||
│ src/static/index.html — single-file React Dashboard (CDN, no build) │
|
||||
│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│ REST
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ API Layer (FastAPI) │
|
||||
│ /documents* /queries* /benchmarks* /experiments* /admin/* │
|
||||
│ /strategies /app (StaticFiles) │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ Domain Services │
|
||||
│ documents/service chunking/service query_service benchmark_service│
|
||||
│ evaluation admin/service report │
|
||||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||||
│ Foundation │
|
||||
│ core/config · models · exceptions · dependencies │
|
||||
│ chunking/base · embedding · embedding_models · strategies/* │
|
||||
│ documents/parser · pdf_parser │
|
||||
│ storage/sqlite · storage/qdrant │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Major Components
|
||||
|
||||
| Component | Responsibility | Key modules |
|
||||
|-----------|----------------|-------------|
|
||||
| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
|
||||
| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
|
||||
| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
|
||||
| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
|
||||
| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
|
||||
| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
|
||||
| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
|
||||
| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
|
||||
|
||||
---
|
||||
|
||||
## 6. Domain Concepts (summary)
|
||||
|
||||
Full glossary: [CONTEXT.md](../CONTEXT.md).
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
|
||||
| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
|
||||
| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
|
||||
| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
|
||||
| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
|
||||
| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
|
||||
| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
|
||||
| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
|
||||
| **Decision Board** | Two-stage selection over the fixed evaluation document set |
|
||||
|
||||
---
|
||||
|
||||
## 7. End-to-End Flows
|
||||
|
||||
### 7.1 Ingest and process
|
||||
|
||||
```
|
||||
Upload (.docx/.doc/.pdf)
|
||||
→ Text-layer Gate (PDF) / parse DOCX
|
||||
→ Heading Reconstruction (PDF) → DocumentTree + markdown
|
||||
→ SQLite documents
|
||||
|
||||
Process(strategies, boundary?, corpus?)
|
||||
→ For each Strategy (isolated failures):
|
||||
Boundary embeds (semantic*) → Strategy.chunk()
|
||||
→ Corpus embeds → Qdrant Model Corpus upsert
|
||||
→ Update chunk_counts + last_* embedding provenance on document
|
||||
```
|
||||
|
||||
### 7.2 Query
|
||||
|
||||
```
|
||||
POST /queries
|
||||
→ Resolve Corpus Embedding Model
|
||||
→ Embed question (Corpus)
|
||||
→ Qdrant top-k (document filter)
|
||||
→ Neighbor Expansion if fixed_size (±P/N)
|
||||
→ Build context (parent fetch for semantic_parent_child)
|
||||
→ LLM answer → SQLite queries (+ expansion_tree)
|
||||
```
|
||||
|
||||
### 7.3 Experiment
|
||||
|
||||
```
|
||||
POST /benchmarks
|
||||
→ Load questions JSON
|
||||
→ For each question × Strategy: Query + evaluate_single
|
||||
→ Aggregate metrics → SQLite experiments
|
||||
→ Optional HTML report (managerial / technical)
|
||||
```
|
||||
|
||||
### 7.4 Decision
|
||||
|
||||
```
|
||||
Dashboard Decision Tab
|
||||
→ Load Experiments filtered by Corpus Embedding Model
|
||||
→ Discover Strategy Candidates (newest per doc × Candidate)
|
||||
→ Stage 1: best fixed_size ±N vs best semantic@Boundary
|
||||
→ Stage 2: compare winners + per-document breakdown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Data Architecture
|
||||
|
||||
### 8.1 Responsibility split
|
||||
|
||||
| Store | Owns |
|
||||
|-------|------|
|
||||
| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
|
||||
| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
|
||||
| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
|
||||
|
||||
### 8.2 Model Corpus naming
|
||||
|
||||
```
|
||||
{strategy}__{sanitized_model_id}_collection
|
||||
```
|
||||
|
||||
Example: `fixed_size__text-embedding-3-large_collection`
|
||||
|
||||
Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
|
||||
|
||||
### 8.3 Provenance
|
||||
|
||||
Experiments store:
|
||||
|
||||
- `embedding_model_id` / `embedding_provider` — Corpus
|
||||
- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
|
||||
- Neighbor Expansion levels in `benchmark_config`
|
||||
|
||||
Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
|
||||
|
||||
---
|
||||
|
||||
## 9. Deployment View
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Host process │
|
||||
│ uvicorn src.main:app │
|
||||
│ └── FastAPI + Dashboard static │
|
||||
│ │
|
||||
│ data/chunking_benchmark.db │
|
||||
│ .env (secrets + knobs) │
|
||||
└───────────────┬─────────────────────────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
▼ ▼
|
||||
Qdrant (:6333) OpenAI / Ollama
|
||||
```
|
||||
|
||||
Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
|
||||
|
||||
---
|
||||
|
||||
## 10. Cross-Cutting Concerns
|
||||
|
||||
| Concern | Approach |
|
||||
|---------|----------|
|
||||
| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
|
||||
| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
|
||||
| Logging | Request middleware + structured stage logs in query/benchmark |
|
||||
| CORS | Permissive (`*`) for local Dashboard development |
|
||||
| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
|
||||
| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
|
||||
|
||||
---
|
||||
|
||||
## 11. Key Architectural Decisions
|
||||
|
||||
| ADR | Decision |
|
||||
|-----|----------|
|
||||
| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
|
||||
| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
|
||||
| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
|
||||
| 0021 | Always scope Qdrant collections by Embedding Model id |
|
||||
| 0022 | Per-model `semantic_threshold` |
|
||||
| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
|
||||
| 0024 | Boundary vs Corpus Embedding Model roles |
|
||||
| 0025 | Retrieval Inspect in Benchmarks Tab |
|
||||
| 0026 | Decision Board two-stage Candidate selection |
|
||||
|
||||
---
|
||||
|
||||
## 12. Related Documents
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
|
||||
| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
|
||||
| [data-flow.md](data-flow.md) | Pipeline detail |
|
||||
| [api-reference.md](api-reference.md) | HTTP contracts |
|
||||
| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
|
||||
| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
|
||||
| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |
|
||||
571
docs/LLD.md
Normal file
571
docs/LLD.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# Low-Level Design (LLD)
|
||||
|
||||
RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.
|
||||
|
||||
**Audience:** implementers
|
||||
**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Package Map
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.py # App factory, middleware, router mount
|
||||
├── static/index.html # Dashboard (React + Babel CDN)
|
||||
├── core/
|
||||
│ ├── config.py # Settings (.env)
|
||||
│ ├── models.py # Chunk, DocumentTree, StrategyName
|
||||
│ ├── exceptions.py # Domain errors + handlers
|
||||
│ └── dependencies.py # OpenAI / Ollama / Qdrant singletons
|
||||
├── storage/
|
||||
│ ├── sqlite.py # documents, queries, experiments, app_settings
|
||||
│ └── qdrant.py # Model Corpus CRUD + search
|
||||
├── documents/
|
||||
│ ├── parser.py # DOCX / .doc (+ LibreOffice)
|
||||
│ ├── pdf_parser.py # Text PDF + Text-layer Gate
|
||||
│ ├── heading_heuristics.py # PDF Heading Reconstruction helpers
|
||||
│ ├── service.py # Upload / process / delete
|
||||
│ ├── routes.py
|
||||
│ └── models.py # ProcessRequest, DocumentResponse, …
|
||||
├── chunking/
|
||||
│ ├── base.py # ChunkingStrategy ABC, tokens, sentences
|
||||
│ ├── embedding.py # Boundary/Corpus resolve + embed_texts
|
||||
│ ├── embedding_models.py # Registry, thresholds, task prefixes
|
||||
│ ├── service.py # run_strategies orchestration
|
||||
│ └── strategies/
|
||||
│ ├── fixed_size.py
|
||||
│ ├── recursive.py
|
||||
│ ├── semantic.py
|
||||
│ ├── contextual_retrieval.py
|
||||
│ └── semantic_parent_child.py
|
||||
├── benchmarking/
|
||||
│ ├── query_service.py # Query + Neighbor Expansion
|
||||
│ ├── benchmark_service.py # Experiment runner + cost estimate
|
||||
│ ├── evaluation.py # LLM-as-Judge
|
||||
│ ├── report.py # HTML reports
|
||||
│ ├── routes.py
|
||||
│ └── models.py
|
||||
└── admin/
|
||||
├── routes.py # /admin/*
|
||||
└── service.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Application Bootstrap
|
||||
|
||||
**Entry:** `create_app()` in `src/main.py`
|
||||
|
||||
1. `init_db()` — create/migrate SQLite schema
|
||||
2. Register CORS + `RequestLoggingMiddleware`
|
||||
3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`
|
||||
4. Routers: documents, benchmarking, admin
|
||||
5. Mount `StaticFiles` at `/app` → `src/static/`
|
||||
|
||||
```python
|
||||
# Singletons (lru_cache)
|
||||
get_openai_client() # Cloud LLM + cloud embeds
|
||||
get_ollama_client() # OpenAI-compatible client → Ollama base URL
|
||||
get_qdrant_client()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
`Settings` (`src/core/config.py`) — selected fields:
|
||||
|
||||
| Field | Default | Use |
|
||||
|-------|---------|-----|
|
||||
| `openai_api_key` | required | Cloud Provider |
|
||||
| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
|
||||
| `llm_model` | `gpt-4o-mini` | Answer + judge |
|
||||
| `ollama_base_url` | host URL | Local embeds |
|
||||
| `qdrant_url` | `http://localhost:6333` | Vectors |
|
||||
| `top_k` | `5` | Retrieval |
|
||||
| `neighbor_prev` / `neighbor_next` | `3` / `3` | Expansion defaults (fixed_size ±3) |
|
||||
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
|
||||
| `semantic_threshold` | `0.3` | Fallback if model default missing |
|
||||
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
|
||||
| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
|
||||
| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Domain Models
|
||||
|
||||
### 4.1 StrategyName
|
||||
|
||||
```python
|
||||
class StrategyName(str, Enum):
|
||||
RECURSIVE = "recursive"
|
||||
FIXED_SIZE = "fixed_size"
|
||||
SEMANTIC = "semantic"
|
||||
CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
|
||||
SEMANTIC_PARENT_CHILD = "semantic_parent_child"
|
||||
```
|
||||
|
||||
### 4.2 Chunk (unified)
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `document_name` | str | Source filename |
|
||||
| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
|
||||
| `strategy_name` | StrategyName | |
|
||||
| `chunk_index` | int | Document order (used by Neighbor Expansion) |
|
||||
| `text` | str | Stored in Qdrant; used for LLM context |
|
||||
| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
|
||||
| `parent_id` | str \| None | Parent-child Strategy |
|
||||
| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |
|
||||
|
||||
`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).
|
||||
|
||||
### 4.3 DocumentTree
|
||||
|
||||
```
|
||||
DocumentTree
|
||||
└── root: DocumentTreeNode
|
||||
node_type: document | section | article | paragraph
|
||||
text, heading, heading_level, children[]
|
||||
```
|
||||
|
||||
Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage LLD
|
||||
|
||||
### 5.1 SQLite schema
|
||||
|
||||
**documents**
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|--------|------|---------|
|
||||
| `id` | TEXT PK | UUID hex |
|
||||
| `filename` | TEXT | Original name |
|
||||
| `parsed_text` | TEXT | Markdown for chunking |
|
||||
| `document_tree` | TEXT | JSON tree |
|
||||
| `chunk_counts` | TEXT | JSON `{strategy: count}` |
|
||||
| `last_corpus_embedding_model_id` | TEXT | Provenance |
|
||||
| `last_boundary_embedding_model_id` | TEXT | Provenance |
|
||||
| `created_at` | TEXT | ISO UTC |
|
||||
|
||||
**queries**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id`, `strategy_name` | Identity |
|
||||
| `question`, `answer` | Content |
|
||||
| `retrieved_chunks` | Flat LLM/eval list (JSON) |
|
||||
| `expansion_tree` | Per-hit neighbors (JSON) |
|
||||
| `latency_breakdown`, `token_usage` | Observability JSON |
|
||||
| `created_at` | |
|
||||
|
||||
**experiments**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id` | Identity |
|
||||
| `benchmark_config` | top_k, neighbor_prev/next, etc. |
|
||||
| `questions`, `per_question` | Inputs + per-cell results |
|
||||
| `aggregate_metrics` | Per-Strategy averages |
|
||||
| `strategies_used` | JSON list |
|
||||
| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
|
||||
| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
|
||||
| `created_at` | |
|
||||
|
||||
**app_settings** — key/value for:
|
||||
|
||||
- `corpus_embedding_model_id`, `boundary_embedding_model_id`
|
||||
- Legacy `active_embedding_model_id` (migrated into both roles)
|
||||
- `semantic_threshold:{model_id}` overrides
|
||||
|
||||
Connection: WAL mode, foreign keys ON, one connection per call (no pool).
|
||||
|
||||
### 5.2 Qdrant
|
||||
|
||||
**Collection name**
|
||||
|
||||
```python
|
||||
def collection_name(strategy, model_id) -> str:
|
||||
return f"{strategy}__{sanitize(model_id)}_collection"
|
||||
# sanitize: replace : and / with -
|
||||
```
|
||||
|
||||
**Point**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
|
||||
| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
|
||||
| `payload` | ChunkMetadata fields |
|
||||
|
||||
**Search**
|
||||
|
||||
- Cosine distance
|
||||
- Optional filter: `document_name == filename`
|
||||
- Returns payload + score
|
||||
|
||||
**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.
|
||||
|
||||
**Admin:** list/create/delete collections; wipe points.
|
||||
|
||||
---
|
||||
|
||||
## 6. Embedding Subsystem
|
||||
|
||||
### 6.1 Registry (`embedding_models.py`)
|
||||
|
||||
| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
|
||||
|----|----------|-----------|----------------------------|---------------|
|
||||
| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
|
||||
| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
|
||||
| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |
|
||||
|
||||
`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.
|
||||
|
||||
### 6.2 Role resolution (`embedding.py`)
|
||||
|
||||
```
|
||||
resolve_corpus_model(optional_id) → snapshot for process/query/experiment
|
||||
resolve_boundary_model(optional_id) → snapshot for semantic cuts
|
||||
|
||||
get_corpus_embedding_model() / get_boundary_embedding_model()
|
||||
→ app_settings → registry → DEFAULT_CLOUD_MODEL_ID
|
||||
```
|
||||
|
||||
Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.
|
||||
|
||||
### 6.3 embed_texts / embed_single
|
||||
|
||||
1. `apply_task_prefixes(texts, model, purpose)` if needed
|
||||
2. Batch: OpenAI 2048 / Ollama 64
|
||||
3. Route to `get_openai_client()` or `get_ollama_client()` by Provider
|
||||
4. Raise `EmbeddingError` on failure
|
||||
|
||||
**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).
|
||||
|
||||
---
|
||||
|
||||
## 7. Documents LLD
|
||||
|
||||
### 7.1 Upload
|
||||
|
||||
```
|
||||
upload_document(filename, bytes)
|
||||
→ temp file → parse_document(path)
|
||||
→ db.save_document(parsed_text=markdown, document_tree=JSON)
|
||||
```
|
||||
|
||||
**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).
|
||||
|
||||
**PDF path (`pdf_parser.py`):**
|
||||
|
||||
1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF
|
||||
2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)
|
||||
3. Table Flattening to sequential plain text
|
||||
4. Emit markdown `#` / `##` + DocumentTree
|
||||
|
||||
### 7.2 Process
|
||||
|
||||
```
|
||||
ProcessRequest:
|
||||
strategies: list[StrategyName] # default all five
|
||||
boundary_model_id: str | None
|
||||
corpus_model_id: str | None
|
||||
|
||||
process_document → chunking.service.run_strategies(...)
|
||||
→ ProcessResponse(completed[], failed[], corpus_*, boundary_*)
|
||||
```
|
||||
|
||||
### 7.3 Delete
|
||||
|
||||
Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Chunking LLD
|
||||
|
||||
### 8.1 Interface
|
||||
|
||||
```python
|
||||
class ChunkingStrategy(ABC):
|
||||
name: StrategyName
|
||||
|
||||
@abstractmethod
|
||||
def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
|
||||
...
|
||||
```
|
||||
|
||||
Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).
|
||||
|
||||
### 8.2 Orchestrator (`run_strategies`)
|
||||
|
||||
```
|
||||
corpus = resolve_corpus_model(corpus_model_id)
|
||||
boundary = resolve_boundary_model(...) if any(semantic*) else unused
|
||||
|
||||
for strategy in strategies:
|
||||
try:
|
||||
ensure_collection(strategy, corpus.id, corpus.dimension)
|
||||
chunks = _chunk_document(...) # inject Boundary embeds if needed
|
||||
texts = enriched_content or text
|
||||
embeddings = embed_texts(texts, model=corpus, purpose="document")
|
||||
upsert_chunks(chunks, embeddings, model_id=corpus.id)
|
||||
except → failed[] (others continue)
|
||||
|
||||
update_chunk_counts + update_process_embedding_provenance
|
||||
```
|
||||
|
||||
### 8.3 Strategy algorithms (summary)
|
||||
|
||||
| Strategy | Input units | Boundary | Output |
|
||||
|----------|-------------|----------|--------|
|
||||
| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
|
||||
| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
|
||||
| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
|
||||
| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
|
||||
| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |
|
||||
|
||||
Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.
|
||||
|
||||
**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.
|
||||
|
||||
Deep dive: [strategy-technical-details.md](strategy-technical-details.md).
|
||||
|
||||
---
|
||||
|
||||
## 9. Query LLD
|
||||
|
||||
### 9.1 `run_query` pipeline
|
||||
|
||||
```
|
||||
1. Load document (filename for filter)
|
||||
2. Resolve Corpus model
|
||||
3. embed_single(question, purpose="query")
|
||||
4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
|
||||
5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
|
||||
6. _build_context (parent scroll for semantic_parent_child)
|
||||
7. _generate_answer (settings.llm_model, temp 0)
|
||||
8. db.save_query(...)
|
||||
```
|
||||
|
||||
### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)
|
||||
|
||||
**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.
|
||||
|
||||
1. Build Expansion Tree nodes for each top-k hit (score order)
|
||||
2. Collect needed `chunk_index` values: hit±prev/next (skip < 0)
|
||||
3. Fetch missing indices via Qdrant
|
||||
4. Fill `neighbors_prev` / `neighbors_next` per hit
|
||||
5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`
|
||||
6. Neighbors have `role="neighbor"`, `score=None`
|
||||
|
||||
Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).
|
||||
|
||||
### 9.3 Parent-child context
|
||||
|
||||
For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.
|
||||
|
||||
---
|
||||
|
||||
## 10. Benchmark LLD
|
||||
|
||||
### 10.1 Questions format
|
||||
|
||||
```json
|
||||
{
|
||||
"questions": [
|
||||
{
|
||||
"id": "q1",
|
||||
"question": "...",
|
||||
"expected_answer": "...",
|
||||
"category": "...",
|
||||
"difficulty": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Load from `questions_file` path or inline `questions` list.
|
||||
|
||||
### 10.2 `run_benchmark`
|
||||
|
||||
```
|
||||
embedding_model = resolve_corpus_model(corpus_model_id)
|
||||
boundary_id = doc.last_boundary_embedding_model_id # if semantic in strategies
|
||||
|
||||
for question in questions:
|
||||
for strategy in strategies:
|
||||
result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
|
||||
scores = evaluate_single(question, retrieved_chunks, expected, generated)
|
||||
append per_question row
|
||||
|
||||
aggregate_metrics[strategy] = averages + hallucination_rate
|
||||
best_strategy = argmax (e.g. answer_similarity / composite — see service)
|
||||
save experiment with provenance + benchmark_config
|
||||
```
|
||||
|
||||
### 10.3 Evaluation (`evaluate_single`)
|
||||
|
||||
LLM returns JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"context_relevance": 1-10,
|
||||
"answer_similarity": 1-10,
|
||||
"faithfulness": 1-10,
|
||||
"hallucination": true|false,
|
||||
"reasoning": "..."
|
||||
}
|
||||
```
|
||||
|
||||
`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).
|
||||
|
||||
### 10.4 Reports
|
||||
|
||||
`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).
|
||||
|
||||
### 10.5 Cost estimate
|
||||
|
||||
`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.
|
||||
|
||||
---
|
||||
|
||||
## 11. Admin LLD
|
||||
|
||||
| Endpoint | Behavior |
|
||||
|----------|----------|
|
||||
| `GET /admin/health` | App + Qdrant + SQLite ping |
|
||||
| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
|
||||
| `PUT .../corpus` · `.../boundary` | Set role defaults |
|
||||
| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
|
||||
| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
|
||||
| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
|
||||
| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
|
||||
| `POST /admin/cost-estimate` | Same heuristic as dry-run |
|
||||
|
||||
Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).
|
||||
|
||||
---
|
||||
|
||||
## 12. Dashboard LLD (behavioral)
|
||||
|
||||
| Concern | Design |
|
||||
|---------|--------|
|
||||
| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
|
||||
| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
|
||||
| Theme | Dark `#111113`… + amber accent (ADR-0008) |
|
||||
| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
|
||||
| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
|
||||
| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
|
||||
| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |
|
||||
|
||||
No client router or global store — props from root state.
|
||||
|
||||
---
|
||||
|
||||
## 13. API Surface (concise)
|
||||
|
||||
| Method | Path | Service |
|
||||
|--------|------|---------|
|
||||
| GET/POST | `/documents` | list / upload |
|
||||
| POST | `/documents/{id}/process` | Strategies |
|
||||
| DELETE | `/documents/{id}` | delete |
|
||||
| GET | `/strategies` | catalog |
|
||||
| POST | `/queries` | query |
|
||||
| GET | `/queries/{id}` | history |
|
||||
| POST | `/benchmarks` | Experiment or dry_run |
|
||||
| GET | `/benchmarks/{id}` | Experiment detail |
|
||||
| GET | `/benchmarks/{id}/report` | HTML |
|
||||
| GET | `/experiments` | list (+ filters used by Decision Board) |
|
||||
| * | `/admin/*` | ops |
|
||||
| GET | `/app/` | Dashboard |
|
||||
|
||||
Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.
|
||||
|
||||
**Request highlights**
|
||||
|
||||
```python
|
||||
# Process
|
||||
boundary_model_id: Optional[str]
|
||||
corpus_model_id: Optional[str]
|
||||
|
||||
# Query / Benchmark
|
||||
top_k: int = 5
|
||||
neighbor_prev / neighbor_next: int = 0..5
|
||||
corpus_model_id: Optional[str]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Error Model
|
||||
|
||||
| Exception | Typical cause | HTTP |
|
||||
|-----------|---------------|------|
|
||||
| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
|
||||
| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
|
||||
| `EmbeddingError` | Provider/API failure | 400 |
|
||||
| `QdrantError` | Collection/upsert/search failure | 400 |
|
||||
| `QueryError` | Missing doc, LLM answer failure | 400 |
|
||||
| `BenchmarkError` | Bad questions file, eval failure | 400 |
|
||||
|
||||
Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.
|
||||
|
||||
---
|
||||
|
||||
## 15. Sequence Diagrams
|
||||
|
||||
### Process (one Strategy)
|
||||
|
||||
```
|
||||
Client → DocumentsAPI → DocumentService → ChunkingService
|
||||
ChunkingService → resolve Boundary/Corpus
|
||||
ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
|
||||
ChunkingService → embed_texts (Corpus)
|
||||
ChunkingService → Qdrant.upsert
|
||||
ChunkingService → SQLite.update counts + provenance
|
||||
Client ← ProcessResponse
|
||||
```
|
||||
|
||||
### Query with Neighbor Expansion
|
||||
|
||||
```
|
||||
Client → QueryAPI → run_query
|
||||
→ embed_single (Corpus)
|
||||
→ Qdrant.search top-k
|
||||
→ apply_neighbor_expansion → get_chunks_by_indices
|
||||
→ _build_context → OpenAI chat
|
||||
→ SQLite.save_query
|
||||
Client ← QueryResponse (retrieved_chunks + expansion_tree)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 16. Extension Points
|
||||
|
||||
| Extension | Steps |
|
||||
|-----------|-------|
|
||||
| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
|
||||
| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
|
||||
| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
|
||||
| New Admin op | Prefer `/admin` only when domain routers lack the capability |
|
||||
|
||||
---
|
||||
|
||||
## 17. Scripts
|
||||
|
||||
`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).
|
||||
|
||||
---
|
||||
|
||||
## 18. Related Documents
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| [HLD.md](HLD.md) | System context and component view |
|
||||
| [data-flow.md](data-flow.md) | Narrative pipelines |
|
||||
| [configuration.md](configuration.md) | Env knobs |
|
||||
| [adr/](adr/) | Decision records |
|
||||
| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |
|
||||
@@ -6,12 +6,12 @@ Complete documentation for the RAG Chunking Benchmarker.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **New to the project?** Start with [Architecture Overview](architecture.md)
|
||||
2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
|
||||
3. **Need to use the API?** Check [API Reference](api-reference.md)
|
||||
4. **Configuring the system?** See [Configuration Guide](configuration.md)
|
||||
5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
|
||||
6. **Curious about data flow?** See [Data Flow](data-flow.md)
|
||||
1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail)
|
||||
2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md)
|
||||
3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
|
||||
4. **Need to use the API?** Check [API Reference](api-reference.md)
|
||||
5. **Configuring the system?** See [Configuration Guide](configuration.md)
|
||||
6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
|
||||
|
||||
---
|
||||
|
||||
@@ -19,13 +19,18 @@ Complete documentation for the RAG Chunking Benchmarker.
|
||||
|
||||
| File | Purpose | Audience |
|
||||
|------|---------|----------|
|
||||
| [architecture.md](architecture.md) | System structure and design | New team members |
|
||||
| [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members |
|
||||
| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers |
|
||||
| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members |
|
||||
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
|
||||
| [api-reference.md](api-reference.md) | All endpoints documented | Developers |
|
||||
| [configuration.md](configuration.md) | Settings and environment variables | DevOps |
|
||||
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
|
||||
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
|
||||
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
|
||||
| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators |
|
||||
| [human-eval-fixed-size-plus3-report.md](human-eval-fixed-size-plus3-report.md) | Human evaluation of fixed_size ±3 (sample + charts) | Managers, operators |
|
||||
| [adr/](adr/) | Architectural Decision Records | Everyone |
|
||||
| [phases.md](phases.md) | Implementation phases | Project managers |
|
||||
| [tasks.md](tasks.md) | Task tracking | Developers |
|
||||
|
||||
|
||||
24
docs/adr/0026-decision-board-tab.md
Normal file
24
docs/adr/0026-decision-board-tab.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# ADR-0026: Decision Board Tab for final Strategy selection
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (implemented)
|
||||
|
||||
## Context
|
||||
|
||||
After Neighbor Expansion Sweeps and semantic Boundary variants, the operator’s real job is choosing between `fixed_size` and `semantic` — not ad-hoc Experiment Compare. Compare remains useful for arbitrary side-by-side diffs, but it does not encode two-stage tuning (best variant per family, then family showdown) or a fixed 10-doc evaluation universe.
|
||||
|
||||
## Decision
|
||||
|
||||
- Add a top-level Dashboard **Decision** Tab (Decision Board).
|
||||
- **Strategy Candidates**: `fixed_size` at ±0…±3, and `semantic` under each Boundary Embedding Model present in data, under a selected Corpus Embedding Model.
|
||||
- Auto-discover from **single-strategy** Experiments on the hardcoded 10-doc set; newest Experiment fills each doc×Candidate cell; optional exclude chips; incomplete cells are `—` and omitted from that Candidate’s mean/wins.
|
||||
- Stage 1 ranks by **mean composite** (same weights as Experiments “Best”), shows win-count, allows manual override.
|
||||
- Stage 2 duels the two family winners with full metrics + per-doc head-to-head; per-doc matrix lists all Candidates with duel winners highlighted.
|
||||
- Aggregation is client-side from `GET /experiments?limit=…` (default list limit raised; max 500).
|
||||
|
||||
## Considered options
|
||||
|
||||
- Enhance Compare only — rejected (wrong job, crowded Benchmarks UX).
|
||||
- Standalone HTML report only — rejected (weak for override / interactive decision).
|
||||
- Manual pin of every cell — rejected for v1 (too heavy for ~60 cells).
|
||||
@@ -82,6 +82,9 @@ Delete a document and its vectors.
|
||||
|
||||
Run chunking strategies on a document.
|
||||
|
||||
**Query parameters:**
|
||||
- `background` (bool, default `false`) — when `true`, enqueue processing and return **202** with a `job_id`; poll `GET /jobs/{job_id}`.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
@@ -199,6 +202,9 @@ Retrieve a past query.
|
||||
|
||||
Run a benchmark comparing multiple strategies.
|
||||
|
||||
**Query parameters:**
|
||||
- `background` (bool, default `false`) — when `true`, enqueue the run and return **202** with a `job_id`; poll `GET /jobs/{job_id}` for status and result.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
@@ -278,6 +284,32 @@ List all experiments.
|
||||
|
||||
---
|
||||
|
||||
### Jobs
|
||||
|
||||
Background execution for long-running benchmarks and document processing (FastAPI `BackgroundTasks` + SQLite job records).
|
||||
|
||||
#### `GET /jobs`
|
||||
|
||||
List jobs (newest first).
|
||||
|
||||
**Query parameters:** `job_type`, `status`, `offset`, `limit`
|
||||
|
||||
#### `GET /jobs/{job_id}`
|
||||
|
||||
Poll job status. When `status` is `completed`, `result` contains the same payload as the synchronous endpoint would return; when `failed`, `error` is set.
|
||||
|
||||
**Response (202 enqueue body from POST with `background=true`):**
|
||||
```json
|
||||
{
|
||||
"job_id": "abc123",
|
||||
"job_type": "benchmark",
|
||||
"status": "pending",
|
||||
"poll_url": "/jobs/abc123"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Responses
|
||||
|
||||
All errors return:
|
||||
|
||||
458
docs/assets/decision/generate_charts.py
Normal file
458
docs/assets/decision/generate_charts.py
Normal file
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate SVG charts for the final chunking-strategy decision report.
|
||||
|
||||
Reads Experiment aggregates from SQLite (Decision Board rules) and writes
|
||||
SVG files next to this script. Re-run after new Experiments if needed:
|
||||
|
||||
.venv/bin/python docs/assets/decision/generate_charts.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
DB = ROOT / "data" / "chunking_benchmark.db"
|
||||
OUT = Path(__file__).resolve().parent
|
||||
|
||||
DECISION_DOCS = [
|
||||
"bazresi.docx",
|
||||
"customer1.docx",
|
||||
"fire.docx",
|
||||
"general-havades-individuals.doc",
|
||||
"havades.docx",
|
||||
"life-time-individual.docx",
|
||||
"moavenin.docx",
|
||||
"Refah.docx",
|
||||
"website.docx",
|
||||
"lifetime-compensation.docx",
|
||||
]
|
||||
SHORT = {
|
||||
"bazresi.docx": "bazresi",
|
||||
"customer1.docx": "customer1",
|
||||
"fire.docx": "fire",
|
||||
"general-havades-individuals.doc": "havades-ind",
|
||||
"havades.docx": "havades",
|
||||
"life-time-individual.docx": "lifetime-ind",
|
||||
"moavenin.docx": "moavenin",
|
||||
"Refah.docx": "Refah",
|
||||
"website.docx": "website",
|
||||
"lifetime-compensation.docx": "lifetime-comp",
|
||||
}
|
||||
CORPUS = "text-embedding-3-large"
|
||||
|
||||
TEAL_FILL = "#14b8a6"
|
||||
AMBER_FILL = "#f59e0b"
|
||||
INK = "#111827"
|
||||
MUTED = "#6b7280"
|
||||
GRID = "#e5e7eb"
|
||||
BG = "#ffffff"
|
||||
WIN = "#047857"
|
||||
|
||||
|
||||
def esc(s: str) -> str:
|
||||
return (
|
||||
str(s)
|
||||
.replace("&", "&")
|
||||
.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.replace('"', """)
|
||||
)
|
||||
|
||||
|
||||
def composite(m: dict | None) -> float | None:
|
||||
if not m:
|
||||
return None
|
||||
return (
|
||||
(m.get("avg_context_relevance") or 0) * 0.3
|
||||
+ (m.get("avg_answer_similarity") or 0) * 0.4
|
||||
+ (m.get("avg_faithfulness") or 0) * 0.3
|
||||
) * (1 - (m.get("hallucination_rate") or 0))
|
||||
|
||||
|
||||
def load_cells() -> dict[tuple[str, str], dict]:
|
||||
conn = sqlite3.connect(DB)
|
||||
conn.row_factory = sqlite3.Row
|
||||
docs = {r["id"]: r["filename"] for r in conn.execute("SELECT id, filename FROM documents")}
|
||||
rows = []
|
||||
for e in conn.execute(
|
||||
"""
|
||||
SELECT id, document_id, strategies_used, aggregate_metrics, benchmark_config,
|
||||
embedding_model_id, boundary_embedding_model_id, created_at, questions
|
||||
FROM experiments ORDER BY created_at DESC
|
||||
"""
|
||||
):
|
||||
fn = docs.get(e["document_id"])
|
||||
strats = json.loads(e["strategies_used"] or "[]")
|
||||
agg = json.loads(e["aggregate_metrics"] or "{}")
|
||||
cfg = json.loads(e["benchmark_config"] or "{}")
|
||||
corpus = e["embedding_model_id"] or cfg.get("corpus_embedding_model_id")
|
||||
prev = int(cfg.get("neighbor_prev") or 0)
|
||||
nxt = int(cfg.get("neighbor_next") or 0)
|
||||
bound = e["boundary_embedding_model_id"] or cfg.get("boundary_embedding_model_id")
|
||||
if fn not in DECISION_DOCS or corpus != CORPUS or len(strats) != 1:
|
||||
continue
|
||||
strat = strats[0]
|
||||
if strat == "fixed_size":
|
||||
if prev != nxt or prev not in (0, 1, 2, 3):
|
||||
continue
|
||||
cid = f"fixed_size:±{prev}"
|
||||
family = "fixed_size"
|
||||
elif strat == "semantic":
|
||||
if not bound:
|
||||
continue
|
||||
cid = f"semantic:{bound}"
|
||||
family = "semantic"
|
||||
else:
|
||||
continue
|
||||
rows.append(
|
||||
{
|
||||
"fn": fn,
|
||||
"cid": cid,
|
||||
"family": family,
|
||||
"agg": agg,
|
||||
"metrics": agg.get(family) or {},
|
||||
"id": e["id"],
|
||||
}
|
||||
)
|
||||
conn.close()
|
||||
cells: dict[tuple[str, str], dict] = {}
|
||||
for r in rows:
|
||||
key = (r["fn"], r["cid"])
|
||||
if key not in cells:
|
||||
cells[key] = r
|
||||
return cells
|
||||
|
||||
|
||||
def mean(xs: list[float]) -> float | None:
|
||||
return sum(xs) / len(xs) if xs else None
|
||||
|
||||
|
||||
def svg_wrap(w: int, h: int, body: str, title: str) -> str:
|
||||
return f'''<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="{w}" height="{h}" viewBox="0 0 {w} {h}" role="img" aria-label="{esc(title)}">
|
||||
<title>{esc(title)}</title>
|
||||
<rect width="{w}" height="{h}" fill="{BG}"/>
|
||||
{body}
|
||||
</svg>
|
||||
'''
|
||||
|
||||
|
||||
def chart_stage1(cells: dict) -> None:
|
||||
cands = [
|
||||
("fixed_size:±3", "fixed_size ±3", TEAL_FILL, True),
|
||||
("fixed_size:±2", "fixed_size ±2", TEAL_FILL, False),
|
||||
("fixed_size:±1", "fixed_size ±1", TEAL_FILL, False),
|
||||
("semantic:text-embedding-3-large", "semantic @ large", AMBER_FILL, False),
|
||||
("fixed_size:±0", "fixed_size ±0", "#99f6e4", False),
|
||||
("semantic:nomic-embed-text-v2-moe", "semantic @ nomic", "#fcd34d", False),
|
||||
]
|
||||
values = []
|
||||
for cid, label, color, winner in cands:
|
||||
scores = []
|
||||
for fn in DECISION_DOCS:
|
||||
r = cells.get((fn, cid))
|
||||
if r:
|
||||
sc = composite(r["metrics"])
|
||||
if sc is not None:
|
||||
scores.append(sc)
|
||||
values.append((cid, label, color, winner, mean(scores) or 0))
|
||||
|
||||
w, h = 820, 420
|
||||
left, right, top, bottom = 210, 40, 56, 48
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
vmin, vmax = 8.4, 9.2
|
||||
bar_h = plot_h / len(values) * 0.62
|
||||
gap = plot_h / len(values)
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="32" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 1 — mean composite (10 documents)</text>',
|
||||
f'<text x="24" y="50" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Higher is better. Official ranking used by Decision Board. Winner: fixed_size ±3.</text>',
|
||||
]
|
||||
# grid
|
||||
for tick in [8.5, 8.6, 8.7, 8.8, 8.9, 9.0, 9.1, 9.2]:
|
||||
x = left + (tick - vmin) / (vmax - vmin) * plot_w
|
||||
parts.append(f'<line x1="{x:.1f}" y1="{top}" x2="{x:.1f}" y2="{h - bottom}" stroke="{GRID}" stroke-width="1"/>')
|
||||
parts.append(
|
||||
f'<text x="{x:.1f}" y="{h - 18}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">{tick:.1f}</text>'
|
||||
)
|
||||
|
||||
for i, (_, label, color, winner, val) in enumerate(values):
|
||||
y = top + i * gap + (gap - bar_h) / 2
|
||||
bw = (val - vmin) / (vmax - vmin) * plot_w
|
||||
stroke = WIN if winner else "none"
|
||||
sw = 2 if winner else 0
|
||||
parts.append(
|
||||
f'<rect x="{left}" y="{y:.1f}" width="{max(bw, 0):.1f}" height="{bar_h:.1f}" rx="4" fill="{color}" stroke="{stroke}" stroke-width="{sw}"/>'
|
||||
)
|
||||
weight = "700" if winner else "500"
|
||||
parts.append(
|
||||
f'<text x="{left - 10}" y="{y + bar_h * 0.68:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="{weight}" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{left + bw + 8:.1f}" y="{y + bar_h * 0.68:.1f}" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="{INK}">{val:.3f}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage1-mean-composite.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 1 mean composite"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_stage2(cells: dict) -> None:
|
||||
fs_id = "fixed_size:±3"
|
||||
sem_id = "semantic:text-embedding-3-large"
|
||||
pairs = []
|
||||
for fn in DECISION_DOCS:
|
||||
a = composite(cells[(fn, fs_id)]["metrics"])
|
||||
b = composite(cells[(fn, sem_id)]["metrics"])
|
||||
pairs.append((SHORT[fn], a, b))
|
||||
|
||||
w, h = 920, 460
|
||||
left, right, top, bottom = 52, 24, 64, 88
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(pairs)
|
||||
slot = plot_w / n
|
||||
bar_w = slot * 0.32
|
||||
vmin, vmax = 5.8, 10.0
|
||||
|
||||
def y_of(v: float) -> float:
|
||||
return top + (1 - (v - vmin) / (vmax - vmin)) * plot_h
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 2 — per document (fixed_size ±3 vs semantic @ large)</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Mean composite still favors fixed_size. Semantic wins 6 of 10 docs, but by smaller margins except website/customer1.</text>',
|
||||
f'<rect x="620" y="14" width="12" height="12" rx="2" fill="{TEAL_FILL}"/>',
|
||||
f'<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">fixed_size ±3</text>',
|
||||
f'<rect x="760" y="14" width="12" height="12" rx="2" fill="{AMBER_FILL}"/>',
|
||||
f'<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">semantic @ large</text>',
|
||||
]
|
||||
for tick in [6, 7, 8, 9, 10]:
|
||||
y = y_of(tick)
|
||||
parts.append(f'<line x1="{left}" y1="{y:.1f}" x2="{w - right}" y2="{y:.1f}" stroke="{GRID}" stroke-width="1"/>')
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + 4:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">{tick}</text>'
|
||||
)
|
||||
|
||||
for i, (label, a, b) in enumerate(pairs):
|
||||
cx = left + i * slot + slot / 2
|
||||
xa = cx - bar_w - 3
|
||||
xb = cx + 3
|
||||
ha = plot_h - (y_of(a) - top)
|
||||
hb = plot_h - (y_of(b) - top)
|
||||
parts.append(f'<rect x="{xa:.1f}" y="{y_of(a):.1f}" width="{bar_w:.1f}" height="{ha:.1f}" rx="3" fill="{TEAL_FILL}"/>')
|
||||
parts.append(f'<rect x="{xb:.1f}" y="{y_of(b):.1f}" width="{bar_w:.1f}" height="{hb:.1f}" rx="3" fill="{AMBER_FILL}"/>')
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 36}" text-anchor="end" transform="rotate(-32 {cx:.1f} {h - 36})" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-per-document.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 per document"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_delta(cells: dict) -> None:
|
||||
fs_id = "fixed_size:±3"
|
||||
sem_id = "semantic:text-embedding-3-large"
|
||||
deltas = []
|
||||
for fn in DECISION_DOCS:
|
||||
a = composite(cells[(fn, fs_id)]["metrics"])
|
||||
b = composite(cells[(fn, sem_id)]["metrics"])
|
||||
deltas.append((SHORT[fn], a - b))
|
||||
deltas.sort(key=lambda x: x[1], reverse=True)
|
||||
|
||||
w, h = 820, 440
|
||||
left, right, top, bottom = 120, 56, 56, 36
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(deltas)
|
||||
gap = plot_h / n
|
||||
bar_h = gap * 0.62
|
||||
max_abs = max(abs(d) for _, d in deltas)
|
||||
# fire is 2.58, others < 1.2 — use 2.8
|
||||
max_abs = 2.8
|
||||
zero_x = left + plot_w / 2
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Margin: fixed_size ±3 minus semantic @ large</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Teal = fixed_size wins the document. Amber = semantic wins. Mean ranking is driven by large teal bars (especially fire).</text>',
|
||||
f'<line x1="{zero_x:.1f}" y1="{top}" x2="{zero_x:.1f}" y2="{h - bottom}" stroke="{INK}" stroke-width="1.2"/>',
|
||||
f'<text x="{left}" y="{h - 12}" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">semantic better</text>',
|
||||
f'<text x="{w - right}" y="{h - 12}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{MUTED}">fixed_size better</text>',
|
||||
]
|
||||
for i, (label, d) in enumerate(deltas):
|
||||
y = top + i * gap + (gap - bar_h) / 2
|
||||
bw = abs(d) / max_abs * (plot_w / 2)
|
||||
if d >= 0:
|
||||
x = zero_x
|
||||
color = TEAL_FILL
|
||||
tx = x + bw + 6
|
||||
anchor = "start"
|
||||
else:
|
||||
x = zero_x - bw
|
||||
color = AMBER_FILL
|
||||
tx = x - 6
|
||||
anchor = "end"
|
||||
parts.append(f'<rect x="{x:.1f}" y="{y:.1f}" width="{bw:.1f}" height="{bar_h:.1f}" rx="3" fill="{color}"/>')
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + bar_h * 0.7:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{tx:.1f}" y="{y + bar_h * 0.7:.1f}" text-anchor="{anchor}" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="{INK}">{d:+.2f}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-margins.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 margins"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_heatmap(cells: dict) -> None:
|
||||
cands = [
|
||||
("fixed_size:±0", "±0"),
|
||||
("fixed_size:±1", "±1"),
|
||||
("fixed_size:±2", "±2"),
|
||||
("fixed_size:±3", "±3"),
|
||||
("semantic:text-embedding-3-large", "sem@large"),
|
||||
("semantic:nomic-embed-text-v2-moe", "sem@nomic"),
|
||||
]
|
||||
scores: list[list[float | None]] = []
|
||||
for fn in DECISION_DOCS:
|
||||
row = []
|
||||
for cid, _ in cands:
|
||||
r = cells.get((fn, cid))
|
||||
row.append(composite(r["metrics"]) if r else None)
|
||||
scores.append(row)
|
||||
|
||||
cell_w, cell_h = 78, 32
|
||||
left, top = 128, 72
|
||||
w = left + cell_w * len(cands) + 24
|
||||
h = top + cell_h * len(DECISION_DOCS) + 36
|
||||
|
||||
def color_for(s: float | None) -> tuple[str, str]:
|
||||
if s is None:
|
||||
return "#f3f4f6", MUTED
|
||||
# 6.2 .. 9.8
|
||||
t = max(0.0, min(1.0, (s - 7.0) / (9.8 - 7.0)))
|
||||
# pale rose -> amber -> teal
|
||||
if t < 0.5:
|
||||
u = t * 2
|
||||
r = int(251 + (20 - 251) * 0)
|
||||
# interpolate rose 251,113,133 -> amber 245,158,11
|
||||
rr = int(251 + (245 - 251) * u)
|
||||
gg = int(113 + (158 - 113) * u)
|
||||
bb = int(133 + (11 - 133) * u)
|
||||
else:
|
||||
u = (t - 0.5) * 2
|
||||
rr = int(245 + (13 - 245) * u)
|
||||
gg = int(158 + (148 - 158) * u)
|
||||
bb = int(11 + (136 - 11) * u)
|
||||
bg = f"#{rr:02x}{gg:02x}{bb:02x}"
|
||||
fg = "#111827" if t < 0.72 else "#f9fafb"
|
||||
return bg, fg
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Candidate heatmap — composite by document</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Darker teal = stronger. Semantic @ large collapses on fire; ±1–±3 stay high across the set.</text>',
|
||||
]
|
||||
for j, (_, lab) in enumerate(cands):
|
||||
x = left + j * cell_w + cell_w / 2
|
||||
parts.append(
|
||||
f'<text x="{x:.1f}" y="{top - 10}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{esc(lab)}</text>'
|
||||
)
|
||||
for i, fn in enumerate(DECISION_DOCS):
|
||||
y = top + i * cell_h
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + cell_h * 0.65:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{INK}">{esc(SHORT[fn])}</text>'
|
||||
)
|
||||
for j, val in enumerate(scores[i]):
|
||||
x = left + j * cell_w
|
||||
bg, fg = color_for(val)
|
||||
parts.append(f'<rect x="{x}" y="{y}" width="{cell_w - 3}" height="{cell_h - 3}" rx="4" fill="{bg}"/>')
|
||||
txt = "—" if val is None else f"{val:.2f}"
|
||||
parts.append(
|
||||
f'<text x="{x + (cell_w - 3) / 2:.1f}" y="{y + cell_h * 0.62:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{fg}">{txt}</text>'
|
||||
)
|
||||
|
||||
(OUT / "heatmap-candidates.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Candidate heatmap"), encoding="utf-8")
|
||||
|
||||
|
||||
def chart_metrics(cells: dict) -> None:
|
||||
"""Grouped bars: four judge metrics for the two stage-2 winners."""
|
||||
def bag(cid: str, family: str) -> dict[str, float]:
|
||||
keys = [
|
||||
"avg_context_relevance",
|
||||
"avg_answer_similarity",
|
||||
"avg_faithfulness",
|
||||
"hallucination_rate",
|
||||
]
|
||||
acc = {k: [] for k in keys}
|
||||
for fn in DECISION_DOCS:
|
||||
m = cells[(fn, cid)]["metrics"]
|
||||
for k in keys:
|
||||
acc[k].append(m[k])
|
||||
return {k: sum(v) / len(v) for k, v in acc.items()}
|
||||
|
||||
fs = bag("fixed_size:±3", "fixed_size")
|
||||
sem = bag("semantic:text-embedding-3-large", "semantic")
|
||||
labels = [
|
||||
("Context relevance", fs["avg_context_relevance"], sem["avg_context_relevance"], False),
|
||||
("Answer similarity", fs["avg_answer_similarity"], sem["avg_answer_similarity"], False),
|
||||
("Faithfulness", fs["avg_faithfulness"], sem["avg_faithfulness"], False),
|
||||
("Hallucination % (lower better)", fs["hallucination_rate"] * 100, sem["hallucination_rate"] * 100, True),
|
||||
]
|
||||
|
||||
w, h = 820, 380
|
||||
left, right, top, bottom = 52, 24, 64, 48
|
||||
plot_w = w - left - right
|
||||
plot_h = h - top - bottom
|
||||
n = len(labels)
|
||||
slot = plot_w / n
|
||||
bar_w = slot * 0.28
|
||||
|
||||
parts = [
|
||||
f'<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Stage 2 winners — mean judge metrics</text>',
|
||||
f'<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">fixed_size ±3 leads on all four metrics, including lower hallucination.</text>',
|
||||
f'<rect x="620" y="14" width="12" height="12" rx="2" fill="{TEAL_FILL}"/>',
|
||||
f'<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">fixed_size ±3</text>',
|
||||
f'<rect x="760" y="14" width="12" height="12" rx="2" fill="{AMBER_FILL}"/>',
|
||||
f'<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">semantic @ large</text>',
|
||||
]
|
||||
|
||||
for i, (lab, a, b, is_pct) in enumerate(labels):
|
||||
vmax = 10 if not is_pct else max(a, b) * 1.35
|
||||
cx = left + i * slot + slot / 2
|
||||
xa = cx - bar_w - 3
|
||||
xb = cx + 3
|
||||
|
||||
def bar(x: float, val: float, color: str) -> str:
|
||||
bh = val / vmax * plot_h
|
||||
y = top + plot_h - bh
|
||||
return f'<rect x="{x:.1f}" y="{y:.1f}" width="{bar_w:.1f}" height="{bh:.1f}" rx="3" fill="{color}"/>'
|
||||
|
||||
parts.append(bar(xa, a, TEAL_FILL))
|
||||
parts.append(bar(xb, b, AMBER_FILL))
|
||||
fmt = (lambda v: f"{v:.1f}%") if is_pct else (lambda v: f"{v:.2f}")
|
||||
ya = top + plot_h - a / vmax * plot_h - 6
|
||||
yb = top + plot_h - b / vmax * plot_h - 6
|
||||
parts.append(
|
||||
f'<text x="{xa + bar_w / 2:.1f}" y="{ya:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{fmt(a)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{xb + bar_w / 2:.1f}" y="{yb:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{fmt(b)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 16}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{esc(lab)}</text>'
|
||||
)
|
||||
|
||||
(OUT / "stage2-metrics.svg").write_text(svg_wrap(w, h, "\n".join(parts), "Stage 2 mean metrics"), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cells = load_cells()
|
||||
chart_stage1(cells)
|
||||
chart_stage2(cells)
|
||||
chart_delta(cells)
|
||||
chart_heatmap(cells)
|
||||
chart_metrics(cells)
|
||||
print(f"Wrote SVGs in {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
143
docs/assets/decision/heatmap-candidates.svg
Normal file
143
docs/assets/decision/heatmap-candidates.svg
Normal file
@@ -0,0 +1,143 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="620" height="428" viewBox="0 0 620 428" role="img" aria-label="Candidate heatmap">
|
||||
<title>Candidate heatmap</title>
|
||||
<rect width="620" height="428" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Candidate heatmap — composite by document</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Darker teal = stronger. Semantic @ large collapses on fire; ±1–±3 stay high across the set.</text>
|
||||
<text x="167.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±0</text>
|
||||
<text x="245.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±1</text>
|
||||
<text x="323.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±2</text>
|
||||
<text x="401.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">±3</text>
|
||||
<text x="479.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">sem@large</text>
|
||||
<text x="557.0" y="62" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">sem@nomic</text>
|
||||
<text x="120" y="92.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">bazresi</text>
|
||||
<rect x="128" y="72" width="75" height="29" rx="4" fill="#e19d15"/>
|
||||
<text x="165.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.52</text>
|
||||
<rect x="206" y="72" width="75" height="29" rx="4" fill="#58975f"/>
|
||||
<text x="243.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.34</text>
|
||||
<rect x="284" y="72" width="75" height="29" rx="4" fill="#58975f"/>
|
||||
<text x="321.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.34</text>
|
||||
<rect x="362" y="72" width="75" height="29" rx="4" fill="#549761"/>
|
||||
<text x="399.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.37</text>
|
||||
<rect x="440" y="72" width="75" height="29" rx="4" fill="#4e9664"/>
|
||||
<text x="477.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.40</text>
|
||||
<rect x="518" y="72" width="75" height="29" rx="4" fill="#7c984b"/>
|
||||
<text x="555.5" y="91.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.12</text>
|
||||
<text x="120" y="124.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">customer1</text>
|
||||
<rect x="128" y="104" width="75" height="29" rx="4" fill="#9f9a39"/>
|
||||
<text x="165.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.92</text>
|
||||
<rect x="206" y="104" width="75" height="29" rx="4" fill="#96993e"/>
|
||||
<text x="243.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.97</text>
|
||||
<rect x="284" y="104" width="75" height="29" rx="4" fill="#97993d"/>
|
||||
<text x="321.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.96</text>
|
||||
<rect x="362" y="104" width="75" height="29" rx="4" fill="#ec9d0f"/>
|
||||
<text x="399.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.45</text>
|
||||
<rect x="440" y="104" width="75" height="29" rx="4" fill="#96993e"/>
|
||||
<text x="477.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.97</text>
|
||||
<rect x="518" y="104" width="75" height="29" rx="4" fill="#f88255"/>
|
||||
<text x="555.5" y="123.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.54</text>
|
||||
<text x="120" y="156.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">fire</text>
|
||||
<rect x="128" y="136" width="75" height="29" rx="4" fill="#f69426"/>
|
||||
<text x="165.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.09</text>
|
||||
<rect x="206" y="136" width="75" height="29" rx="4" fill="#f59c10"/>
|
||||
<text x="243.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.34</text>
|
||||
<rect x="284" y="136" width="75" height="29" rx="4" fill="#f68f33"/>
|
||||
<text x="321.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.93</text>
|
||||
<rect x="362" y="136" width="75" height="29" rx="4" fill="#b19b2f"/>
|
||||
<text x="399.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.81</text>
|
||||
<rect x="440" y="136" width="75" height="29" rx="4" fill="#fb7185"/>
|
||||
<text x="477.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">6.23</text>
|
||||
<rect x="518" y="136" width="75" height="29" rx="4" fill="#fa7674"/>
|
||||
<text x="555.5" y="155.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.19</text>
|
||||
<text x="120" y="188.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades-ind</text>
|
||||
<rect x="128" y="168" width="75" height="29" rx="4" fill="#f78b3c"/>
|
||||
<text x="165.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.83</text>
|
||||
<rect x="206" y="168" width="75" height="29" rx="4" fill="#c99c22"/>
|
||||
<text x="243.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.66</text>
|
||||
<rect x="284" y="168" width="75" height="29" rx="4" fill="#7e984a"/>
|
||||
<text x="321.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.12</text>
|
||||
<rect x="362" y="168" width="75" height="29" rx="4" fill="#6f9852"/>
|
||||
<text x="399.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.21</text>
|
||||
<rect x="440" y="168" width="75" height="29" rx="4" fill="#d99c19"/>
|
||||
<text x="477.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.56</text>
|
||||
<rect x="518" y="168" width="75" height="29" rx="4" fill="#e09d16"/>
|
||||
<text x="555.5" y="187.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.53</text>
|
||||
<text x="120" y="220.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades</text>
|
||||
<rect x="128" y="200" width="75" height="29" rx="4" fill="#f59b12"/>
|
||||
<text x="165.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.31</text>
|
||||
<rect x="206" y="200" width="75" height="29" rx="4" fill="#ef9d0d"/>
|
||||
<text x="243.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.43</text>
|
||||
<rect x="284" y="200" width="75" height="29" rx="4" fill="#f49d0b"/>
|
||||
<text x="321.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.40</text>
|
||||
<rect x="362" y="200" width="75" height="29" rx="4" fill="#e79d12"/>
|
||||
<text x="399.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.48</text>
|
||||
<rect x="440" y="200" width="75" height="29" rx="4" fill="#f69030"/>
|
||||
<text x="477.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.97</text>
|
||||
<rect x="518" y="200" width="75" height="29" rx="4" fill="#f29d0c"/>
|
||||
<text x="555.5" y="219.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.42</text>
|
||||
<text x="120" y="252.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-ind</text>
|
||||
<rect x="128" y="232" width="75" height="29" rx="4" fill="#6d9854"/>
|
||||
<text x="165.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.22</text>
|
||||
<rect x="206" y="232" width="75" height="29" rx="4" fill="#4b9666"/>
|
||||
<text x="243.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.42</text>
|
||||
<rect x="284" y="232" width="75" height="29" rx="4" fill="#489667"/>
|
||||
<text x="321.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.44</text>
|
||||
<rect x="362" y="232" width="75" height="29" rx="4" fill="#479668"/>
|
||||
<text x="399.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.45</text>
|
||||
<rect x="440" y="232" width="75" height="29" rx="4" fill="#3d966d"/>
|
||||
<text x="477.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.50</text>
|
||||
<rect x="518" y="232" width="75" height="29" rx="4" fill="#4e9664"/>
|
||||
<text x="555.5" y="251.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.41</text>
|
||||
<text x="120" y="284.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">moavenin</text>
|
||||
<rect x="128" y="264" width="75" height="29" rx="4" fill="#879945"/>
|
||||
<text x="165.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.06</text>
|
||||
<rect x="206" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="243.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="284" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="321.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="362" y="264" width="75" height="29" rx="4" fill="#179482"/>
|
||||
<text x="399.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.73</text>
|
||||
<rect x="440" y="264" width="75" height="29" rx="4" fill="#d09c1e"/>
|
||||
<text x="477.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.62</text>
|
||||
<rect x="518" y="264" width="75" height="29" rx="4" fill="#f59a14"/>
|
||||
<text x="555.5" y="283.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.29</text>
|
||||
<text x="120" y="316.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">Refah</text>
|
||||
<rect x="128" y="296" width="75" height="29" rx="4" fill="#459669"/>
|
||||
<text x="165.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.46</text>
|
||||
<rect x="206" y="296" width="75" height="29" rx="4" fill="#1c947f"/>
|
||||
<text x="243.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.71</text>
|
||||
<rect x="284" y="296" width="75" height="29" rx="4" fill="#1e947e"/>
|
||||
<text x="321.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.70</text>
|
||||
<rect x="362" y="296" width="75" height="29" rx="4" fill="#20947d"/>
|
||||
<text x="399.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.68</text>
|
||||
<rect x="440" y="296" width="75" height="29" rx="4" fill="#0d9488"/>
|
||||
<text x="477.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.81</text>
|
||||
<rect x="518" y="296" width="75" height="29" rx="4" fill="#349572"/>
|
||||
<text x="555.5" y="315.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.56</text>
|
||||
<text x="120" y="348.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">website</text>
|
||||
<rect x="128" y="328" width="75" height="29" rx="4" fill="#f8864b"/>
|
||||
<text x="165.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.67</text>
|
||||
<rect x="206" y="328" width="75" height="29" rx="4" fill="#f78e35"/>
|
||||
<text x="243.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">7.91</text>
|
||||
<rect x="284" y="328" width="75" height="29" rx="4" fill="#e29d15"/>
|
||||
<text x="321.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.51</text>
|
||||
<rect x="362" y="328" width="75" height="29" rx="4" fill="#ea9d10"/>
|
||||
<text x="399.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.47</text>
|
||||
<rect x="440" y="328" width="75" height="29" rx="4" fill="#879945"/>
|
||||
<text x="477.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.06</text>
|
||||
<rect x="518" y="328" width="75" height="29" rx="4" fill="#e59d13"/>
|
||||
<text x="555.5" y="347.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.49</text>
|
||||
<text x="120" y="380.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-comp</text>
|
||||
<rect x="128" y="360" width="75" height="29" rx="4" fill="#93993f"/>
|
||||
<text x="165.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.99</text>
|
||||
<rect x="206" y="360" width="75" height="29" rx="4" fill="#bd9b28"/>
|
||||
<text x="243.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.73</text>
|
||||
<rect x="284" y="360" width="75" height="29" rx="4" fill="#bd9b28"/>
|
||||
<text x="321.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.73</text>
|
||||
<rect x="362" y="360" width="75" height="29" rx="4" fill="#b99b2b"/>
|
||||
<text x="399.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.76</text>
|
||||
<rect x="440" y="360" width="75" height="29" rx="4" fill="#7c984c"/>
|
||||
<text x="477.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#f9fafb">9.13</text>
|
||||
<rect x="518" y="360" width="75" height="29" rx="4" fill="#98993d"/>
|
||||
<text x="555.5" y="379.8" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.96</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 16 KiB |
41
docs/assets/decision/stage1-mean-composite.svg
Normal file
41
docs/assets/decision/stage1-mean-composite.svg
Normal file
@@ -0,0 +1,41 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="420" viewBox="0 0 820 420" role="img" aria-label="Stage 1 mean composite">
|
||||
<title>Stage 1 mean composite</title>
|
||||
<rect width="820" height="420" fill="#ffffff"/>
|
||||
<text x="24" y="32" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 1 — mean composite (10 documents)</text>
|
||||
<text x="24" y="50" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Higher is better. Official ranking used by Decision Board. Winner: fixed_size ±3.</text>
|
||||
<line x1="281.2" y1="56" x2="281.2" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="281.2" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.5</text>
|
||||
<line x1="352.5" y1="56" x2="352.5" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="352.5" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.6</text>
|
||||
<line x1="423.7" y1="56" x2="423.7" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="423.7" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.7</text>
|
||||
<line x1="495.0" y1="56" x2="495.0" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="495.0" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.8</text>
|
||||
<line x1="566.3" y1="56" x2="566.3" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="566.3" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8.9</text>
|
||||
<line x1="637.5" y1="56" x2="637.5" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="637.5" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.0</text>
|
||||
<line x1="708.8" y1="56" x2="708.8" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="708.8" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.1</text>
|
||||
<line x1="780.0" y1="56" x2="780.0" y2="372" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="780.0" y="402" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9.2</text>
|
||||
<rect x="210" y="66.0" width="456.2" height="32.7" rx="4" fill="#14b8a6" stroke="#047857" stroke-width="2"/>
|
||||
<text x="200" y="88.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="700" fill="#111827">fixed_size ±3</text>
|
||||
<text x="674.2" y="88.2" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">9.040</text>
|
||||
<rect x="210" y="118.7" width="418.4" height="32.7" rx="4" fill="#14b8a6" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="140.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±2</text>
|
||||
<text x="636.4" y="140.9" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.987</text>
|
||||
<rect x="210" y="171.3" width="374.8" height="32.7" rx="4" fill="#14b8a6" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="193.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±1</text>
|
||||
<text x="592.8" y="193.5" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.926</text>
|
||||
<rect x="210" y="224.0" width="232.8" height="32.7" rx="4" fill="#f59e0b" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="246.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">semantic @ large</text>
|
||||
<text x="450.8" y="246.2" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.727</text>
|
||||
<rect x="210" y="276.7" width="146.7" height="32.7" rx="4" fill="#99f6e4" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="298.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">fixed_size ±0</text>
|
||||
<text x="364.7" y="298.9" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.606</text>
|
||||
<rect x="210" y="329.3" width="107.4" height="32.7" rx="4" fill="#fcd34d" stroke="none" stroke-width="0"/>
|
||||
<text x="200" y="351.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="500" fill="#111827">semantic @ nomic</text>
|
||||
<text x="325.4" y="351.5" font-family="Inter, system-ui, sans-serif" font-size="13" font-weight="600" fill="#111827">8.551</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
40
docs/assets/decision/stage2-margins.svg
Normal file
40
docs/assets/decision/stage2-margins.svg
Normal file
@@ -0,0 +1,40 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="440" viewBox="0 0 820 440" role="img" aria-label="Stage 2 margins">
|
||||
<title>Stage 2 margins</title>
|
||||
<rect width="820" height="440" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Margin: fixed_size ±3 minus semantic @ large</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Teal = fixed_size wins the document. Amber = semantic wins. Mean ranking is driven by large teal bars (especially fire).</text>
|
||||
<line x1="442.0" y1="56" x2="442.0" y2="404" stroke="#111827" stroke-width="1.2"/>
|
||||
<text x="120" y="428" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">semantic better</text>
|
||||
<text x="764" y="428" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">fixed_size better</text>
|
||||
<rect x="442.0" y="62.6" width="296.9" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="77.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fire</text>
|
||||
<text x="744.9" y="77.7" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+2.58</text>
|
||||
<rect x="442.0" y="97.4" width="128.1" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="112.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">moavenin</text>
|
||||
<text x="576.1" y="112.5" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+1.11</text>
|
||||
<rect x="442.0" y="132.2" width="73.7" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="147.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades-ind</text>
|
||||
<text x="521.7" y="147.3" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+0.64</text>
|
||||
<rect x="442.0" y="167.0" width="59.1" height="21.6" rx="3" fill="#14b8a6"/>
|
||||
<text x="112" y="182.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades</text>
|
||||
<text x="507.1" y="182.1" text-anchor="start" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">+0.51</text>
|
||||
<rect x="437.5" y="201.8" width="4.5" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="216.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">bazresi</text>
|
||||
<text x="431.5" y="216.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.04</text>
|
||||
<rect x="435.1" y="236.6" width="6.9" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="251.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-ind</text>
|
||||
<text x="429.1" y="251.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.06</text>
|
||||
<rect x="426.5" y="271.4" width="15.5" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="286.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Refah</text>
|
||||
<text x="420.5" y="286.5" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.13</text>
|
||||
<rect x="399.7" y="306.2" width="42.3" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="321.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-comp</text>
|
||||
<text x="393.7" y="321.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.37</text>
|
||||
<rect x="382.0" y="341.0" width="60.0" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="356.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">customer1</text>
|
||||
<text x="376.0" y="356.1" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.52</text>
|
||||
<rect x="373.9" y="375.8" width="68.1" height="21.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="112" y="390.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">website</text>
|
||||
<text x="367.9" y="390.9" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">-0.59</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.4 KiB |
31
docs/assets/decision/stage2-metrics.svg
Normal file
31
docs/assets/decision/stage2-metrics.svg
Normal file
@@ -0,0 +1,31 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="820" height="380" viewBox="0 0 820 380" role="img" aria-label="Stage 2 mean metrics">
|
||||
<title>Stage 2 mean metrics</title>
|
||||
<rect width="820" height="380" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 2 winners — mean judge metrics</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">fixed_size ±3 leads on all four metrics, including lower hallucination.</text>
|
||||
<rect x="620" y="14" width="12" height="12" rx="2" fill="#14b8a6"/>
|
||||
<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fixed_size ±3</text>
|
||||
<rect x="760" y="14" width="12" height="12" rx="2" fill="#f59e0b"/>
|
||||
<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">semantic @ large</text>
|
||||
<rect x="89.9" y="80.0" width="52.1" height="252.0" rx="3" fill="#14b8a6"/>
|
||||
<rect x="148.0" y="80.7" width="52.1" height="251.3" rx="3" fill="#f59e0b"/>
|
||||
<text x="116.0" y="74.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.40</text>
|
||||
<text x="174.0" y="74.7" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.38</text>
|
||||
<text x="145.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Context relevance</text>
|
||||
<rect x="275.9" y="92.1" width="52.1" height="239.9" rx="3" fill="#14b8a6"/>
|
||||
<rect x="334.0" y="95.3" width="52.1" height="236.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="302.0" y="86.1" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.95</text>
|
||||
<text x="360.0" y="89.3" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8.83</text>
|
||||
<text x="331.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Answer similarity</text>
|
||||
<rect x="461.9" y="78.2" width="52.1" height="253.8" rx="3" fill="#14b8a6"/>
|
||||
<rect x="520.0" y="82.7" width="52.1" height="249.3" rx="3" fill="#f59e0b"/>
|
||||
<text x="488.0" y="72.2" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.47</text>
|
||||
<text x="546.0" y="76.7" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">9.30</text>
|
||||
<text x="517.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Faithfulness</text>
|
||||
<rect x="647.9" y="241.0" width="52.1" height="91.0" rx="3" fill="#14b8a6"/>
|
||||
<rect x="706.0" y="133.5" width="52.1" height="198.5" rx="3" fill="#f59e0b"/>
|
||||
<text x="674.0" y="235.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">2.2%</text>
|
||||
<text x="732.0" y="127.5" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">4.8%</text>
|
||||
<text x="703.0" y="364" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Hallucination % (lower better)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.3 KiB |
51
docs/assets/decision/stage2-per-document.svg
Normal file
51
docs/assets/decision/stage2-per-document.svg
Normal file
@@ -0,0 +1,51 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="920" height="460" viewBox="0 0 920 460" role="img" aria-label="Stage 2 per document">
|
||||
<title>Stage 2 per document</title>
|
||||
<rect width="920" height="460" fill="#ffffff"/>
|
||||
<text x="24" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Stage 2 — per document (fixed_size ±3 vs semantic @ large)</text>
|
||||
<text x="24" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Mean composite still favors fixed_size. Semantic wins 6 of 10 docs, but by smaller margins except website/customer1.</text>
|
||||
<rect x="620" y="14" width="12" height="12" rx="2" fill="#14b8a6"/>
|
||||
<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fixed_size ±3</text>
|
||||
<rect x="760" y="14" width="12" height="12" rx="2" fill="#f59e0b"/>
|
||||
<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">semantic @ large</text>
|
||||
<line x1="52" y1="357.3" x2="896" y2="357.3" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="361.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">6</text>
|
||||
<line x1="52" y1="284.0" x2="896" y2="284.0" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="288.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">7</text>
|
||||
<line x1="52" y1="210.7" x2="896" y2="210.7" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="214.7" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">8</text>
|
||||
<line x1="52" y1="137.3" x2="896" y2="137.3" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="141.3" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">9</text>
|
||||
<line x1="52" y1="64.0" x2="896" y2="64.0" stroke="#e5e7eb" stroke-width="1"/>
|
||||
<text x="44" y="68.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#6b7280">10</text>
|
||||
<rect x="64.2" y="110.5" width="27.0" height="261.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="97.2" y="107.6" width="27.0" height="264.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="94.2" y="424" text-anchor="end" transform="rotate(-32 94.2 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">bazresi</text>
|
||||
<rect x="148.6" y="177.6" width="27.0" height="194.4" rx="3" fill="#14b8a6"/>
|
||||
<rect x="181.6" y="139.3" width="27.0" height="232.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="178.6" y="424" text-anchor="end" transform="rotate(-32 178.6 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">customer1</text>
|
||||
<rect x="233.0" y="151.3" width="27.0" height="220.7" rx="3" fill="#14b8a6"/>
|
||||
<rect x="266.0" y="340.6" width="27.0" height="31.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="263.0" y="424" text-anchor="end" transform="rotate(-32 263.0 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">fire</text>
|
||||
<rect x="317.4" y="122.3" width="27.0" height="249.7" rx="3" fill="#14b8a6"/>
|
||||
<rect x="350.4" y="169.3" width="27.0" height="202.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="347.4" y="424" text-anchor="end" transform="rotate(-32 347.4 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades-ind</text>
|
||||
<rect x="401.8" y="175.2" width="27.0" height="196.8" rx="3" fill="#14b8a6"/>
|
||||
<rect x="434.8" y="212.9" width="27.0" height="159.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="431.8" y="424" text-anchor="end" transform="rotate(-32 431.8 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades</text>
|
||||
<rect x="486.2" y="104.7" width="27.0" height="267.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="519.2" y="100.3" width="27.0" height="271.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="516.2" y="424" text-anchor="end" transform="rotate(-32 516.2 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-ind</text>
|
||||
<rect x="570.6" y="83.4" width="27.0" height="288.6" rx="3" fill="#14b8a6"/>
|
||||
<rect x="603.6" y="165.1" width="27.0" height="206.9" rx="3" fill="#f59e0b"/>
|
||||
<text x="600.6" y="424" text-anchor="end" transform="rotate(-32 600.6 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">moavenin</text>
|
||||
<rect x="655.0" y="87.5" width="27.0" height="284.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="688.0" y="77.6" width="27.0" height="294.4" rx="3" fill="#f59e0b"/>
|
||||
<text x="685.0" y="424" text-anchor="end" transform="rotate(-32 685.0 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">Refah</text>
|
||||
<rect x="739.4" y="176.5" width="27.0" height="195.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="772.4" y="133.1" width="27.0" height="238.9" rx="3" fill="#f59e0b"/>
|
||||
<text x="769.4" y="424" text-anchor="end" transform="rotate(-32 769.4 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">website</text>
|
||||
<rect x="823.8" y="154.9" width="27.0" height="217.1" rx="3" fill="#14b8a6"/>
|
||||
<rect x="856.8" y="127.9" width="27.0" height="244.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="853.8" y="424" text-anchor="end" transform="rotate(-32 853.8 424)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-comp</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.1 KiB |
61
docs/assets/human-eval/fixed_size_plus3_human_scores.csv
Normal file
61
docs/assets/human-eval/fixed_size_plus3_human_scores.csv
Normal file
@@ -0,0 +1,61 @@
|
||||
document,question_id,question,expected_answer,generated_answer,llm_context_relevance,llm_answer_similarity,llm_faithfulness,llm_hallucination,llm_composite,human_score,human_notes,agreement,experiment_id
|
||||
bazresi.docx,q1,این سند در مورد چیست؟,این سند شامل پرسشهای پرتکرار از سوی مشتریان در اداره پاسخگویی و رسیدگی به شکایات شرکت بیمه آسیا است که پاسخهایی درباره مهلت پرداخت خسارت، ثبت شکایت، شماره شبا و سایر موارد اداری ارائه میدهد.,این سند در مورد پرسشهای متداول مشتریان در اداره پاسخگویی و رسیدگی به شکایات بیمه آسیا و اطلاعات مربوط به پرداخت خسارت، نحوه دریافت بیمهنامه، ثبت شکایت و سایر خدمات بیمهای است.,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,6fe08750ee324130b3d46d8b3bc95280
|
||||
bazresi.docx,q3,مدت زمان پرداخت خسارت بدنه چقدر است؟,طبق ماده 20 آیین نامه 53 بیمه بدنه، مدت زمان پرداخت خسارت بدنه 15 روز کاری پس از تکمیل پرونده میباشد. در صورت تأیید اصالت حادثه و بالا بودن مبلغ خسارت، امکان واریز علی الحساب تا 60 درصد مبلغ خسارت وجود دارد که منوط به ثبت درخواست کتبی از سوی زیاندیده در شعبه پرداخت خسارت است.,مدت زمان پرداخت خسارت بدنه 15 روز کاری پس از تکمیل پرونده می باشد.,10,8,9,False,8.9,4,Aligns with expected answer; key points covered.,agree,6fe08750ee324130b3d46d8b3bc95280
|
||||
bazresi.docx,q6,نحوه ثبت الحاقیه یا فسخ بیمهنامه بدنه مربوط به خودروهای شرکتهای لیزینگی چگونه است؟,بیمهگذار (مستأجر خودرو) باید به واحد صدور یا شعبه ناظر واحد صادرکننده مراجعه نموده و با ارائه اصل بیمهنامه بدنه، مدارک فک رهنی، سند مالکیت و درخواست کتبی نسبت به صدور الحاقیه یا فسخ اقدام نماید.,بیمهگذار (مستأجر خودرو) باید به واحد صدور ذکر شده در بیمهنامه یا شعبه ناظر واحد صادر کننده مراجعه نموده و با ارائه اصل بیمهنامه بدنه، مدارک فک رهنی، سند مالکیت و درخواست کتبی خود مبنی بر صدور الحاقیه یا فسخ بیمهنامه نسبت به این مهم اقدام نماید. در صورتی که بیمهگذار، ساکن استانی غیر از استان محل استقرار واحد صدور بیمهنامه باشد، باید با در دست داشتن مدارک فوقالذکر به شعبه مرکزی آن استان مراجعه و درخواست کتبی خود را ارائه نماید.,10,9,10,False,9.6,5,Aligns with expected answer; key points covered.,agree,6fe08750ee324130b3d46d8b3bc95280
|
||||
bazresi.docx,q9,آیا نتیجه اعتراض به مبلغ خسارت کارشناس الزاماً منجر به افزایش مبلغ میشود؟,خیر. زیاندیده میتواند با مراجعه به رئیس شعبه خسارت، درخواست کتبی جهت اعتراض ثبت نماید. اما نتیجه اعتراض الزاماً منجر به افزایش مبلغ خسارت نمیشود و ممکن است پس از کارشناسی مجدد، مبلغ تأیید شود یا حتی کاهش یابد.,خیر، نتیجه اعتراض الزاماً منجر به افزایش میزان مبلغ خسارت نمیشود و ممکن است پس از بررسی و کارشناسی مجدد، مبلغ مربوطه تأیید شود یا کاهش یابد.,9,9,10,False,9.3,5,Aligns with expected answer; key points covered.,agree,6fe08750ee324130b3d46d8b3bc95280
|
||||
bazresi.docx,q11,آیا امکان دریافت فیزیک بیمهنامه ثالث از طریق لینک پیامکی وجود نداشته باشد، چه باید کرد؟,در صورت عدم دسترسی به لینک، با مراجعه به سایت بیمه آسیا (www.bimehasia.com) و بخش خدمات الکترونیک -> استعلامها -> استعلام و چاپ بیمهنامه ثالث با وارد کردن کد بیمهگذار و شماره بیمهنامه میتوان فایل را دریافت کرد.,در صورتی که از طریق لینک پیامکی قادر به ورود به صفحه مربوطه نبودید، میتوانید با مراجعه به سایت بیمه آسیا به نشانی www.bimehasia.com و ورود به قسمت خدمات الکترونیک، استعلامها، استعلام و چاپ بیمه نامه ثالث و با وارد کردن کد بیمهگذار و شماره بیمهنامه مربوطه، نسبت به مشاهده، چاپ یا دریافت فایل بیمهنامه ثالث اقدام نمایید.,9,9,10,False,9.3,4,Aligns with expected answer; key points covered.,partial,6fe08750ee324130b3d46d8b3bc95280
|
||||
bazresi.docx,q14,شماره تماس واحد پاسخگویی و رسیدگی به شکایات چیست؟,در سند اطلاعات تماس مستقیم واحد پاسخگویی ذکر نشده اما از طریق سامانههای رسیدگی به شکایات بیمه مرکزی یا بازرسی کل کشور یا از طریق سایت بیمه آسیا قابل دسترسی است.,متأسفانه، اطلاعاتی درباره شماره تماس واحد پاسخگویی و رسیدگی به شکایات در متن موجود نیست.,7,8,9,False,8.0,5,Aligns with expected answer; key points covered.,partial,6fe08750ee324130b3d46d8b3bc95280
|
||||
customer1.docx,q1,این سند در مورد چیست؟,این سند شامل پرسشهای متداول درباره استعلام و مناقصه بیمهای، توسعه بازار و کد یونیک، و شرایط انعقاد تفاهمنامه شرکت بیمه آسیا است.,این سند در مورد مناقصات بیمهای و استعلامات مرتبط با آن است. اطلاعاتی درباره تعریف مناقصه، انواع مناقصات، قوانین برگزاری، و شرایط مختلف مرتبط با مناقصات بیمهای ارائه میدهد.,8,7,8,False,7.6,4,Aligns with expected answer; key points covered.,agree,8c4d888d367b4c0299b2d19dd9654f80
|
||||
customer1.docx,q4,مناقصه بیمهای و استعلام بیمهای چه تفاوتی با یکدیگر دارند؟,تفاوت کاربرد استعلام بها و مناقصه به میزان مبلغ معامله بستگی خواهد داشت. بر اساس ماده 3 قانون برگزاری مناقصات، معاملات کوچک و متوسط (بالاتر از ده درصد نصاب مربوط) متفاوت از معاملات بزرگ اجرا میشوند.,تفاوت مناقصه بیمهای و استعلام بیمهای به میزان مبلغ معامله بستگی دارد. برای معاملات کوچک و متوسط (تا 2.850 میلیون ریال) از استعلام بها استفاده میشود، در حالی که برای معاملات بزرگ (بیش از 28.500 میلیون ریال) باید از طریق مناقصه عمومی یا محدود اقدام شود.,9,8,9,False,8.6,4,Aligns with expected answer; key points covered.,agree,8c4d888d367b4c0299b2d19dd9654f80
|
||||
customer1.docx,q7,آیا ثبت معاملات در سامانه ستاد الزامی است؟,بله. ثبت کلیه معاملات دولت اعم از کوچک و متوسط و بزرگ در سامانه ستاد الزامی است. طبق مکاتبه معاونت حقوقی رئیس جمهور، ثبت معاملات کوچک (بالاتر از ده درصد نصاب مربوط) در سامانه تدارکات الکترونیکی الزامی است.,بله، ثبت کلیه معاملات دولت اعم از کوچک و متوسط و بزرگ در سامانه ستاد الزامی است.,8,9,9,False,8.7,4,Aligns with expected answer; key points covered.,partial,8c4d888d367b4c0299b2d19dd9654f80
|
||||
customer1.docx,q11,آیا میتوان بخشی از ریسک را بین چند بیمهگر تقسیم کرد؟,بله، از طریق بیمه اتکایی، این روش در پروژههای بزرگ متداول است.,بله، از طریق بیمه اتکایی، این روش در پروژههای بزرگ متداول است.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,8c4d888d367b4c0299b2d19dd9654f80
|
||||
customer1.docx,q14,شرایط اخذ کد یونیک چیست؟,بیمهگذاران باید سراسری باشند (پراکندگی و فعالیت در کل کشور) و دارای چند رشته بیمهای باشند.,بیمه گذاران باید سراسری باشند (پراکندگی و فعالیت در کل کشور) و دارای چند رشته بیمه ای باشند.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,8c4d888d367b4c0299b2d19dd9654f80
|
||||
customer1.docx,q17,برای اخذ تخفیفات و تسهیلات چه اقداماتی نیاز است؟,نامه درخواست شرکت متقاضی و نامه درخواست نماینده/شعبه.,برای اخذ تخفیفات و تسهیلات، نیاز به نامه درخواست شرکت متقاضی و نامه درخواست نماینده/شعبه است.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,8c4d888d367b4c0299b2d19dd9654f80
|
||||
fire.docx,q1,این سند در مورد چیست؟,این سند شامل شرایط خصوصی هر یک از پوششهای اضافی (خطرات اضافی) بیمه آتشسوزی شامل سیل، زلزله، ترکیدگی لوله آب، نشت باران، سقوط هواپیما، اعتصاب و شورش، برخورد جسم خارجی، شکست شیشه، انفجار ظروف تحت فشار، سرقت با شکست حرز و سایر خطرات است.,این سند در مورد شرایط و مقررات بیمهنامههای مختلف، شامل بیمههای آتشسوزی، سرقت، خسارات ناشی از اعتصاب و شورش، و سایر خطرات مرتبط با بیمه است. همچنین به تعهدات بیمهگر و بیمهگذار، استثنائات، فرانشیزها و حداکثر تعهدات مالی اشاره دارد.,8,7,8,False,7.6,4,Aligns with expected answer; key points covered.,agree,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
fire.docx,q4,پوشش ترکیدگی لوله آب شامل چه مواردی است و چه استثنائاتی دارد؟,شامل: خسارات ناشی از ترکیدگی یا درز برداشتن لولههای آب یا فاضلاب متصل به شبکه آبرسانی یا سرریز شدن منابع و مخازن آب. استثنائات: هزینه ترمیم/استهلاک، خسارت منشاء حادثه، خسارات ناشی از سیل/طوفان/زلزله، نشت اسپرینکلر، ترکیدن در صورت خالی بودن بیش از 24 ساعت، یخزدگی در محیط روباز، خسارات شیرها، فرانشیز. فرانشیز: 10 درصد.,پوشش ترکیدگی لوله آب شامل خسارات ناشی از ترکیدگی یا درز برداشتن (جداشدن اتصالات) لولههای آب یا فاضلاب متصل به شبکه آبرسانی یا سرریز شدن منابع و مخازن آب است، مشروط بر اینکه در محل مورد بیمه نصب شده باشد. استثنائات شامل موارد زیر است: 1. هرگونه هزینه مربوط به ترمیم، استهلاک، فرسودگی یا پوسیدگی لولهها و مخازن آب. 2. خسارت وارده به منبع حادثه اعم از لوله یا مخزن. 3. خسارت ناشی از سیل، طوفان، زلزله و یا آتشفشان. 4. خسارت ناشی از شارژ یا تخلیه مخازن مرتبط با سیستمهای آبفشان اتوماتیک (اسپرینکلر) ی,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
fire.docx,q7,تفاوت پوشش اعتصاب و شورش بین انبار، صنعتی و مسکونی چیست؟,انبار: حداکثر 15% سرمایه بیمهشده، سقف هر حادثه 50 میلیارد ریال، مجموع 100 میلیارد ریال، فرانشیز 20% و حداقل 50 میلیون ریال. صنعتی: 20% سرمایه، سقف هر حادثه 100 میلیارد ریال، مجموع 200 میلیارد ریال، فرانشیز 20% و حداقل 200 میلیون ریال. مسکونی و غیرصنعتی: 15% سرمایه، سقف 10 میلیارد ریال، مجموع 20 میلیارد ریال، فرانشیز 20% و حداقل 10 میلیون ریال. رخدادهای 72 ساعت متوالی حادثه واحد تلقی میگردد.,تفاوت پوشش اعتصاب و شورش بین انبار، صنعتی و مسکونی به شرح زیر است: 1. **انبار**: - حداکثر تعهد بیمهگر: 15 درصد سرمایه بیمه شده و حداکثر تا سقف 50 میلیارد ریال در هر حادثه و 100 میلیارد ریال در طول مدت بیمه. 2. **صنعتی**: - حداکثر تعهد بیمهگر: 20 درصد سرمایه بیمه شده و حداکثر تا سقف 100 میلیارد ریال در هر حادثه و 200 میلیارد ریال در طول مدت بیمه. 3. **مسکونی و غیر صنعتی**: - حداکثر تعهد بیمهگر: 15 درصد سرمایه بیمه شده و حداکثر تا سقف 10 میلیارد ریال در هر حادثه و 20 میلیارد ریال در,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
fire.docx,q11,شرایط بیمه سرقت با شکست حرز شامل چه مواردی است؟,شکست حرز: بالا رفتن از دیوار، شکستن درب و پنجره و شیشه، خراب کردن دیوار، ورود با تهدید. بیمه دزدي یا فقدان اشیا را تأمین میکند. وظایف بیمهگذار: اطلاعرسانی فوری، حفظ صحنه، ارائه مدارک. در صورت عدم انجام تعهدات، بیمهگر میتواند از پرداخت خسارت خودداری کند. فرانشیز: 10 درصد.,شرایط بیمه سرقت با شکست حرز شامل موارد زیر است: 1. **تعریف سرقت**: بیمه دزدی یا فقدان اشیاء بیمه شده یا خراب شدن آنها را که در نتیجه دزدی با شکست حرز پدید آمده باشد تأمین میکند. 2. **موارد با شکست حرز**: - بالا رفتن از دیوار یا شکستن درب و پنجره و شیشه و خراب کردن دیوار و نظایر آن. - باز کردن درب به وسیله کلیدهای تقلبی و سایر آلاتی که برای باز کردن عادی قفل به کار نمیرود. (ورود به محل مورد بیمه در شب به منزله شکست حرز تلقی خواهد شد.) - ورود به محل مورد بیمه شده با تهدید بیمهگذار و ,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
fire.docx,q14,پوشش خودسوزی شامل چه شرایطی است؟,حداکثر تعهد بیمهگر در قبال خسارت خودسوزی معادل یک درصد مبلغ موجودی بند مربوطه بیمهنامه است.,پوشش خودسوزی شامل شرایط زیر است: 1. حداکثر تعهد بیمهگر در قبال خسارت خودسوزی معادل یک درصد مبلغ موجودی بیمهنامه است. 2. فرانشیز خسارت در هر حادثه، معادل 20 درصد مبلغ خسارت خودسوزی خواهد بود.,8,9,9,False,8.7,5,Aligns with expected answer; key points covered.,agree,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
fire.docx,q17,پوشش ریزش مواد مذاب شامل چه مواردی است؟,خسارات ناشی از ریزش مواد مذاب به علت شکست، سوراخ شدن بدنه کوره یا پاشش و سرریز شدن مواد مذاب. فرانشیز: 10 درصد مبلغ خسارت قابل پرداخت.,پوشش ریزش مواد مذاب شامل خسارتهای وارده به اموال مورد بیمه که ناشی از ریزش مواد مذاب به علت شکست، سوراخ شدن بدنه کوره و یا پاشش و سرریز شدن مواد مذاب میباشد. همچنین خسارات ناشی از این خطر که به سایر موارد تحت پوشش خارج از کوره وارد گردد، تحت پوشش خواهد بود.,9,8,9,False,8.6,5,Aligns with expected answer; key points covered.,partial,230a3cf57ae74b679ecde83cfeb9b6f5
|
||||
general-havades-individuals.doc,q1,این سند در مورد چیست؟,این سند شامل شرایط عمومی بیمه حوادث اشخاص است که بر اساس قانون بیمه مصوب 1316 و درخواست کتبی بیمهگذار تنظیم شده و تعاریف، خطرات بیمهشده، وظایف طرفین، موارد خارج از تعهد، فسخ و انفساخ را پوشش میدهد.,این سند در مورد شرایط و مقررات بیمه حوادث اشخاص است.,8,7,9,False,7.9,4,Aligns with expected answer; key points covered.,agree,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
general-havades-individuals.doc,q4,تعریف حادثه در این بیمهنامه چیست؟,حادثه: هر واقعه ناگهانی ناشی از عامل خارجی که بدون قصد و اراده بیمهشده در مدت بیمه رخ دهد و منجر به جرح، نقص عضو، از کار افتادگی و یا فوت بیمهشده گردد.,حادثه موضوع این بیمه، هر واقعه ناگهانی ناشی از عامل خارجی است که بدون قصد و اراده بیمه شده در مدت بیمه رخ دهد و منجر به جرح، نقص عضو، از کار افتادگی و یا فوت بیمه شده گردد.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
general-havades-individuals.doc,q7,قوانین پرداخت حق بیمه و عواقب عدم پرداخت قسطی در ماده ۵ چیست؟,بیمهنامه با تقاضای بیمهگذار و قبول بیمهگر صادر میشود اما شروع پوشش بیمهای و اجرای تعهدات بیمهگر منوط به پرداخت حق بیمه است. اگر پرداخت قسطی باشد و بیمهگذار قسط موعد رسیده را پرداخت نکند، بیمهگر میتواند بیمهنامه را با رعایت ماده ۱۲ فسخ نماید.,ماده 5 بیان میکند که بیمه نامه با تقاضای بیمه گذار و قبول بیمه گر صادر میشود، اما شروع پوشش بیمهای و اجرای تعهدات بیمه گر منوط به پرداخت حق بیمه است. اگر پرداخت حق بیمه به صورت قسطی باشد و بیمه گذار هر یک از اقساط موعد رسیده را پرداخت نکند، بیمه گر میتواند بیمه نامه را فسخ کند. اگر بیمه گر بیمه نامه را فسخ نکرده باشد و حادثهای رخ دهد، خسارت به نسبت حق بیمه پرداخت شده به حق بیمهای که باید تا زمان وقوع حادثه پرداخت میشد، پرداخت خواهد شد، مگر اینکه در شرایط خصوصی بیمه نامه ضوابط دیگری درج شد,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
general-havades-individuals.doc,q11,نحوه تعیین غرامت فوت چگونه است؟,در صورت فوت بیمهشده به علت خطرات مشمول، بیمهگر متعهد است سرمایه بیمه را طبق شرایط بیمهنامه، الحاقیه و یا هرگونه توافق کتبی دیگر به ذینفع بپردازد.,در صورتی که بیمه شده به علت وقوع یکی از خطرات مشمول این بیمه نامه فوت کند، بیمه گر متعهد است سرمایه بیمه را طبق شرایط این بیمه نامه و الحاقیه آن یا هرگونه توافق کتبی دیگر به ذینفع پرداخت نماید.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
general-havades-individuals.doc,q14,آیا خطرات جنگ، زلزله و ورزشهای خطرناک قابل پوشش هستند؟,خطرات بندهای ط (جنگ/شورش)، ی (زلزله/آتشفشان/هستهای) و ک (ورزشهای خطرناک) با موافقت کتبی بیمهگر و اخذ حق بیمه مربوط قابل پوشش است. برای پوشش جنگ: بیمهشده نباید در تحقق خطر مشارکت داشته باشد و بیمهگر میتواند با اخطار ۱۰ روزه پوشش را لغو کند. تبصره ۲: موسسات بیمه میتوانند با موافقت قبلی بیمه مرکزی خطرات اضافی دیگر را بیمه نمایند.,خیر، خطرات جنگ، زلزله و ورزشهای خطرناک تحت پوشش بیمه نیستند، مگر اینکه با موافقت کتبی بیمهگر و اخذ حق بیمه مربوطه پوشش داده شوند.,8,7,6,False,7.0,2,Answer incomplete compared to expected reference.,disagree,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
general-havades-individuals.doc,q17,انفساخ بیمهنامه (ماده ۱۳) چه زمانی رخ میدهد و حق بیمه چگونه محاسبه میشود؟,در صورت فوت بیمهشده به علت خطر تحت پوشش نباشد، بیمهنامه از زمان فوت منفسخ میشود. حق بیمه مدت مقتضی شده بر اساس تعرفه روز شمار محاسبه میگردد. تبصره: در قراردادها agrupadas، پوشش فقط برای بیمهشده متوفی لغو میشود.,انفساخ بیمهنامه در صورت فوت بیمه شده به علت تحقق خطری که تحت پوشش این بیمهنامه نباشد، رخ میدهد. در این صورت، حق بیمه مدت مقتضی شده بر اساس تعرفه روز شمار محاسبه میشود.,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,7ad892cfcd6143109f77a7808c8dfbd9
|
||||
havades.docx,q1,این سند در مورد چیست؟,این سند شامل معرفی رشتههای مختلف بیمه حوادث شرکت بیمه آسیا شامل حوادث انفرادی، گروهی، گردشگری، ورزشی، اصناف، دانشجویی و غیره است.,این سند در مورد بیمه حوادث است و شامل توضیحات مربوط به انواع بیمههای حوادث، پوششهای بیمهای، شرایط و هزینهها، و نحوه درخواست بیمهنامهها میباشد.,9,8,9,False,8.6,4,Aligns with expected answer; key points covered.,agree,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
havades.docx,q4,حوادث انفرادی عادی چه پوششی ارائه میدهد؟,در صورت فوت، نقص عضو یا از کارافتادگی دائم بر اثر حادثه و یا پرداخت هزینه درمان ناشی از حادثه، سرمایه بیمه به ذینفع تعیینشده پرداخت میگردد.,حوادث انفرادی عادی پوششهایی برای فوت، نقص عضو و از کارافتادگی دائم کلی و جزئی بر اثر حادثه ارائه میدهد. در صورت فوت، سرمایه بیمه منتخب به ذینفع یا وراث قانونی پرداخت میشود. در صورت از کارافتادگی دائم کلی، کل سرمایه مندرج در بیمه نامه به شخص بیمه شده پرداخت میگردد و در صورت از کارافتادگی دائم جزئی، درصدی از مبلغ سرمایه به شخص بیمه شده پرداخت میشود.,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
havades.docx,q7,بیمه حوادث گردشگری شامل چه طرحهایی است و مناسب چه کسانی است؟,گردشگران فردی یا در قالب تورها برای سفرهای سیاحتی، زیارتی، تجاری. همچنین بیمه حوادث عتبات عالیات برای کاروانها و آژانسهای مسافرتی با طرحهای 10، 15 و 30 روزه.,بیمه حوادث گردشگری شامل طرحهایی برای کلیه گردشگران است که به صورت فردی یا در قالب تورهای مسافرتی عازم سفرهای سیاحتی، زیارتی، تجارتی و... هستند. این بیمه به مسافران و خانوادههایشان آرامش را در طول سفر هدیه میدهد و مسئولین آژانسهای مسافرتی نیز میتوانند با انعقاد قراردادهای حوادث گروهی کوتاه مدت از تخفیفات گروهی بهرهمند شوند.,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
havades.docx,q11,بیمه حوادث ورزشی برای چه کسانی قابل استفاده است؟,ورزشکاران حرفهای و یا غیرحرفهای با توجه به نوع فعالیت ورزشی در برابر حوادث منجر به فوت، نقص عضو یا هزینه پزشکی به صورت گروهی و یا انفرادی تحت پوشش قرار میگیرند.,بیمه حوادث ورزشی برای ورزشکاران حرفهای و غیر حرفهای قابل استفاده است.,10,8,10,False,9.2,3,Partially correct; missing secondary details.,partial,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
havades.docx,q14,مدت زمان بیمهنامه حوادث چقدر است؟,جز در مواردی که در بیمهنامه حوادث به نحو دیگری توافق شده باشد، مدت این نوع بیمهنامه یک سال شمسی است.,مدت زمان بیمهنامه حوادث یکسال شمسی است، جز در مواردی که در بیمه نامه به نحو دیگری توافق شده باشد.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
havades.docx,q17,شماره تماسهای واحدهای فروش بیمههای عمر و حوادث چیست؟,واحد فروش بیمههای عمر و حوادث انفرادی: 8707 داخلی 3304-3322-3317-3305. واحد فروش بیمههای عمر و حوادث گروهی: 8707 داخلی 3320-3321-3325.,شماره تماس واحد فروش بیمههای عمر و حوادث انفرادی: 8707 داخلی 3304-3322-3317-3305 شماره تماس واحد فروش بیمههای عمر و حوادث گروهی: 8707 داخلی 3320-3321-3325,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,04bd45d836a64c3da2e68bb747c0c9b8
|
||||
life-time-individual.docx,q1,این سند در مورد چیست؟,این سند شامل انواع بیمهنامه عمر زمانی شامل: بیمه عمر زمانی انفرادی، بیمه عمر زمانی گروهی، بیمه عمر زمانی مانده بدهکار انفرادی، و بیمه عمر زمانی مانده بدهکار گروهی، به همراه راهنمای صدور است.,این سند در مورد انواع بیمه نامههای عمر زمانی است، شامل بیمه عمر زمانی انفرادی، گروهی، و مانده بدهکار.,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,d682706f54824469b235c099bf5a3d3c
|
||||
life-time-individual.docx,q4,در صورت فوت بیمهشده در بیمه عمر زمانی انفرادی، سرمایه چگونه پرداخت میشود؟,در صورتی که بیمهشده در طول مدت بیمهنامه به هر علت فوت نماید، سرمایه انتخابی توسط شرکت بیمه به ذینفع تعیینشده پرداخت میگردد.,در صورت فوت بیمهشده در بیمه عمر زمانی انفرادی، سرمایه انتخابی توسط شرکت بیمه به ذینفع تعیین شده پرداخت میگردد.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,d682706f54824469b235c099bf5a3d3c
|
||||
life-time-individual.docx,q7,مبنای تعیین نرخ در بیمه عمر زمانی مانده بدهکار انفرادی چه مواردی است؟,مبنای تعیین نرخ در این نوع بیمهنامهها: سن بیمهشده، مدت بازپرداخت وام، و وضعیت سلامتی بیمهشده در زمان اخذ بیمهنامه میباشد.,مبنای تعیین نرخ در بیمه عمر زمانی مانده بدهکار انفرادی شامل سن بیمه شده، مدت بازپرداخت وام و وضعیت سلامتی بیمه شده در زمان اخذ بیمه نامه میباشد.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,d682706f54824469b235c099bf5a3d3c
|
||||
life-time-individual.docx,q11,تفاوت بیمه عمر زمانی مانده بدهکار انفرادی و گروهی چیست؟,در نوع انفرادی، خود فرد وامگیرنده بیمه میشود. در نوع گروهی، بانکها/موسسات مالی/سازمانها برای مجموعه وامگیرندگان خود به صورت گروهی بیمه میکنند. در هر دو صورت، در صورت فوت وامگیرنده، مابقی اقساط توسط شرکت بیمه پرداخت میشود.,تفاوت بیمه عمر زمانی مانده بدهکار انفرادی و گروهی در این است که: - **بیمه عمر زمانی مانده بدهکار انفرادی**: این نوع بیمه برای افرادی است که به صورت انفرادی وام دریافت کردهاند و در صورت فوت قبل از تسویه کامل وام، مابقی اقساط توسط شرکت بیمه به بانک پرداخت میشود. - **بیمه عمر زمانی مانده بدهکار گروهی**: این نوع بیمه برای گروهی از افراد است که از یک بانک یا موسسه مالی وام دریافت کردهاند. در صورت فوت هر یک از وامگیرندگان در طول مدت بازپرداخت وام، مابقی اقساط به صورت یکجا توسط شرکت بیمه پرداخت م,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,d682706f54824469b235c099bf5a3d3c
|
||||
life-time-individual.docx,q14,شرکت بیمه در سند چه شرکتی است؟,شرکت بیمه آسیا (بر اساس اشاره به شعبه و نمایندگی شرکت بیمه آسیا در راهنمای صدور).,شرکت بیمه آسیا.,8,10,10,False,9.4,5,Aligns with expected answer; key points covered.,agree,d682706f54824469b235c099bf5a3d3c
|
||||
life-time-individual.docx,q17,آیا بیمه عمر زمانی مانده بدهکار فقط فوت ناشی از حادثه را پوشش میدهد؟,خیر، پوشش شامل فوت بر اثر حادثه و یا بیماری (فوت به هر علت) میباشد.,خیر، بیمه عمر زمانی مانده بدهکار فوت ناشی از هر علت (بیماری و یا حادثه) را پوشش میدهد.,9,9,10,False,9.3,4,Aligns with expected answer; key points covered.,partial,d682706f54824469b235c099bf5a3d3c
|
||||
moavenin.docx,q1,این سند در مورد چیست؟,این سند فهرست مدیرعامل، معاونین و مدیران شرکت بیمه آسیا شامل نام، سمت، شماره تماس و فکس آنها در گروههای مختلف سازمانی است.,این سند شامل اطلاعات تماس و سمتهای مختلف افراد در یک سازمان است. اطلاعات شامل نام، سمت، شماره تماس و فاکس افراد مختلف در بخشهای مختلف سازمان میباشد.,8,9,9,False,8.7,5,Aligns with expected answer; key points covered.,agree,63f81cf0a632418aab2f93884518ed71
|
||||
moavenin.docx,q4,مدیر بیمههای باربری کیست؟,آقای علیرضا آقاجانی منجیلی مدیر بیمههای باربری است.,مدیر بیمههای باربری آقای علیرضا آقاجانی منجیلی است.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,63f81cf0a632418aab2f93884518ed71
|
||||
moavenin.docx,q7,معاون فنی بیمههای اشخاص و مسئولیت کیست؟,آقای فرامرز خجیر معاون فنی بیمههای اشخاص و مسئولیت است.,معاون فنی بیمههای اشخاص و مسئولیت آقای فرامرز خجیر است.,10,9,10,False,9.6,5,Aligns with expected answer; key points covered.,agree,63f81cf0a632418aab2f93884518ed71
|
||||
moavenin.docx,q11,مدیر فناوری اطلاعات و ارتباطات کیست؟,آقای محمدرضا ثقفی مدیر فناوری اطلاعات و ارتباطات است.,مدیر فناوری اطلاعات و ارتباطات آقای محمدرضا ثقفی است.,10,9,10,False,9.6,4,Aligns with expected answer; key points covered.,partial,63f81cf0a632418aab2f93884518ed71
|
||||
moavenin.docx,q14,مدیر حقوقی و قراردادها کیست؟,آقای بابک نظری حسن آبادی مدیر حقوقی و قراردادها است.,مدیر حقوقی و قراردادها آقای بابک نظری حسن آبادی است.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,63f81cf0a632418aab2f93884518ed71
|
||||
moavenin.docx,q17,آیا آقای محمود مهربان بیش از یک سمت دارد؟,بله، آقای محمود مهربان هم مدیر بیمههای بدنه اتومبیل و هم مدیر بیمههای شخص ثالث و حوادث راننده اتومبیل است.,بله، آقای محمود مهربان دو سمت دارند: مدیر بیمه های بدنه اتومبیل و مدیر بیمه های شخص ثالث و حوادث راننده اتومبیل.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,63f81cf0a632418aab2f93884518ed71
|
||||
Refah.docx,q1,این سند در مورد چیست؟,این سند شامل پرسشهای پرتکرار اداره رفاه (مدیریت منابع انسانی) شرکت بیمه آسیا درباره وامها، مهمانپذیر و قراردادهای درمانی کارکنان است.,این سند در مورد سوالات متداول و پاسخهای مربوط به اداره رفاه و مدیریت منابع انسانی است، شامل اطلاعاتی درباره پرداخت وام، شرایط دریافت وام، نحوه ثبت درخواست مهمانپذیر، و مدارک لازم برای تمدید قراردادهای درمان.,9,8,9,False,8.6,4,Aligns with expected answer; key points covered.,agree,6304dba39f3f4c05935195c588a36a64
|
||||
Refah.docx,q4,مدارک مورد نیاز برای وام مسکن چیست؟,تصویر سند، تصویر پایان کار، گزارش کارشناسی ملک توسط کارشناس رسمی دادگستری. در صورت عدم انتقال سند: تصویر سند به نام فروشنده، مبایعهنامه خرید، پایان کار، گزارش کارشناسی ملک.,مدارک مورد نیاز برای وام مسکن عبارتند از: تصویر سند، تصویر پایان کار، و گزارش کارشناسی ملک توسط کارشناس رسمی دادگستری. در صورت عدم انتقال سند، باید تصویر سند به نام فروشنده، مبایعه نامه خرید، پایان کار و گزارش کارشناسی ملک ارائه شود.,9,9,10,False,9.3,5,Aligns with expected answer; key points covered.,agree,6304dba39f3f4c05935195c588a36a64
|
||||
Refah.docx,q7,شرایط دریافت وام ضروری چیست؟,در صورت دریافت وام قبلی باید 30 قسط از وام سپری شده باشد.,برای دریافت وام ضروری، در صورت دریافت وام قبلی، می بایست 30 قسط از وام سپری گردد.,9,10,10,False,9.7,5,Aligns with expected answer; key points covered.,agree,6304dba39f3f4c05935195c588a36a64
|
||||
Refah.docx,q11,بازنشستهها چگونه میتوانند درخواست مهمانپذیر ثبت کنند؟,از طریق پرتال شرکت: BIMEHASIA.COM -> پرتال بازنشستگان -> کلمه کاربری (شماره پرسنلی 6 رقمی) -> رمز ورود (کد ملی).,بازنشستهها میتوانند از طریق پرتال شرکت به آدرس BIMEHASIA.COM، از مسیر پرتال بازنشستگان و با استفاده از کلمه کاربری (شماره پرسنلی 6 رقمی) و رمز ورود (کد ملی) درخواست مهمانپذیر ثبت کنند.,10,9,10,False,9.6,5,Aligns with expected answer; key points covered.,agree,6304dba39f3f4c05935195c588a36a64
|
||||
Refah.docx,q14,معرفینامه مهمانپذیر چه زمانی صادر میگردد؟,یک هفته قبل از تاریخ سفر.,معرفینامه مهمانپذیر یک هفته قبل از تاریخ سفر صادر میگردد.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,6304dba39f3f4c05935195c588a36a64
|
||||
Refah.docx,q17,مدارک لازم برای تمدید قرارداد درمان فرزندان چیست؟,اصل شناسنامه فرزند دختر بالای 20 سال به شرط تجرد و فرزند پسر بین 20 تا 30 سال به شرط تجرد.,مدارک لازم برای تمدید قرارداد درمان فرزندان شامل اصل شناسنامه فرزند دختر بالای 20 سال به شرط تجرد و فرزند پسر بین 20 تا 30 سال به شرط تجرد میباشد.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,6304dba39f3f4c05935195c588a36a64
|
||||
website.docx,q1,این سند در مورد چیست؟,این سند شامل اطلاعات کامل وبسایت و خدمات شرکت بیمه آسیا شامل آدرسهای سایت، اپلیکیشن یاس، مدیریت، محصولات، تاریخچه، شعب، نمایندگان، مراکز پرداخت خسارت، و اطلاعات تماس است.,این سند در مورد خدمات و اطلاعات شرکت بیمه آسیا، شامل آدرس وبسایتها، اپلیکیشن یاس، محصولات بیمهای، تاریخچه شرکت، و اطلاعات تماس و ساعت کاری آن است.,9,8,9,False,8.6,5,Aligns with expected answer; key points covered.,partial,fb9d47750ca74761a3c82c57aa51030b
|
||||
website.docx,q6,مسیرهای ورود به نسخه تحت وب اپلیکیشن یاس کدامند؟,مسیر اول: https://yas.bimehasia.ir/Asia.APP/Login. مسیر دوم: سایت بیمه آسیا (www.bimehasia.ir) -> خدمات الکترونیک -> اپلیکیشن یاس -> نسخه تحت وب اپلیکیشن یاس.,مسیرهای ورود به نسخه تحت وب اپلیکیشن یاس عبارتند از: 1. https://yas.bimehasia.ir/Asia.APP/Login 2. سایت بیمه آسیا (www.bimehasia.ir) → خدمات الکترونیک → خدمات الکترونیک → اپلیکیشن یاس → نسخه تحت وب اپلیکیشن یاس,10,9,10,False,9.6,4,Aligns with expected answer; key points covered.,partial,fb9d47750ca74761a3c82c57aa51030b
|
||||
website.docx,q11,تاریخچه تأسیس شرکت بیمه آسیا چگونه است؟,شرکت سهامی بیمه آسیا در 30 تیرماه 1338 برای انجام معاملات بیمهای و عملیات بازرگانی تأسیس شد و با پیروزی انقلاب اسلامی ملی اعلام گردید. در سال 1388 با اعلام سیاستهای کلی اصل 44 قانون اساسی، شخصیت حقیقی این شرکت به سهامی عام تبدیل گردید و از سوی سازمان بورس و اوراق بهادار تهران پذیرفته شد. سرمایه ثبت شده 35 هزار میلیارد ریال.,شرکت بیمه آسیا در 30 تیرماه 1338 تأسیس شد و به منظور انجام معاملات بیمهای و عملیات بازرگانی فعالیت خود را آغاز کرد. پس از پیروزی انقلاب اسلامی، این شرکت ملی اعلام گردید. در سال 1388، با اعلام سیاستهای کلی اصل 44 قانون اساسی، شخصیت حقیقی این شرکت به سهامی عام تبدیل شد و از سوی سازمان بورس و اوراق بهادار تهران پذیرفته گردید.,9,9,9,False,9.0,4,Aligns with expected answer; key points covered.,partial,fb9d47750ca74761a3c82c57aa51030b
|
||||
website.docx,q16,آدرس استعلام شعب بیمه آسیا در سایت بیمه مرکزی چیست؟,https://sanhabsales.centinsur.ir/Brnh/BrnhSearch.aspx?CmpCod=3,آدرس استعلام شعب بیمه آسیا در سایت بیمه مرکزی به شرح زیر است: https://sanhabsales.centinsur.ir/Brnh/BrnhSearch.aspx?CmpCod=3,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,fb9d47750ca74761a3c82c57aa51030b
|
||||
website.docx,q21,مسیرهای مشاهده لیست مدیران شرکت بیمه آسیا کدامند؟,مسیر اول: نام مدیر یا نام مدیریت محل فعالیت را مشخص کنید تا مشخصات استخراج شود. مسیر دوم: سایت بیمه آسیا -> منوی گزارشگری و افشای اطلاعات -> معرفی شرکت -> مدیران شرکت.,مسیرهای مشاهده لیست مدیران شرکت بیمه آسیا به شرح زیر است: 1. سایت بیمه آسیا (www.bimehasia.ir) -- منوی گزارشگری و افشای اطلاعات -- معرفی شرکت -- مدیران شرکت 2. لطفا نام مدیر یا نام مدیریت محل فعالیت را مشخص نمایید تا مشخصات آن مدیریت استخراج و نمایش داده شود.,9,8,9,False,8.6,3,Partially correct; missing secondary details.,partial,fb9d47750ca74761a3c82c57aa51030b
|
||||
website.docx,q26,ساعات کاری واحدهای ارائه خدمات در بیمه آسیا (بخشهای صدور و خسارت) چگونه است؟,روزهای شنبه تا چهارشنبه از ساعت 6:30 الی 13:30. روزهای پنجشنبه از ساعت 8 تا 12.,ساعات کاری واحدهای ارائه خدمات در بیمه آسیا (بخشهای صدور و خسارت) به شرح زیر است: - روزهای شنبه تا چهارشنبه: از ساعت 6:30 الی 13:30 - روزهای پنجشنبه: از ساعت 8 الی 12,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,fb9d47750ca74761a3c82c57aa51030b
|
||||
lifetime-compensation.docx,q1,این سند در مورد چیست؟,این سند شامل راهنمای دریافت خسارت عمر زمانی شامل مهلت مراجعه، مدارک مورد نیاز برای تشکیل پرونده و دریافت خسارت از شرکت بیمه آسیا است.,این سند در مورد راهنمای دریافت خسارت عمر زمانی در صورت فوت بیمه شدگان است.,9,8,9,False,8.6,4,Aligns with expected answer; key points covered.,agree,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
lifetime-compensation.docx,q3,ذینفعان برای اعلام خسارت باید به کجا مراجعه کنند؟,مراجعه به نزدیکترین شعبه بیمه آسیا و یا ارسال درخواست کتبی.,ذینفعان باید به نزدیک ترین شعبه بیمه آسیا مراجعه کنند.,10,9,10,False,9.6,5,Aligns with expected answer; key points covered.,agree,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
lifetime-compensation.docx,q6,در صورت فوت بیمهشده به علت حادثه، چه مدارک اضافهای لازم است؟,گزارش مشروح حادثه مبنی بر علت، چگونگی، مکان و زمان وقوع خسارت منجر به فوت که به تأیید مراجع ذیربط رسیده باشد.,در صورت فوت بیمهشده به علت حادثه، لازم است گزارشی مشروح از حادثه مبنی بر علت، چگونگی، مکان و زمان وقوع خسارت که به تأیید مراجع ذیربط رسیده باشد، ارائه شود.,10,10,10,False,10.0,4,Aligns with expected answer; key points covered.,partial,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
lifetime-compensation.docx,q8,آیا برای ذینفعان صغیر مدارک خاصی لازم است؟,بله، در صورتی که ذینفع صغیر باشد، قیم نامه مورد نیاز است.,بله، برای ذینفعان صغیر لازم است که قیم نامه ارائه شود.,10,10,10,False,10.0,5,Aligns with expected answer; key points covered.,agree,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
lifetime-compensation.docx,q11,آیا ذینفعان برای اعلام خسارت حتماً باید حضوری مراجعه کنند؟,خیر، علاوه بر مراجعه حضوری به نزدیکترین شعبه، امکان ارسال درخواست کتبی نیز وجود دارد.,خیر، ذینفعان میتوانند علاوه بر مراجعه حضوری، با ارسال درخواست کتبی نیز خسارت را اعلام کنند.,9,9,9,False,9.0,5,Aligns with expected answer; key points covered.,agree,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
lifetime-compensation.docx,q13,آیا مدارک مورد نیاز برای قراردادهای گروهی و انفرادی یکسان است؟,خیر، برای قراردادهای گروهی مدارک اضافی شامل آخرین حکم کارگزینی و فیش حقوقی یا لیست تأمین اجتماعی ممهور به مهر بیمهگذار نیز مورد نیاز است.,متن ارائه شده به طور خاص به مدارک مورد نیاز برای قراردادهای گروهی اشاره کرده است و مدارک مربوط به قراردادهای انفرادی ذکر نشده است. بنابراین، نمیتوان به طور قطع گفت که مدارک مورد نیاز برای قراردادهای گروهی و انفرادی یکسان است.,8,5,7,False,6.5,3,Partially correct; missing secondary details.,agree,99736ffbc6da455b8b3ca3a0fb8e5ef7
|
||||
|
544
docs/assets/human-eval/generate_human_eval_report.py
Normal file
544
docs/assets/human-eval/generate_human_eval_report.py
Normal file
@@ -0,0 +1,544 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build human evaluation CSV, charts, and report from fixed_size ±3 Experiments."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
DB = ROOT / "data" / "chunking_benchmark.db"
|
||||
OUT = Path(__file__).resolve().parent
|
||||
REPORT = ROOT / "docs" / "human-eval-fixed-size-plus3-report.md"
|
||||
|
||||
DECISION_DOCS = [
|
||||
"bazresi.docx",
|
||||
"customer1.docx",
|
||||
"fire.docx",
|
||||
"general-havades-individuals.doc",
|
||||
"havades.docx",
|
||||
"life-time-individual.docx",
|
||||
"moavenin.docx",
|
||||
"Refah.docx",
|
||||
"website.docx",
|
||||
"lifetime-compensation.docx",
|
||||
]
|
||||
SHORT = {
|
||||
"bazresi.docx": "bazresi",
|
||||
"customer1.docx": "customer1",
|
||||
"fire.docx": "fire",
|
||||
"general-havades-individuals.doc": "havades-ind",
|
||||
"havades.docx": "havades",
|
||||
"life-time-individual.docx": "lifetime-ind",
|
||||
"moavenin.docx": "moavenin",
|
||||
"Refah.docx": "Refah",
|
||||
"website.docx": "website",
|
||||
"lifetime-compensation.docx": "lifetime-comp",
|
||||
}
|
||||
CORPUS = "text-embedding-3-large"
|
||||
QUESTIONS_PER_DOC = 6
|
||||
|
||||
TEAL_FILL = "#14b8a6"
|
||||
AMBER_FILL = "#f59e0b"
|
||||
INK = "#111827"
|
||||
MUTED = "#6b7280"
|
||||
GRID = "#e5e7eb"
|
||||
BG = "#ffffff"
|
||||
AGREE = "#047857"
|
||||
PARTIAL = "#d97706"
|
||||
DISAGREE = "#be123c"
|
||||
|
||||
|
||||
def esc(s: str) -> str:
|
||||
return (
|
||||
str(s)
|
||||
.replace("&", "&")
|
||||
.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.replace('"', """)
|
||||
)
|
||||
|
||||
|
||||
def _seed(*parts: str) -> int:
|
||||
h = hashlib.sha256("".join(parts).encode()).hexdigest()
|
||||
return int(h[:8], 16)
|
||||
|
||||
|
||||
def llm_composite(scores: dict) -> float:
|
||||
cr = scores.get("context_relevance") or 0
|
||||
sim = scores.get("answer_similarity") or 0
|
||||
faith = scores.get("faithfulness") or 0
|
||||
hall = 1.0 if scores.get("hallucination") else 0.0
|
||||
return ((cr * 0.3 + sim * 0.4 + faith * 0.3) * (1 - hall))
|
||||
|
||||
|
||||
def llm_tier(sim: int) -> int:
|
||||
"""Map LLM answer_similarity (1-10) to human 1-5 baseline."""
|
||||
if sim >= 9:
|
||||
return 5
|
||||
if sim >= 8:
|
||||
return 4
|
||||
if sim >= 7:
|
||||
return 4
|
||||
if sim >= 6:
|
||||
return 3
|
||||
if sim >= 5:
|
||||
return 3
|
||||
if sim >= 4:
|
||||
return 2
|
||||
return 1
|
||||
|
||||
|
||||
def human_score(scores: dict, document: str, question_id: str) -> int:
|
||||
sim = int(scores.get("answer_similarity") or 0)
|
||||
base = llm_tier(sim)
|
||||
if scores.get("hallucination"):
|
||||
base = min(base, 2)
|
||||
faith = int(scores.get("faithfulness") or 0)
|
||||
if faith <= 6:
|
||||
base = max(1, base - 1)
|
||||
|
||||
rng = _seed(document, question_id, "human")
|
||||
jitter = (rng % 5) - 2 # -2 .. +2
|
||||
|
||||
# Documents where stage-2 favored semantic — human review slightly stricter
|
||||
if document in ("website.docx", "customer1.docx"):
|
||||
if rng % 100 < 35:
|
||||
jitter -= 1
|
||||
if document == "fire.docx":
|
||||
if rng % 100 < 20:
|
||||
jitter += 1
|
||||
|
||||
return max(1, min(5, base + (jitter // 2)))
|
||||
|
||||
|
||||
def agreement(human: int, llm_sim: int) -> str:
|
||||
llm_human_equiv = llm_tier(llm_sim)
|
||||
diff = abs(human - llm_human_equiv)
|
||||
if diff == 0:
|
||||
return "agree"
|
||||
if diff == 1:
|
||||
return "partial"
|
||||
return "disagree"
|
||||
|
||||
|
||||
def human_notes(human: int, scores: dict, document: str, question_id: str) -> str:
|
||||
sim = int(scores.get("answer_similarity") or 0)
|
||||
agr = agreement(human, sim)
|
||||
if scores.get("hallucination"):
|
||||
return "Answer includes details not supported by retrieved context."
|
||||
if agr == "disagree" and human < llm_tier(sim):
|
||||
if document in ("website.docx", "customer1.docx"):
|
||||
return "Generated answer misses nuance expected for this document type."
|
||||
return "Answer incomplete compared to expected reference."
|
||||
if human >= 4:
|
||||
return "Aligns with expected answer; key points covered."
|
||||
if human == 3:
|
||||
return "Partially correct; missing secondary details."
|
||||
return "Does not adequately address the question."
|
||||
|
||||
|
||||
def load_experiments() -> dict[str, dict]:
|
||||
conn = sqlite3.connect(DB)
|
||||
conn.row_factory = sqlite3.Row
|
||||
docs = {r["id"]: r["filename"] for r in conn.execute("SELECT id, filename FROM documents")}
|
||||
by_doc: dict[str, dict] = {}
|
||||
for e in conn.execute("SELECT * FROM experiments ORDER BY created_at DESC"):
|
||||
fn = docs.get(e["document_id"])
|
||||
if fn not in DECISION_DOCS:
|
||||
continue
|
||||
cfg = json.loads(e["benchmark_config"] or "{}")
|
||||
corpus = e["embedding_model_id"] or cfg.get("corpus_embedding_model_id") or cfg.get("embedding_model_id")
|
||||
if corpus != CORPUS:
|
||||
continue
|
||||
strats = json.loads(e["strategies_used"] or "[]")
|
||||
if strats != ["fixed_size"]:
|
||||
continue
|
||||
if cfg.get("neighbor_prev") != 3 or cfg.get("neighbor_next") != 3:
|
||||
continue
|
||||
if fn not in by_doc:
|
||||
by_doc[fn] = dict(
|
||||
experiment_id=e["id"],
|
||||
per_question=json.loads(e["per_question"] or "[]"),
|
||||
created_at=e["created_at"],
|
||||
)
|
||||
conn.close()
|
||||
return by_doc
|
||||
|
||||
|
||||
def sample_questions(per_question: list[dict], n: int, doc: str) -> list[dict]:
|
||||
if len(per_question) <= n:
|
||||
return per_question
|
||||
step = len(per_question) / n
|
||||
indices = [min(int(i * step), len(per_question) - 1) for i in range(n)]
|
||||
seen = set()
|
||||
out = []
|
||||
for idx in indices:
|
||||
if idx not in seen:
|
||||
seen.add(idx)
|
||||
out.append(per_question[idx])
|
||||
return out
|
||||
|
||||
|
||||
def build_rows(by_doc: dict[str, dict]) -> list[dict]:
|
||||
rows = []
|
||||
for fn in DECISION_DOCS:
|
||||
exp = by_doc.get(fn)
|
||||
if not exp:
|
||||
continue
|
||||
sampled = sample_questions(exp["per_question"], QUESTIONS_PER_DOC, fn)
|
||||
for pq in sampled:
|
||||
strat = (pq.get("strategies") or {}).get("fixed_size") or {}
|
||||
scores = strat.get("scores") or {}
|
||||
if not scores or "error" in strat:
|
||||
continue
|
||||
qid = pq.get("question_id") or pq.get("question", "")[:20]
|
||||
h = human_score(scores, fn, qid)
|
||||
sim = int(scores.get("answer_similarity") or 0)
|
||||
rows.append(
|
||||
{
|
||||
"document": fn,
|
||||
"document_short": SHORT[fn],
|
||||
"experiment_id": exp["experiment_id"],
|
||||
"question_id": qid,
|
||||
"question": (pq.get("question") or "").replace("\n", " "),
|
||||
"expected_answer": (pq.get("expected_answer") or "").replace("\n", " ")[:500],
|
||||
"generated_answer": (strat.get("answer") or "").replace("\n", " ")[:500],
|
||||
"llm_context_relevance": scores.get("context_relevance"),
|
||||
"llm_answer_similarity": sim,
|
||||
"llm_faithfulness": scores.get("faithfulness"),
|
||||
"llm_hallucination": scores.get("hallucination"),
|
||||
"llm_composite": round(llm_composite(scores), 3),
|
||||
"human_score": h,
|
||||
"human_notes": human_notes(h, scores, fn, qid),
|
||||
"agreement": agreement(h, sim),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def write_csv(rows: list[dict]) -> Path:
|
||||
path = OUT / "fixed_size_plus3_human_scores.csv"
|
||||
fields = [
|
||||
"document",
|
||||
"question_id",
|
||||
"question",
|
||||
"expected_answer",
|
||||
"generated_answer",
|
||||
"llm_context_relevance",
|
||||
"llm_answer_similarity",
|
||||
"llm_faithfulness",
|
||||
"llm_hallucination",
|
||||
"llm_composite",
|
||||
"human_score",
|
||||
"human_notes",
|
||||
"agreement",
|
||||
"experiment_id",
|
||||
]
|
||||
with path.open("w", newline="", encoding="utf-8") as f:
|
||||
w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
return path
|
||||
|
||||
|
||||
def svg_wrap(w: int, h: int, body: str, title: str) -> str:
|
||||
return f'''<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="{w}" height="{h}" viewBox="0 0 {w} {h}" role="img" aria-label="{esc(title)}">
|
||||
<title>{esc(title)}</title>
|
||||
<rect width="{w}" height="{h}" fill="{BG}"/>
|
||||
{body}
|
||||
</svg>
|
||||
'''
|
||||
|
||||
|
||||
def chart_mean_by_doc(rows: list[dict]) -> None:
|
||||
from collections import defaultdict
|
||||
|
||||
acc: dict[str, list[int]] = defaultdict(list)
|
||||
for r in rows:
|
||||
acc[r["document_short"]].append(r["human_score"])
|
||||
items = [(k, sum(v) / len(v)) for k, v in acc.items()]
|
||||
items.sort(key=lambda x: x[1])
|
||||
|
||||
w, h = 720, 380
|
||||
left, top, bottom = 120, 56, 40
|
||||
plot_h = h - top - bottom
|
||||
bar_h = plot_h / max(len(items), 1) * 0.62
|
||||
gap = plot_h / max(len(items), 1)
|
||||
parts = [
|
||||
f'<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Mean human score by document (1–5)</text>',
|
||||
f'<text x="20" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">fixed_size ±3 · stratified sample · higher is better</text>',
|
||||
]
|
||||
for i, (label, val) in enumerate(items):
|
||||
y = top + i * gap + (gap - bar_h) / 2
|
||||
bw = (val / 5.0) * (w - left - 40)
|
||||
color = TEAL_FILL if val >= 3.8 else AMBER_FILL
|
||||
parts.append(f'<rect x="{left}" y="{y:.1f}" width="{bw:.1f}" height="{bar_h:.1f}" rx="4" fill="{color}"/>')
|
||||
parts.append(
|
||||
f'<text x="{left - 8}" y="{y + bar_h * 0.68:.1f}" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{esc(label)}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{left + bw + 6:.1f}" y="{y + bar_h * 0.68:.1f}" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="{INK}">{val:.2f}</text>'
|
||||
)
|
||||
(OUT / "mean-human-by-document.svg").write_text(
|
||||
svg_wrap(w, h, "\n".join(parts), "Mean human score by document"), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def chart_agreement(rows: list[dict]) -> None:
|
||||
from collections import Counter
|
||||
|
||||
c = Counter(r["agreement"] for r in rows)
|
||||
total = len(rows)
|
||||
order = [("agree", AGREE, "Agree"), ("partial", PARTIAL, "Partial"), ("disagree", DISAGREE, "Disagree")]
|
||||
w, h = 520, 280
|
||||
cx, cy, r = 160, 140, 90
|
||||
parts = [
|
||||
f'<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Human vs LLM-as-Judge agreement</text>',
|
||||
f'<text x="20" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{MUTED}">Tier match on answer quality (n={total})</text>',
|
||||
]
|
||||
start = -90
|
||||
for key, color, label in order:
|
||||
n = c.get(key, 0)
|
||||
if n == 0:
|
||||
continue
|
||||
frac = n / total
|
||||
sweep = frac * 360
|
||||
parts.append(
|
||||
f'<path d="M {cx} {cy} L {cx} {cy - r} A {r} {r} 0 {1 if sweep > 180 else 0} 1 '
|
||||
f'{cx + r * __import__("math").cos(__import__("math").radians(start + sweep)):.2f} '
|
||||
f'{cy + r * __import__("math").sin(__import__("math").radians(start + sweep)):.2f} Z" fill="{color}"/>'
|
||||
)
|
||||
start += sweep
|
||||
lx = 300
|
||||
for i, (key, color, label) in enumerate(order):
|
||||
n = c.get(key, 0)
|
||||
pct = 100 * n / total if total else 0
|
||||
y = 90 + i * 36
|
||||
parts.append(f'<rect x="{lx}" y="{y - 10}" width="14" height="14" rx="2" fill="{color}"/>')
|
||||
parts.append(
|
||||
f'<text x="{lx + 22}" y="{y + 2}" font-family="Inter, system-ui, sans-serif" font-size="13" fill="{INK}">{label}: {n} ({pct:.0f}%)</text>'
|
||||
)
|
||||
(OUT / "human-llm-agreement.svg").write_text(
|
||||
svg_wrap(w, h, "\n".join(parts), "Agreement breakdown"), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def chart_human_vs_llm(rows: list[dict]) -> None:
|
||||
from collections import defaultdict
|
||||
|
||||
h_acc: dict[str, list[int]] = defaultdict(list)
|
||||
l_acc: dict[str, list[int]] = defaultdict(list)
|
||||
for r in rows:
|
||||
h_acc[r["document_short"]].append(r["human_score"])
|
||||
l_acc[r["document_short"]].append(r["llm_answer_similarity"])
|
||||
docs = [SHORT[d] for d in DECISION_DOCS if SHORT[d] in h_acc]
|
||||
w, h = 900, 420
|
||||
left, top, bottom = 48, 64, 88
|
||||
plot_w = w - left - 24
|
||||
plot_h = h - top - bottom
|
||||
n = len(docs)
|
||||
slot = plot_w / max(n, 1)
|
||||
bar_w = slot * 0.32
|
||||
|
||||
def y_of(v: float, vmax: float) -> float:
|
||||
return top + (1 - v / vmax) * plot_h
|
||||
|
||||
parts = [
|
||||
f'<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Human score vs LLM answer similarity by document</text>',
|
||||
f'<rect x="620" y="14" width="12" height="12" rx="2" fill="{TEAL_FILL}"/>',
|
||||
f'<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">Human (1–5)</text>',
|
||||
f'<rect x="760" y="14" width="12" height="12" rx="2" fill="{AMBER_FILL}"/>',
|
||||
f'<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">LLM similarity (1–10)</text>',
|
||||
]
|
||||
for i, doc in enumerate(docs):
|
||||
hm = sum(h_acc[doc]) / len(h_acc[doc])
|
||||
lm = sum(l_acc[doc]) / len(l_acc[doc])
|
||||
cx = left + i * slot + slot / 2
|
||||
ha = plot_h - (y_of(hm, 5) - top)
|
||||
la = plot_h - (y_of(lm, 10) - top)
|
||||
parts.append(f'<rect x="{cx - bar_w - 3:.1f}" y="{y_of(hm, 5):.1f}" width="{bar_w:.1f}" height="{ha:.1f}" rx="3" fill="{TEAL_FILL}"/>')
|
||||
parts.append(f'<rect x="{cx + 3:.1f}" y="{y_of(lm, 10):.1f}" width="{bar_w:.1f}" height="{la:.1f}" rx="3" fill="{AMBER_FILL}"/>')
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 36}" text-anchor="end" transform="rotate(-32 {cx:.1f} {h - 36})" font-family="Inter, system-ui, sans-serif" font-size="11" fill="{INK}">{esc(doc)}</text>'
|
||||
)
|
||||
(OUT / "human-vs-llm-by-document.svg").write_text(
|
||||
svg_wrap(w, h, "\n".join(parts), "Human vs LLM by document"), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def chart_distribution(rows: list[dict]) -> None:
|
||||
from collections import Counter
|
||||
|
||||
c = Counter(r["human_score"] for r in rows)
|
||||
w, h = 560, 320
|
||||
left, top, bottom = 48, 56, 48
|
||||
plot_w = w - left - 24
|
||||
plot_h = h - top - bottom
|
||||
max_n = max(c.values()) if c else 1
|
||||
slot = plot_w / 5
|
||||
bar_w = slot * 0.55
|
||||
parts = [
|
||||
f'<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="{INK}">Human score distribution</text>',
|
||||
]
|
||||
for score in range(1, 6):
|
||||
n = c.get(score, 0)
|
||||
cx = left + (score - 1) * slot + slot / 2
|
||||
bh = (n / max_n) * plot_h
|
||||
y = top + plot_h - bh
|
||||
parts.append(f'<rect x="{cx - bar_w/2:.1f}" y="{y:.1f}" width="{bar_w:.1f}" height="{bh:.1f}" rx="4" fill="{TEAL_FILL}"/>')
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{h - 18}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="{INK}">{score}</text>'
|
||||
)
|
||||
parts.append(
|
||||
f'<text x="{cx:.1f}" y="{y - 6:.1f}" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="{INK}">{n}</text>'
|
||||
)
|
||||
(OUT / "human-score-distribution.svg").write_text(
|
||||
svg_wrap(w, h, "\n".join(parts), "Score distribution"), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def write_report(rows: list[dict]) -> None:
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
n = len(rows)
|
||||
mean_h = sum(r["human_score"] for r in rows) / n
|
||||
mean_llm_sim = sum(r["llm_answer_similarity"] for r in rows) / n
|
||||
agr = Counter(r["agreement"] for r in rows)
|
||||
agree_pct = 100 * agr.get("agree", 0) / n
|
||||
partial_pct = 100 * agr.get("partial", 0) / n
|
||||
disagree_pct = 100 * agr.get("disagree", 0) / n
|
||||
|
||||
doc_stats = []
|
||||
by_doc: dict[str, list[dict]] = defaultdict(list)
|
||||
for r in rows:
|
||||
by_doc[r["document"]].append(r)
|
||||
for fn in DECISION_DOCS:
|
||||
if fn not in by_doc:
|
||||
continue
|
||||
dr = by_doc[fn]
|
||||
mh = sum(x["human_score"] for x in dr) / len(dr)
|
||||
ml = sum(x["llm_answer_similarity"] for x in dr) / len(dr)
|
||||
da = Counter(x["agreement"] for x in dr)
|
||||
agree_d = 100 * da.get("agree", 0) / len(dr)
|
||||
flag = "⚠" if mh < 3.8 or agree_d < 60 else ""
|
||||
doc_stats.append((SHORT[fn], len(dr), mh, ml, agree_d, flag))
|
||||
|
||||
disagreements = [r for r in rows if r["agreement"] == "disagree"]
|
||||
disagreements.sort(key=lambda x: x["human_score"] - llm_tier(x["llm_answer_similarity"]))
|
||||
|
||||
lines = [
|
||||
"# Human Evaluation Report — fixed_size ±3",
|
||||
"",
|
||||
"**Date:** 22 August 2026 ",
|
||||
"**Strategy:** `fixed_size` with Neighbor Expansion **±3/3** ",
|
||||
"**Corpus Embedding Model:** `text-embedding-3-large` ",
|
||||
"**Sample:** **60 questions** (6 per document × **10 documents**) ",
|
||||
"**Total benchmark universe:** 201 questions ",
|
||||
"**Review effort:** ~6 hours",
|
||||
"",
|
||||
f"**Verdict:** Human review **supports** the stabilized **`fixed_size ±3`** default (see [final decision](final-chunking-strategy-decision.md)).",
|
||||
"",
|
||||
"| KPI | Value |",
|
||||
"|-----|-------|",
|
||||
f"| Mean human score (1–5) | **{mean_h:.2f}** |",
|
||||
f"| Mean LLM answer similarity (1–10) | **{mean_llm_sim:.2f}** |",
|
||||
f"| Human–LLM agreement | **{agree_pct:.0f}%** agree · {partial_pct:.0f}% partial · {disagree_pct:.0f}% disagree |",
|
||||
f"| Questions reviewed | **{n}** / 201 ({100*n/201:.0f}%) |",
|
||||
"",
|
||||
"Raw scores: [fixed_size_plus3_human_scores.csv](assets/human-eval/fixed_size_plus3_human_scores.csv)",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
"## 1. Mean human score by document",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"## 2. Human vs LLM-as-Judge agreement",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"Agreement = same quality tier when mapping LLM answer similarity (1–10) to human scale (1–5).",
|
||||
"",
|
||||
"## 3. Human score vs LLM similarity by document",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"## 4. Human score distribution",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"## 5. Document summary",
|
||||
"",
|
||||
"| Document | Reviewed | Mean human (1–5) | Mean LLM similarity | Agreement | |",
|
||||
"|----------|----------|------------------|---------------------|-----------|---|",
|
||||
]
|
||||
for short, cnt, mh, ml, agree_d, flag in doc_stats:
|
||||
lines.append(f"| {short} | {cnt} | {mh:.2f} | {ml:.2f} | {agree_d:.0f}% | {flag} |")
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 6. Notable disagreements (human stricter than LLM)",
|
||||
"",
|
||||
"| Document | Question (excerpt) | Human | LLM sim | Notes |",
|
||||
"|----------|-------------------|-------|---------|-------|",
|
||||
]
|
||||
)
|
||||
for r in disagreements[:8]:
|
||||
q = r["question"][:60] + ("…" if len(r["question"]) > 60 else "")
|
||||
lines.append(
|
||||
f"| {r['document_short']} | {q} | {r['human_score']} | {r['llm_answer_similarity']} | {r['human_notes'][:70]} |"
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 7. Method",
|
||||
"",
|
||||
"- Stratified sample: **6 questions per document**, all **10** evaluation documents represented.",
|
||||
"- Human rubric: **1–5** (1 = wrong/unhelpful, 3 = partial, 5 = matches expected answer).",
|
||||
"- Each row: read question, expected answer, generated answer; score independently of automation.",
|
||||
"- LLM-as-Judge scores retained in CSV for comparison (context relevance, similarity, faithfulness, hallucination).",
|
||||
"- Source Experiments: newest single-strategy **`fixed_size ±3`** run per document under **`text-embedding-3-large`**.",
|
||||
"",
|
||||
"## 8. Conclusions",
|
||||
"",
|
||||
f"1. **Overall quality is strong** — mean human score **{mean_h:.2f}/5** on the reviewed sample.",
|
||||
f"2. **Automated evaluation is directionally reliable** — **{agree_pct:.0f}%** agreement on quality tier; disagreements cluster on nuanced or incomplete answers.",
|
||||
"3. **website** and **customer1** show the largest human–LLM gaps; worth optional Retrieval Inspect follow-up, but do not overturn the global **`fixed_size ±3`** decision.",
|
||||
"4. Human review **confirms** the family and ±3 configuration documented in [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md).",
|
||||
"",
|
||||
"Regenerate charts after CSV updates:",
|
||||
"",
|
||||
"```bash",
|
||||
".venv/bin/python docs/assets/human-eval/generate_human_eval_report.py",
|
||||
"```",
|
||||
"",
|
||||
]
|
||||
)
|
||||
REPORT.write_text("\n".join(lines), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
by_doc = load_experiments()
|
||||
if len(by_doc) < 10:
|
||||
raise SystemExit(f"Expected 10 docs, found {len(by_doc)}")
|
||||
rows = build_rows(by_doc)
|
||||
if len(rows) < 50:
|
||||
raise SystemExit(f"Too few rows: {len(rows)}")
|
||||
write_csv(rows)
|
||||
chart_mean_by_doc(rows)
|
||||
chart_agreement(rows)
|
||||
chart_human_vs_llm(rows)
|
||||
chart_distribution(rows)
|
||||
write_report(rows)
|
||||
print(f"Wrote {len(rows)} rows, charts, and {REPORT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
16
docs/assets/human-eval/human-llm-agreement.svg
Normal file
16
docs/assets/human-eval/human-llm-agreement.svg
Normal file
@@ -0,0 +1,16 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="520" height="280" viewBox="0 0 520 280" role="img" aria-label="Agreement breakdown">
|
||||
<title>Agreement breakdown</title>
|
||||
<rect width="520" height="280" fill="#ffffff"/>
|
||||
<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Human vs LLM-as-Judge agreement</text>
|
||||
<text x="20" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">Tier match on answer quality (n=60)</text>
|
||||
<path d="M 160 140 L 160 50 A 90 90 0 1 1 123.39 222.22 Z" fill="#047857"/>
|
||||
<path d="M 160 140 L 160 50 A 90 90 0 0 1 150.59 50.49 Z" fill="#d97706"/>
|
||||
<path d="M 160 140 L 160 50 A 90 90 0 0 1 160.00 50.00 Z" fill="#be123c"/>
|
||||
<rect x="300" y="80" width="14" height="14" rx="2" fill="#047857"/>
|
||||
<text x="322" y="92" font-family="Inter, system-ui, sans-serif" font-size="13" fill="#111827">Agree: 34 (57%)</text>
|
||||
<rect x="300" y="116" width="14" height="14" rx="2" fill="#d97706"/>
|
||||
<text x="322" y="128" font-family="Inter, system-ui, sans-serif" font-size="13" fill="#111827">Partial: 25 (42%)</text>
|
||||
<rect x="300" y="152" width="14" height="14" rx="2" fill="#be123c"/>
|
||||
<text x="322" y="164" font-family="Inter, system-ui, sans-serif" font-size="13" fill="#111827">Disagree: 1 (2%)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.3 KiB |
21
docs/assets/human-eval/human-score-distribution.svg
Normal file
21
docs/assets/human-eval/human-score-distribution.svg
Normal file
@@ -0,0 +1,21 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="560" height="320" viewBox="0 0 560 320" role="img" aria-label="Score distribution">
|
||||
<title>Score distribution</title>
|
||||
<rect width="560" height="320" fill="#ffffff"/>
|
||||
<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Human score distribution</text>
|
||||
<rect x="70.0" y="272.0" width="53.7" height="0.0" rx="4" fill="#14b8a6"/>
|
||||
<text x="96.8" y="302" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">1</text>
|
||||
<text x="96.8" y="266.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">0</text>
|
||||
<rect x="167.6" y="264.3" width="53.7" height="7.7" rx="4" fill="#14b8a6"/>
|
||||
<text x="194.4" y="302" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">2</text>
|
||||
<text x="194.4" y="258.3" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">1</text>
|
||||
<rect x="265.2" y="210.3" width="53.7" height="61.7" rx="4" fill="#14b8a6"/>
|
||||
<text x="292.0" y="302" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">3</text>
|
||||
<text x="292.0" y="204.3" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">8</text>
|
||||
<rect x="362.8" y="94.6" width="53.7" height="177.4" rx="4" fill="#14b8a6"/>
|
||||
<text x="389.6" y="302" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">4</text>
|
||||
<text x="389.6" y="88.6" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">23</text>
|
||||
<rect x="460.4" y="56.0" width="53.7" height="216.0" rx="4" fill="#14b8a6"/>
|
||||
<text x="487.2" y="302" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">5</text>
|
||||
<text x="487.2" y="50.0" text-anchor="middle" font-family="Inter, system-ui, sans-serif" font-size="11" font-weight="600" fill="#111827">28</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.1 KiB |
40
docs/assets/human-eval/human-vs-llm-by-document.svg
Normal file
40
docs/assets/human-eval/human-vs-llm-by-document.svg
Normal file
@@ -0,0 +1,40 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="420" viewBox="0 0 900 420" role="img" aria-label="Human vs LLM by document">
|
||||
<title>Human vs LLM by document</title>
|
||||
<rect width="900" height="420" fill="#ffffff"/>
|
||||
<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Human score vs LLM answer similarity by document</text>
|
||||
<rect x="620" y="14" width="12" height="12" rx="2" fill="#14b8a6"/>
|
||||
<text x="638" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Human (1–5)</text>
|
||||
<rect x="760" y="14" width="12" height="12" rx="2" fill="#f59e0b"/>
|
||||
<text x="778" y="25" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">LLM similarity (1–10)</text>
|
||||
<rect x="59.9" y="99.7" width="26.5" height="232.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="92.4" y="104.2" width="26.5" height="227.8" rx="3" fill="#f59e0b"/>
|
||||
<text x="89.4" y="384" text-anchor="end" transform="rotate(-32 89.4 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">bazresi</text>
|
||||
<rect x="142.7" y="117.6" width="26.5" height="214.4" rx="3" fill="#14b8a6"/>
|
||||
<rect x="175.2" y="90.8" width="26.5" height="241.2" rx="3" fill="#f59e0b"/>
|
||||
<text x="172.2" y="384" text-anchor="end" transform="rotate(-32 172.2 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">customer1</text>
|
||||
<rect x="225.5" y="90.8" width="26.5" height="241.2" rx="3" fill="#14b8a6"/>
|
||||
<rect x="258.0" y="108.7" width="26.5" height="223.3" rx="3" fill="#f59e0b"/>
|
||||
<text x="255.0" y="384" text-anchor="end" transform="rotate(-32 255.0 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">fire</text>
|
||||
<rect x="308.3" y="126.5" width="26.5" height="205.5" rx="3" fill="#14b8a6"/>
|
||||
<rect x="340.8" y="104.2" width="26.5" height="227.8" rx="3" fill="#f59e0b"/>
|
||||
<text x="337.8" y="384" text-anchor="end" transform="rotate(-32 337.8 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades-ind</text>
|
||||
<rect x="391.1" y="117.6" width="26.5" height="214.4" rx="3" fill="#14b8a6"/>
|
||||
<rect x="423.6" y="95.3" width="26.5" height="236.7" rx="3" fill="#f59e0b"/>
|
||||
<text x="420.6" y="384" text-anchor="end" transform="rotate(-32 420.6 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">havades</text>
|
||||
<rect x="473.9" y="99.7" width="26.5" height="232.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="506.4" y="81.9" width="26.5" height="250.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="503.4" y="384" text-anchor="end" transform="rotate(-32 503.4 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-ind</text>
|
||||
<rect x="556.7" y="72.9" width="26.5" height="259.1" rx="3" fill="#14b8a6"/>
|
||||
<rect x="589.2" y="77.4" width="26.5" height="254.6" rx="3" fill="#f59e0b"/>
|
||||
<text x="586.2" y="384" text-anchor="end" transform="rotate(-32 586.2 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">moavenin</text>
|
||||
<rect x="639.5" y="81.9" width="26.5" height="250.1" rx="3" fill="#14b8a6"/>
|
||||
<rect x="672.0" y="81.9" width="26.5" height="250.1" rx="3" fill="#f59e0b"/>
|
||||
<text x="669.0" y="384" text-anchor="end" transform="rotate(-32 669.0 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">Refah</text>
|
||||
<rect x="722.3" y="108.7" width="26.5" height="223.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="754.8" y="90.8" width="26.5" height="241.2" rx="3" fill="#f59e0b"/>
|
||||
<text x="751.8" y="384" text-anchor="end" transform="rotate(-32 751.8 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">website</text>
|
||||
<rect x="805.1" y="99.7" width="26.5" height="232.3" rx="3" fill="#14b8a6"/>
|
||||
<rect x="837.6" y="104.2" width="26.5" height="227.8" rx="3" fill="#f59e0b"/>
|
||||
<text x="834.6" y="384" text-anchor="end" transform="rotate(-32 834.6 384)" font-family="Inter, system-ui, sans-serif" font-size="11" fill="#111827">lifetime-comp</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.9 KiB |
37
docs/assets/human-eval/mean-human-by-document.svg
Normal file
37
docs/assets/human-eval/mean-human-by-document.svg
Normal file
@@ -0,0 +1,37 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="720" height="380" viewBox="0 0 720 380" role="img" aria-label="Mean human score by document">
|
||||
<title>Mean human score by document</title>
|
||||
<rect width="720" height="380" fill="#ffffff"/>
|
||||
<text x="20" y="28" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="700" fill="#111827">Mean human score by document (1–5)</text>
|
||||
<text x="20" y="48" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#6b7280">fixed_size ±3 · stratified sample · higher is better</text>
|
||||
<rect x="120" y="61.4" width="429.3" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="73.4" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades-ind</text>
|
||||
<text x="555.3" y="73.4" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">3.83</text>
|
||||
<rect x="120" y="89.8" width="448.0" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="101.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">customer1</text>
|
||||
<text x="574.0" y="101.8" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.00</text>
|
||||
<rect x="120" y="118.2" width="448.0" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="130.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">havades</text>
|
||||
<text x="574.0" y="130.2" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.00</text>
|
||||
<rect x="120" y="146.6" width="466.7" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="158.6" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">website</text>
|
||||
<text x="592.7" y="158.6" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.17</text>
|
||||
<rect x="120" y="175.0" width="485.3" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="187.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">bazresi</text>
|
||||
<text x="611.3" y="187.0" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.33</text>
|
||||
<rect x="120" y="203.4" width="485.3" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="215.4" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-ind</text>
|
||||
<text x="611.3" y="215.4" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.33</text>
|
||||
<rect x="120" y="231.8" width="485.3" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="243.8" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">lifetime-comp</text>
|
||||
<text x="611.3" y="243.8" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.33</text>
|
||||
<rect x="120" y="260.2" width="504.0" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="272.2" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">fire</text>
|
||||
<text x="630.0" y="272.2" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.50</text>
|
||||
<rect x="120" y="288.6" width="522.7" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="300.6" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">Refah</text>
|
||||
<text x="648.7" y="300.6" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.67</text>
|
||||
<rect x="120" y="317.0" width="541.3" height="17.6" rx="4" fill="#14b8a6"/>
|
||||
<text x="112" y="329.0" text-anchor="end" font-family="Inter, system-ui, sans-serif" font-size="12" fill="#111827">moavenin</text>
|
||||
<text x="667.3" y="329.0" font-family="Inter, system-ui, sans-serif" font-size="12" font-weight="600" fill="#111827">4.83</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.9 KiB |
171
docs/final-chunking-strategy-decision.md
Normal file
171
docs/final-chunking-strategy-decision.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Chunking Strategy — Final Decision
|
||||
|
||||
**Date:** 17 August 2026 (family); **22 August 2026** (±N locked)
|
||||
**Decision:** Adopt **`fixed_size`** as the chunking Strategy family with **Neighbor Expansion ±3/3**.
|
||||
**Corpus Embedding Model:** `text-embedding-3-large`
|
||||
**Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe)
|
||||
**Questions:** 201 per Candidate (nomic semantic: 195)
|
||||
|
||||
**One-line close:** **`fixed_size ±3`** is the stabilized Strategy Candidate (**9.040** mean composite vs **8.727** for semantic @ large). **Do not use ±0** as the default — it loses to semantic on mean.
|
||||
|
||||
Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board:
|
||||
|
||||
```text
|
||||
(0.3 × context relevance + 0.4 × answer similarity + 0.3 × faithfulness)
|
||||
× (1 − hallucination rate)
|
||||
```
|
||||
|
||||
Regenerate figures after new Experiments:
|
||||
|
||||
```bash
|
||||
.venv/bin/python docs/assets/decision/generate_charts.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Why `fixed_size` is the winner
|
||||
|
||||
Decision Board ranks by **mean composite**, not by “how many documents won.” Stage 1 auto-picks **`fixed_size ±3`** vs **`semantic @ text-embedding-3-large`**. Stage 2 mean composite is **+0.313** for `fixed_size`.
|
||||
|
||||
|
||||
|
||||
| Candidate | Mean composite | Context | Similarity | Faithfulness | Hallucination | Docs |
|
||||
|-----------|----------------|---------|------------|--------------|---------------|------|
|
||||
| **fixed_size ±3** | **9.040** | 9.40 | 8.95 | 9.47 | 2.2% | 10/10 |
|
||||
| fixed_size ±2 | 8.987 | 9.43 | 8.93 | 9.44 | 2.7% | 10/10 |
|
||||
| fixed_size ±1 | 8.926 | 9.38 | 8.89 | 9.42 | 3.0% | 10/10 |
|
||||
| **semantic @ text-embedding-3-large** | **8.727** | 9.38 | 8.83 | 9.30 | 4.8% | 10/10 |
|
||||
| fixed_size ±0 | 8.606 | 9.18 | 8.79 | 9.25 | 4.9% | 10/10 |
|
||||
| semantic @ nomic-embed-text-v2-moe | 8.551 | 9.19 | 8.65 | 9.14 | 4.7% | 10/10 |
|
||||
|
||||
`fixed_size ±1`, `±2`, and `±3` all beat the best semantic Candidate on **mean**. **`±0` (no Neighbor Expansion) does not** (8.606 vs 8.727). Shipping `fixed_size` without expansion would weaken this family decision.
|
||||
|
||||
---
|
||||
|
||||
## 2. Judge metrics (stage 2 winners)
|
||||
|
||||
`fixed_size ±3` leads on every judge metric, including **lower hallucination**.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 3. Per-document showdown
|
||||
|
||||
Mean ranking still favors `fixed_size`. **Head-to-head at ±3: semantic 6 / `fixed_size` 4.** Semantic’s six wins are mostly **small**; `fixed_size`’s four wins are **larger**, especially **fire**.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
| Document | Questions | fixed_size ±3 | semantic @ large | Winner |
|
||||
|----------|-----------|---------------|------------------|--------|
|
||||
| fire.docx | 20 | **8.810** | 6.228 | **fixed_size** (+2.58) |
|
||||
| moavenin.docx | 20 | **9.735** | 8.621 | **fixed_size** (+1.11) |
|
||||
| general-havades-individuals.doc | 20 | **9.205** | 8.564 | **fixed_size** (+0.64) |
|
||||
| havades.docx | 20 | **8.483** | 7.969 | **fixed_size** (+0.51) |
|
||||
| bazresi.docx | 16 | 9.366 | **9.405** | semantic (−0.04) |
|
||||
| life-time-individual.docx | 20 | 9.445 | **9.505** | semantic (−0.06) |
|
||||
| Refah.docx | 20 | 9.680 | **9.815** | semantic (−0.14) |
|
||||
| lifetime-compensation.docx | 15 | 8.761 | **9.129** | semantic (−0.37) |
|
||||
| customer1.docx | 20 | 8.451 | **8.973** | semantic (−0.52) |
|
||||
| website.docx | 30 | 8.466 | **9.058** | semantic (−0.59) |
|
||||
|
||||
Win-count would pick semantic. **Official product ranking (mean composite) picks `fixed_size`.** This memo follows the Decision Board rule (ADR-0026).
|
||||
|
||||
---
|
||||
|
||||
## 4. Full Candidate heatmap
|
||||
|
||||
Every cell is a newest single-strategy Experiment under `text-embedding-3-large`. Semantic @ large **collapses on fire**; `fixed_size ±1…±3` stay high across the set.
|
||||
|
||||
|
||||
|
||||
Best `fixed_size` ±N **by document** (does not change the family call):
|
||||
|
||||
| ±N | Documents where it is the best fixed_size cell |
|
||||
|----|------------------------------------------------|
|
||||
| ±3 | bazresi, fire, general-havades-individuals, havades, life-time-individual (5) |
|
||||
| ±1 | customer1, moavenin, Refah (3) |
|
||||
| ±2 | website (1) |
|
||||
| ±0 | lifetime-compensation (1) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Method
|
||||
|
||||
- Corpus Embedding Model locked to **`text-embedding-3-large`**
|
||||
- Newest **single-strategy** Experiment per document × Candidate (Decision Board cells)
|
||||
- Retrieval `top_k = 5`
|
||||
- Stage 1: best `fixed_size` ±N vs best `semantic@Boundary`
|
||||
- Stage 2: mean composite + per-document breakdown
|
||||
- Experiments dated **9–10 August 2026**
|
||||
- `logs/neighbor_sweep.log` recorded mid-run connection errors on some units; **SQLite now has a full 10×4 `fixed_size` grid** — treat the database as source of truth
|
||||
|
||||
---
|
||||
|
||||
## 6. Excluded strategies
|
||||
|
||||
No comparable **10-doc, single-strategy, `text-embedding-3-large`** Experiments exist for the three Strategies below. They are not Decision Board Candidates (ADR-0026).
|
||||
|
||||
| Strategy | Why it is not the winner |
|
||||
|----------|--------------------------|
|
||||
| **recursive** | Not on the Decision Board grid. Informal PDF / `text-embedding-3-small` multi-strategy runs (not comparable to this close-out) were mixed; recursive **does not get Neighbor Expansion** (expansion is `fixed_size` only). |
|
||||
| **contextual_retrieval** | Extra LLM call per chunk at process time. Same informal PDF/`small` runs scored **below** recursive and `fixed_size`. |
|
||||
| **semantic_parent_child** | Boundary-detection cost plus parent fetch at query. Informal PDF/`small` composites were **much worse** (~1.8–2.8). Fail-hard if Boundary embeds mismatch (ADR-0020). |
|
||||
|
||||
One leftover five-strategy Experiment on `customer1` under large is **invalid** (four Strategies scored 0 / errors) and was ignored.
|
||||
|
||||
---
|
||||
|
||||
## 7. Binding decision
|
||||
|
||||
**Use `fixed_size` for chunking** under **`text-embedding-3-large`.**
|
||||
|
||||
**Use Neighbor Expansion `neighbor_prev=3`, `neighbor_next=3`** (symmetric **±3/3**) for Query and Experiment defaults when running `fixed_size`.
|
||||
|
||||
**Do not use `semantic` as the default Strategy** on this evaluation universe.
|
||||
|
||||
**Do not default to ±0.** Without expansion, semantic @ large beats `fixed_size` on mean composite (8.727 vs 8.606).
|
||||
|
||||
Operational defaults are set in `src/core/config.py`, `.env.example`, and the Dashboard Query/Benchmarks forms. Override per request is still supported.
|
||||
|
||||
---
|
||||
|
||||
## 8. Addendum — why ±3 (22 August 2026)
|
||||
|
||||
Stage 1 ranked all `fixed_size` Candidates on the 10-doc grid:
|
||||
|
||||
| ±N | Mean composite | vs semantic @ large |
|
||||
|----|----------------|---------------------|
|
||||
| ±0 | 8.606 | **loses** (−0.121) |
|
||||
| ±1 | 8.926 | wins (+0.199) |
|
||||
| ±2 | 8.987 | wins (+0.260) |
|
||||
| **±3** | **9.040** | **wins (+0.313)** |
|
||||
|
||||
**±3** is the stage 1 auto-pick and the highest mean composite. **±1** and **±2** are close; **±0** is ruled out.
|
||||
|
||||
Per-document best ±N varies (±3 wins on 5 docs, ±1 on 3, ±2 on 1, ±0 on 1). The **global** default is still **±3** because Decision Board ranks by mean composite across the full set, not by win-count among ±N levels.
|
||||
|
||||
**Caveats kept from stage 2:** semantic still wins head-to-head on 6/10 docs vs ±3 (website, customer1, etc.), but mean composite and judge metrics favor **`fixed_size ±3`**. Retrieval Inspect on outlier docs remains optional follow-up.
|
||||
|
||||
**Production RAG outside this repo** is not changed by this addendum — only benchmarker defaults and this record.
|
||||
|
||||
---
|
||||
|
||||
## 9. Evidence (stage 2 cells)
|
||||
|
||||
| Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id |
|
||||
|----------|-------------------------------|----------------------------------|
|
||||
| bazresi.docx | `6fe08750ee324130b3d46d8b3bc95280` | `6263c4f9e63347cea9eb19f4d36309c0` |
|
||||
| customer1.docx | `8c4d888d367b4c0299b2d19dd9654f80` | `0c4572e80902408796f0dd11776e8e55` |
|
||||
| fire.docx | `230a3cf57ae74b679ecde83cfeb9b6f5` | `414162efc314406db082ba8537c2423f` |
|
||||
| general-havades-individuals.doc | `7ad892cfcd6143109f77a7808c8dfbd9` | `14fa4f872373446c8b042fc8e84e6983` |
|
||||
| havades.docx | `04bd45d836a64c3da2e68bb747c0c9b8` | `232238b4d3304243b82b02326ef64617` |
|
||||
| life-time-individual.docx | `d682706f54824469b235c099bf5a3d3c` | `bc7d9f1fbe3540c097a5f8cc1a1c44bb` |
|
||||
| moavenin.docx | `63f81cf0a632418aab2f93884518ed71` | `4e01960b5db14e81975a4fbc86c4e10c` |
|
||||
| Refah.docx | `6304dba39f3f4c05935195c588a36a64` | `b5c630c45615451a93d97d80dee361df` |
|
||||
| website.docx | `fb9d47750ca74761a3c82c57aa51030b` | `53256897cfb74dd7a6508e750ff1ba63` |
|
||||
| lifetime-compensation.docx | `99736ffbc6da455b8b3ca3a0fb8e5ef7` | `e51c2b57e7364dfa97e5429f615d6e4e` |
|
||||
|
||||
HTML reports: `GET /benchmarks/{id}/report`. Dashboard: Decision Tab, Corpus = `text-embedding-3-large`.
|
||||
81
docs/human-eval-fixed-size-plus3-report.md
Normal file
81
docs/human-eval-fixed-size-plus3-report.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Human Evaluation Report — fixed_size ±3
|
||||
|
||||
**Date:** 22 August 2026
|
||||
**Strategy:** `fixed_size` with Neighbor Expansion **±3/3**
|
||||
**Corpus Embedding Model:** `text-embedding-3-large`
|
||||
**Sample:** **60 questions** (6 per document × **10 documents**)
|
||||
**Total benchmark universe:** 201 questions
|
||||
**Review effort:** ~6 hours
|
||||
|
||||
**Verdict:** Human review **supports** the stabilized **`fixed_size ±3`** default (see [final decision](final-chunking-strategy-decision.md)).
|
||||
|
||||
| KPI | Value |
|
||||
|-----|-------|
|
||||
| Mean human score (1–5) | **4.30** |
|
||||
| Mean LLM answer similarity (1–10) | **8.88** |
|
||||
| Human–LLM agreement | **57%** agree · 42% partial · 2% disagree |
|
||||
| Questions reviewed | **60** / 201 (30%) |
|
||||
|
||||
Raw scores: [fixed_size_plus3_human_scores.csv](assets/human-eval/fixed_size_plus3_human_scores.csv)
|
||||
|
||||
---
|
||||
|
||||
## 1. Mean human score by document
|
||||
|
||||
|
||||
|
||||
## 2. Human vs LLM-as-Judge agreement
|
||||
|
||||
|
||||
|
||||
Agreement = same quality tier when mapping LLM answer similarity (1–10) to human scale (1–5).
|
||||
|
||||
## 3. Human score vs LLM similarity by document
|
||||
|
||||
|
||||
|
||||
## 4. Human score distribution
|
||||
|
||||
|
||||
|
||||
## 5. Document summary
|
||||
|
||||
| Document | Reviewed | Mean human (1–5) | Mean LLM similarity | Agreement | |
|
||||
|----------|----------|------------------|---------------------|-----------|---|
|
||||
| bazresi | 6 | 4.33 | 8.50 | 50% | ⚠ |
|
||||
| customer1 | 6 | 4.00 | 9.00 | 33% | ⚠ |
|
||||
| fire | 6 | 4.50 | 8.33 | 67% | |
|
||||
| havades-ind | 6 | 3.83 | 8.50 | 50% | ⚠ |
|
||||
| havades | 6 | 4.00 | 8.83 | 50% | ⚠ |
|
||||
| lifetime-ind | 6 | 4.33 | 9.33 | 50% | ⚠ |
|
||||
| moavenin | 6 | 4.83 | 9.50 | 83% | |
|
||||
| Refah | 6 | 4.67 | 9.33 | 83% | |
|
||||
| website | 6 | 4.17 | 9.00 | 17% | ⚠ |
|
||||
| lifetime-comp | 6 | 4.33 | 8.50 | 83% | |
|
||||
|
||||
## 6. Notable disagreements (human stricter than LLM)
|
||||
|
||||
| Document | Question (excerpt) | Human | LLM sim | Notes |
|
||||
|----------|-------------------|-------|---------|-------|
|
||||
| havades-ind | آیا خطرات جنگ، زلزله و ورزشهای خطرناک قابل پوشش هستند؟ | 2 | 7 | Answer incomplete compared to expected reference. |
|
||||
|
||||
## 7. Method
|
||||
|
||||
- Stratified sample: **6 questions per document**, all **10** evaluation documents represented.
|
||||
- Human rubric: **1–5** (1 = wrong/unhelpful, 3 = partial, 5 = matches expected answer).
|
||||
- Each row: read question, expected answer, generated answer; score independently of automation.
|
||||
- LLM-as-Judge scores retained in CSV for comparison (context relevance, similarity, faithfulness, hallucination).
|
||||
- Source Experiments: newest single-strategy **`fixed_size ±3`** run per document under **`text-embedding-3-large`**.
|
||||
|
||||
## 8. Conclusions
|
||||
|
||||
1. **Overall quality is strong** — mean human score **4.30/5** on the reviewed sample.
|
||||
2. **Automated evaluation is directionally reliable** — **57%** agreement on quality tier; disagreements cluster on nuanced or incomplete answers.
|
||||
3. **website** and **customer1** show the largest human–LLM gaps; worth optional Retrieval Inspect follow-up, but do not overturn the global **`fixed_size ±3`** decision.
|
||||
4. Human review **confirms** the family and ±3 configuration documented in [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md).
|
||||
|
||||
Regenerate charts after CSV updates:
|
||||
|
||||
```bash
|
||||
.venv/bin/python docs/assets/human-eval/generate_human_eval_report.py
|
||||
```
|
||||
@@ -43,7 +43,7 @@ Status legend: `DONE` `IN_PROGRESS` `TODO`
|
||||
| 17 | Create question-answer evaluation dataset from insurance regulation document | DONE | 12 |
|
||||
| 18 | Implement HTML benchmark report generation system | DONE | 14 |
|
||||
| 19 | Design HTML report structure for experiment comparison and visualization | DONE | 14 |
|
||||
| 20 | Create background processing jobs for document ingestion and benchmarking | TODO | 12 |
|
||||
| 20 | Create background processing jobs for document ingestion and benchmarking | DONE | 12 |
|
||||
|
||||
## Phase 5 — Wiring + Verification
|
||||
|
||||
|
||||
@@ -57,6 +57,26 @@ def load_questions(file_path: str | Path) -> list[dict]:
|
||||
raise BenchmarkError(f"Invalid JSON in questions file: {exc}") from exc
|
||||
|
||||
|
||||
def resolve_benchmark_questions(
|
||||
*,
|
||||
questions: list[dict] | None = None,
|
||||
questions_file: str | None = None,
|
||||
) -> list[dict]:
|
||||
"""Resolve and validate benchmark questions from inline list or file."""
|
||||
if questions_file:
|
||||
return load_questions(questions_file)
|
||||
if questions and len(questions) > 0:
|
||||
valid_questions = [
|
||||
q for q in questions if isinstance(q, dict) and "question" in q
|
||||
]
|
||||
if not valid_questions:
|
||||
raise BenchmarkError(
|
||||
"Invalid questions: each question must have a 'question' field"
|
||||
)
|
||||
return valid_questions
|
||||
raise BenchmarkError("Either 'questions' or 'questions_file' must be provided")
|
||||
|
||||
|
||||
def load_questions_from_string(questions_json: str) -> list[dict]:
|
||||
"""Load questions from a JSON string.
|
||||
|
||||
@@ -399,6 +419,11 @@ def get_experiment(experiment_id: str) -> dict[str, Any] | None:
|
||||
return db.get_experiment(experiment_id)
|
||||
|
||||
|
||||
def list_experiments(document_id: str | None = None) -> dict[str, Any]:
|
||||
def list_experiments(
|
||||
document_id: str | None = None,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 200,
|
||||
) -> dict[str, Any]:
|
||||
"""List experiments, optionally filtered by document."""
|
||||
return db.list_experiments(document_id=document_id)
|
||||
return db.list_experiments(document_id=document_id, offset=offset, limit=limit)
|
||||
|
||||
@@ -3,17 +3,19 @@
|
||||
Endpoints:
|
||||
POST /queries Ask a question against a strategy
|
||||
GET /queries/{id} Retrieve a past query
|
||||
POST /benchmarks Run a benchmark (or dry run)
|
||||
POST /benchmarks Run a benchmark (or dry run); ?background=true enqueues a job
|
||||
GET /benchmarks/{id} Retrieve experiment results
|
||||
GET /experiments List all experiments
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi import APIRouter, BackgroundTasks, Query
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
|
||||
from src.core.exceptions import BenchmarkError, QueryError
|
||||
from src.core.models import StrategyName
|
||||
from src.benchmarking import benchmark_service, query_service
|
||||
from src.jobs import service as jobs_service
|
||||
from src.jobs.models import JobCreatedResponse
|
||||
from src.benchmarking.models import (
|
||||
BenchmarkRequest,
|
||||
BenchmarkResponse,
|
||||
@@ -88,50 +90,64 @@ async def get_query(query_id: str):
|
||||
|
||||
# ── Benchmark Endpoints ──────────────────────────────────────────
|
||||
|
||||
@router.post("/benchmarks", response_model=BenchmarkResponse, status_code=201)
|
||||
async def create_benchmark(request: BenchmarkRequest):
|
||||
@router.post("/benchmarks")
|
||||
async def create_benchmark(
|
||||
request: BenchmarkRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
background: bool = Query(
|
||||
False, description="Run in background; returns job_id (HTTP 202)"
|
||||
),
|
||||
):
|
||||
"""Run a benchmark comparing multiple strategies on multiple questions.
|
||||
|
||||
Can run in dry_run mode to get cost estimate without executing.
|
||||
Pass ``background=true`` to enqueue and poll ``GET /jobs/{job_id}``.
|
||||
"""
|
||||
# Load questions - prefer questions_file over inline questions
|
||||
if request.questions_file:
|
||||
questions = benchmark_service.load_questions(request.questions_file)
|
||||
elif request.questions and len(request.questions) > 0:
|
||||
# Validate that questions have required fields
|
||||
valid_questions = [
|
||||
q for q in request.questions
|
||||
if isinstance(q, dict) and "question" in q
|
||||
]
|
||||
if not valid_questions:
|
||||
raise BenchmarkError("Invalid questions: each question must have a 'question' field")
|
||||
questions = valid_questions
|
||||
else:
|
||||
raise BenchmarkError("Either 'questions' or 'questions_file' must be provided")
|
||||
questions = benchmark_service.resolve_benchmark_questions(
|
||||
questions=request.questions,
|
||||
questions_file=request.questions_file,
|
||||
)
|
||||
|
||||
if not questions:
|
||||
raise BenchmarkError("No questions to benchmark")
|
||||
|
||||
# Dry run - return cost estimate only
|
||||
if request.dry_run:
|
||||
if background:
|
||||
raise BenchmarkError("dry_run cannot be used with background=true")
|
||||
estimate = benchmark_service.estimate_cost(
|
||||
num_questions=len(questions),
|
||||
num_strategies=len(request.strategies),
|
||||
)
|
||||
# Return as BenchmarkResponse with minimal data
|
||||
return BenchmarkResponse(
|
||||
experiment_id="dry_run",
|
||||
document_id=request.document_id,
|
||||
strategies_used=[s.value for s in request.strategies],
|
||||
questions_count=len(questions),
|
||||
aggregate_metrics={},
|
||||
best_strategy="N/A (dry run)",
|
||||
total_latency_seconds=0,
|
||||
estimated_cost_usd=estimate["estimated_cost_usd"],
|
||||
created_at="N/A",
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content=BenchmarkResponse(
|
||||
experiment_id="dry_run",
|
||||
document_id=request.document_id,
|
||||
strategies_used=[s.value for s in request.strategies],
|
||||
questions_count=len(questions),
|
||||
aggregate_metrics={},
|
||||
best_strategy="N/A (dry run)",
|
||||
total_latency_seconds=0,
|
||||
estimated_cost_usd=estimate["estimated_cost_usd"],
|
||||
created_at="N/A",
|
||||
).model_dump(),
|
||||
)
|
||||
|
||||
# Run full benchmark
|
||||
if background:
|
||||
payload = request.model_dump(mode="json")
|
||||
payload["questions"] = questions
|
||||
payload.pop("questions_file", None)
|
||||
payload.pop("dry_run", None)
|
||||
job = jobs_service.enqueue_benchmark(payload)
|
||||
background_tasks.add_task(jobs_service.run_benchmark_job, job["id"])
|
||||
body = JobCreatedResponse(
|
||||
job_id=job["id"],
|
||||
job_type=job["job_type"],
|
||||
status=job["status"],
|
||||
poll_url=f"/jobs/{job['id']}",
|
||||
)
|
||||
return JSONResponse(status_code=202, content=body.model_dump())
|
||||
|
||||
result = benchmark_service.run_benchmark(
|
||||
document_id=request.document_id,
|
||||
strategies=request.strategies,
|
||||
@@ -142,7 +158,7 @@ async def create_benchmark(request: BenchmarkRequest):
|
||||
corpus_model_id=request.corpus_model_id,
|
||||
)
|
||||
|
||||
return BenchmarkResponse(
|
||||
response = BenchmarkResponse(
|
||||
experiment_id=result["experiment_id"],
|
||||
document_id=result["document_id"],
|
||||
strategies_used=result["strategies_used"],
|
||||
@@ -155,6 +171,7 @@ async def create_benchmark(request: BenchmarkRequest):
|
||||
estimated_cost_usd=result["estimated_cost_usd"],
|
||||
created_at=result["created_at"],
|
||||
)
|
||||
return JSONResponse(status_code=201, content=response.model_dump())
|
||||
|
||||
|
||||
@router.get("/benchmarks/{experiment_id}", response_model=ExperimentDetailResponse)
|
||||
@@ -185,9 +202,15 @@ async def get_benchmark(experiment_id: str):
|
||||
|
||||
|
||||
@router.get("/experiments")
|
||||
async def list_experiments(document_id: str | None = None):
|
||||
async def list_experiments(
|
||||
document_id: str | None = None,
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(200, ge=1, le=500),
|
||||
):
|
||||
"""List all experiments, optionally filtered by document."""
|
||||
result = benchmark_service.list_experiments(document_id=document_id)
|
||||
result = benchmark_service.list_experiments(
|
||||
document_id=document_id, offset=offset, limit=limit
|
||||
)
|
||||
# Enrich with document filenames and best_strategy
|
||||
from src.storage import sqlite as db
|
||||
for item in result.get("items", []):
|
||||
|
||||
@@ -22,9 +22,9 @@ class Settings(BaseSettings):
|
||||
|
||||
# Retrieval
|
||||
top_k: int = 5
|
||||
# Neighbor Expansion for fixed_size (ADR-0023); 0/0 = off
|
||||
neighbor_prev: int = 0
|
||||
neighbor_next: int = 0
|
||||
# Neighbor Expansion for fixed_size (ADR-0023); 3/3 = decision default (see final-chunking-strategy-decision.md)
|
||||
neighbor_prev: int = 3
|
||||
neighbor_next: int = 3
|
||||
|
||||
# LLM generation
|
||||
temperature: float = 0.0
|
||||
|
||||
@@ -2,12 +2,13 @@
|
||||
|
||||
Endpoints:
|
||||
POST /documents Upload a .docx / .doc / .pdf file
|
||||
POST /documents/{id}/process Run chunking strategies
|
||||
POST /documents/{id}/process Run chunking strategies; ?background=true enqueues a job
|
||||
DELETE /documents/{id} Remove document + vectors
|
||||
GET /strategies List available chunking strategies
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, File, UploadFile
|
||||
from fastapi import APIRouter, BackgroundTasks, File, Query, UploadFile
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from src.core.exceptions import DocumentProcessingError
|
||||
from src.documents.models import (
|
||||
@@ -21,6 +22,9 @@ from src.documents.models import (
|
||||
)
|
||||
from src.documents.parser import SUPPORTED_SUFFIXES
|
||||
from src.documents import service
|
||||
from src.jobs import service as jobs_service
|
||||
from src.jobs.models import JobCreatedResponse
|
||||
from src.storage import sqlite as db
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -63,9 +67,32 @@ async def upload_document(file: UploadFile = File(...)):
|
||||
)
|
||||
|
||||
|
||||
@router.post("/documents/{doc_id}/process", response_model=ProcessResponse)
|
||||
async def process_document(doc_id: str, request: ProcessRequest):
|
||||
"""Run selected chunking strategies on an uploaded document."""
|
||||
@router.post("/documents/{doc_id}/process")
|
||||
async def process_document(
|
||||
doc_id: str,
|
||||
request: ProcessRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
background: bool = Query(
|
||||
False, description="Run in background; returns job_id (HTTP 202)"
|
||||
),
|
||||
):
|
||||
"""Run selected chunking strategies on an uploaded document.
|
||||
|
||||
Pass ``background=true`` to enqueue and poll ``GET /jobs/{job_id}``.
|
||||
"""
|
||||
if background:
|
||||
if db.get_document(doc_id) is None:
|
||||
raise DocumentProcessingError(f"Document not found: {doc_id}")
|
||||
job = jobs_service.enqueue_process(doc_id, request)
|
||||
background_tasks.add_task(jobs_service.run_process_job, job["id"])
|
||||
body = JobCreatedResponse(
|
||||
job_id=job["id"],
|
||||
job_type=job["job_type"],
|
||||
status=job["status"],
|
||||
poll_url=f"/jobs/{job['id']}",
|
||||
)
|
||||
return JSONResponse(status_code=202, content=body.model_dump())
|
||||
|
||||
return service.process_document(doc_id, request)
|
||||
|
||||
|
||||
|
||||
1
src/jobs/__init__.py
Normal file
1
src/jobs/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Background job tracking for long-running benchmark and process operations."""
|
||||
48
src/jobs/models.py
Normal file
48
src/jobs/models.py
Normal file
@@ -0,0 +1,48 @@
|
||||
"""Request/response schemas for the Jobs API."""
|
||||
|
||||
from enum import Enum
|
||||
from typing import Any, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class JobType(str, Enum):
|
||||
BENCHMARK = "benchmark"
|
||||
PROCESS = "process"
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
class JobCreatedResponse(BaseModel):
|
||||
"""Returned when a long-running operation is enqueued (HTTP 202)."""
|
||||
job_id: str
|
||||
job_type: str
|
||||
status: str = "pending"
|
||||
poll_url: str
|
||||
|
||||
|
||||
class JobDetailResponse(BaseModel):
|
||||
"""Full job status including result or error when finished."""
|
||||
id: str
|
||||
job_type: str
|
||||
status: str
|
||||
payload: dict[str, Any] = Field(default_factory=dict)
|
||||
result: Optional[dict[str, Any]] = None
|
||||
error: Optional[str] = None
|
||||
progress: dict[str, Any] = Field(default_factory=dict)
|
||||
created_at: str
|
||||
started_at: Optional[str] = None
|
||||
completed_at: Optional[str] = None
|
||||
|
||||
|
||||
class JobListResponse(BaseModel):
|
||||
"""Paginated list of jobs."""
|
||||
items: list[JobDetailResponse]
|
||||
total: int
|
||||
offset: int
|
||||
limit: int
|
||||
42
src/jobs/routes.py
Normal file
42
src/jobs/routes.py
Normal file
@@ -0,0 +1,42 @@
|
||||
"""Job status API routes.
|
||||
|
||||
Endpoints:
|
||||
GET /jobs List background jobs
|
||||
GET /jobs/{id} Poll job status and result
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from src.core.exceptions import BenchmarkError
|
||||
from src.jobs.models import JobDetailResponse, JobListResponse
|
||||
from src.storage import sqlite as db
|
||||
|
||||
router = APIRouter(tags=["jobs"])
|
||||
|
||||
|
||||
@router.get("/jobs", response_model=JobListResponse)
|
||||
async def list_jobs(
|
||||
job_type: str | None = Query(None, description="Filter by job_type (benchmark|process)"),
|
||||
status: str | None = Query(None, description="Filter by status"),
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=200),
|
||||
):
|
||||
"""List background jobs, newest first."""
|
||||
result = db.list_jobs(
|
||||
job_type=job_type, status=status, offset=offset, limit=limit
|
||||
)
|
||||
return JobListResponse(
|
||||
items=[JobDetailResponse(**item) for item in result["items"]],
|
||||
total=result["total"],
|
||||
offset=result["offset"],
|
||||
limit=result["limit"],
|
||||
)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=JobDetailResponse)
|
||||
async def get_job(job_id: str):
|
||||
"""Poll a background job by ID."""
|
||||
job = db.get_job(job_id)
|
||||
if job is None:
|
||||
raise BenchmarkError(f"Job not found: {job_id}")
|
||||
return JobDetailResponse(**job)
|
||||
114
src/jobs/service.py
Normal file
114
src/jobs/service.py
Normal file
@@ -0,0 +1,114 @@
|
||||
"""Background job enqueue and execution via FastAPI BackgroundTasks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from src.benchmarking import benchmark_service
|
||||
from src.core.models import StrategyName
|
||||
from src.documents import service as documents_service
|
||||
from src.documents.models import ProcessRequest
|
||||
from src.storage import sqlite as db
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def enqueue_benchmark(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Create a pending benchmark job with a resolved questions payload."""
|
||||
return db.create_job(job_type="benchmark", payload=payload)
|
||||
|
||||
|
||||
def enqueue_process(document_id: str, request: ProcessRequest) -> dict[str, Any]:
|
||||
"""Create a pending document processing job."""
|
||||
payload = {"document_id": document_id, **request.model_dump(mode="json")}
|
||||
return db.create_job(job_type="process", payload=payload)
|
||||
|
||||
|
||||
def run_benchmark_job(job_id: str) -> None:
|
||||
"""Execute a benchmark job (runs in a FastAPI background task)."""
|
||||
job = db.get_job(job_id)
|
||||
if job is None:
|
||||
logger.error("Benchmark job not found: %s", job_id)
|
||||
return
|
||||
|
||||
payload = job["payload"]
|
||||
try:
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="running",
|
||||
started_at=_now(),
|
||||
progress={"message": "Running benchmark"},
|
||||
)
|
||||
strategies = [StrategyName(s) for s in payload["strategies"]]
|
||||
result = benchmark_service.run_benchmark(
|
||||
document_id=payload["document_id"],
|
||||
strategies=strategies,
|
||||
questions=payload["questions"],
|
||||
top_k=payload.get("top_k", 5),
|
||||
neighbor_prev=payload.get("neighbor_prev", 0),
|
||||
neighbor_next=payload.get("neighbor_next", 0),
|
||||
corpus_model_id=payload.get("corpus_model_id"),
|
||||
)
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="completed",
|
||||
result=result,
|
||||
completed_at=_now(),
|
||||
progress={"message": "Benchmark completed"},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("Benchmark job %s failed", job_id)
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="failed",
|
||||
error=str(exc),
|
||||
completed_at=_now(),
|
||||
progress={"message": "Benchmark failed"},
|
||||
)
|
||||
|
||||
|
||||
def run_process_job(job_id: str) -> None:
|
||||
"""Execute a document processing job (runs in a FastAPI background task)."""
|
||||
job = db.get_job(job_id)
|
||||
if job is None:
|
||||
logger.error("Process job not found: %s", job_id)
|
||||
return
|
||||
|
||||
payload = job["payload"]
|
||||
document_id = payload["document_id"]
|
||||
request = ProcessRequest(
|
||||
strategies=[StrategyName(s) for s in payload["strategies"]],
|
||||
boundary_model_id=payload.get("boundary_model_id"),
|
||||
corpus_model_id=payload.get("corpus_model_id"),
|
||||
)
|
||||
|
||||
try:
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="running",
|
||||
started_at=_now(),
|
||||
progress={"message": "Processing document"},
|
||||
)
|
||||
response = documents_service.process_document(document_id, request)
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="completed",
|
||||
result=response.model_dump(mode="json"),
|
||||
completed_at=_now(),
|
||||
progress={"message": "Processing completed"},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("Process job %s failed", job_id)
|
||||
db.update_job(
|
||||
job_id,
|
||||
status="failed",
|
||||
error=str(exc),
|
||||
completed_at=_now(),
|
||||
progress={"message": "Processing failed"},
|
||||
)
|
||||
@@ -21,6 +21,7 @@ from src.core.exceptions import (
|
||||
from src.documents.routes import router as documents_router
|
||||
from src.benchmarking.routes import router as benchmarking_router
|
||||
from src.admin.routes import router as admin_router
|
||||
from src.jobs.routes import router as jobs_router
|
||||
from src.storage.sqlite import init_db
|
||||
|
||||
# Configure logging
|
||||
@@ -95,6 +96,7 @@ def create_app() -> FastAPI:
|
||||
app.include_router(documents_router)
|
||||
app.include_router(benchmarking_router)
|
||||
app.include_router(admin_router)
|
||||
app.include_router(jobs_router)
|
||||
|
||||
# Mount dashboard at /app
|
||||
static_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "static")
|
||||
|
||||
@@ -434,6 +434,128 @@
|
||||
.cmp-exp-2 { background: rgba(99,102,241,0.15); color: #818cf8; }
|
||||
.cmp-exp-3 { background: rgba(244,63,94,0.15); color: #fb7185; }
|
||||
.cmp-divider { height: 1px; background: var(--border); margin: 20px 0; }
|
||||
/* ── Decision Board ─────────────────────────────────────── */
|
||||
.decision-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
|
||||
@media (max-width: 960px) { .decision-grid { grid-template-columns: 1fr; } }
|
||||
.decision-cand {
|
||||
border: 1px solid var(--border); border-radius: var(--radius);
|
||||
padding: 12px 14px; margin-bottom: 8px; cursor: pointer;
|
||||
background: var(--bg-base); transition: border-color 0.15s, background 0.15s;
|
||||
}
|
||||
.decision-cand:hover { border-color: #3a3a42; }
|
||||
.decision-cand.selected {
|
||||
border-color: var(--accent); background: rgba(234,179,8,0.08);
|
||||
}
|
||||
.decision-cand .cand-title { font-weight: 600; color: var(--text-heading); margin-bottom: 6px; }
|
||||
.decision-metrics { display: flex; flex-wrap: wrap; gap: 8px 14px; font-size: 12px; color: var(--text-muted); }
|
||||
.decision-metrics strong { color: var(--text-body); font-variant-numeric: tabular-nums; }
|
||||
.decision-legend {
|
||||
display: flex; flex-wrap: wrap; gap: 10px 16px; align-items: center;
|
||||
margin-bottom: 14px; padding: 12px 14px;
|
||||
background: rgba(15, 20, 25, 0.55);
|
||||
border: 1px solid rgba(255,255,255,0.06);
|
||||
border-radius: 10px;
|
||||
font-size: 12px; color: var(--text-muted);
|
||||
}
|
||||
.decision-legend-item { display: inline-flex; align-items: center; gap: 7px; }
|
||||
.decision-legend-swatch {
|
||||
width: 22px; height: 14px; border-radius: 4px;
|
||||
border: 1px solid rgba(255,255,255,0.08);
|
||||
box-shadow: inset 3px 0 0 var(--swatch-accent, transparent);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.decision-matrix-wrap {
|
||||
overflow-x: auto;
|
||||
border: 1px solid rgba(255,255,255,0.06);
|
||||
border-radius: 12px;
|
||||
background: rgba(15, 20, 25, 0.35);
|
||||
}
|
||||
.decision-matrix {
|
||||
width: 100%; border-collapse: separate; border-spacing: 0;
|
||||
font-size: 13px; margin: 0;
|
||||
}
|
||||
.decision-matrix th,
|
||||
.decision-matrix td {
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid rgba(255,255,255,0.05);
|
||||
border-right: 1px solid rgba(255,255,255,0.04);
|
||||
text-align: center;
|
||||
vertical-align: middle;
|
||||
}
|
||||
.decision-matrix th:last-child,
|
||||
.decision-matrix td:last-child { border-right: none; }
|
||||
.decision-matrix tbody tr:last-child th,
|
||||
.decision-matrix tbody tr:last-child td { border-bottom: none; }
|
||||
.decision-matrix thead th {
|
||||
position: sticky; top: 0; z-index: 2;
|
||||
background: rgba(21, 32, 51, 0.96);
|
||||
backdrop-filter: blur(8px);
|
||||
color: #94A3B8;
|
||||
font-size: 11px; font-weight: 600;
|
||||
letter-spacing: 0.03em;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.decision-matrix .sticky-col {
|
||||
position: sticky; left: 0; z-index: 1;
|
||||
text-align: left; font-weight: 500; white-space: nowrap;
|
||||
background: #1A2332; color: var(--text-heading);
|
||||
min-width: 160px; max-width: 220px;
|
||||
}
|
||||
.decision-matrix thead .sticky-col { z-index: 3; background: rgba(21, 32, 51, 0.96); }
|
||||
.decision-matrix tr.mean-row td,
|
||||
.decision-matrix tr.mean-row .sticky-col {
|
||||
border-top: 1px solid rgba(255,255,255,0.1);
|
||||
background: rgba(15, 20, 25, 0.65);
|
||||
font-weight: 700;
|
||||
}
|
||||
.decision-matrix tr.wins-row td,
|
||||
.decision-matrix tr.wins-row .sticky-col {
|
||||
background: rgba(15, 20, 25, 0.45);
|
||||
color: var(--text-muted);
|
||||
font-weight: 600;
|
||||
}
|
||||
.decision-matrix th.col-duel {
|
||||
color: #FBBF24;
|
||||
}
|
||||
.decision-heat {
|
||||
font-variant-numeric: tabular-nums;
|
||||
font-size: 12.5px; font-weight: 600;
|
||||
min-width: 76px;
|
||||
letter-spacing: 0.01em;
|
||||
transition: filter 0.15s ease;
|
||||
}
|
||||
.decision-heat:hover { filter: brightness(1.12); }
|
||||
.decision-heat .best-mark,
|
||||
.decision-legend .best-mark,
|
||||
.best-mark {
|
||||
display: inline-block;
|
||||
font-size: 9px; font-weight: 700; line-height: 1;
|
||||
margin-right: 5px; padding: 2px 5px;
|
||||
border-radius: 999px; vertical-align: middle;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
.decision-heat .best-mark.fs,
|
||||
.decision-legend .best-mark.fs,
|
||||
.best-mark.fs {
|
||||
background: rgba(56, 189, 248, 0.15);
|
||||
color: #7DD3FC;
|
||||
border: 1px solid rgba(56, 189, 248, 0.28);
|
||||
}
|
||||
.decision-heat .best-mark.sem,
|
||||
.decision-legend .best-mark.sem,
|
||||
.best-mark.sem {
|
||||
background: rgba(167, 139, 250, 0.15);
|
||||
color: #C4B5FD;
|
||||
border: 1px solid rgba(167, 139, 250, 0.28);
|
||||
}
|
||||
.decision-duel {
|
||||
display: grid; grid-template-columns: 1fr auto 1fr; gap: 16px; align-items: stretch;
|
||||
}
|
||||
@media (max-width: 800px) { .decision-duel { grid-template-columns: 1fr; } }
|
||||
.decision-vs {
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-weight: 700; color: var(--text-muted); font-size: 18px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
@@ -554,7 +676,7 @@ function HomeTab() {
|
||||
useEffect(() => {
|
||||
Promise.all([
|
||||
api('/documents').catch(() => ({ total: 0 })),
|
||||
api('/experiments').catch(() => []),
|
||||
api('/experiments?limit=500').catch(() => []),
|
||||
]).then(([docs, exps]) => {
|
||||
setStats({ docs: docs.total || 0, experiments: (exps?.items || []).length });
|
||||
setLoading(false);
|
||||
@@ -589,6 +711,7 @@ function HomeTab() {
|
||||
React.createElement('li', null, 'Process it with chunking strategies'),
|
||||
React.createElement('li', null, 'Ask a question in ', React.createElement('b', null, 'Query'), ' (or inside PDF Workspace)'),
|
||||
React.createElement('li', null, 'Run a full benchmark in ', React.createElement('b', null, 'Benchmarks'), ' / PDF Workspace'),
|
||||
React.createElement('li', null, 'Pick a final Strategy on the ', React.createElement('b', null, 'Decision'), ' board'),
|
||||
React.createElement('li', null, 'Check system health in the ', React.createElement('b', null, 'Admin'), ' tab')
|
||||
)
|
||||
)
|
||||
@@ -1164,8 +1287,8 @@ function RetrievalInspectView({ experiment, onBack }) {
|
||||
|
||||
function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
|
||||
const [form, setForm] = useState({
|
||||
document_id: '', strategy: '', question: '', top_k: 5,
|
||||
neighbor_prev: 0, neighbor_next: 0, corpus_model_id: '',
|
||||
document_id: '', strategy: 'fixed_size', question: '', top_k: 5,
|
||||
neighbor_prev: 3, neighbor_next: 3, corpus_model_id: '',
|
||||
});
|
||||
const [result, setResult] = useState(null);
|
||||
const [loading, setLoading] = useState(false);
|
||||
@@ -1638,9 +1761,9 @@ function ComparisonView({ experiments, addToast, onBack }) {
|
||||
function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) {
|
||||
const [experiments, setExperiments] = useState([]);
|
||||
const [form, setForm] = useState({
|
||||
document_id: '', strategies: strategies.map(s => s.name),
|
||||
document_id: '', strategies: ['fixed_size'],
|
||||
questions_file: questionsFile, top_k: 5,
|
||||
neighbor_prev: 0, neighbor_next: 0, dry_run: false, corpus_model_id: '',
|
||||
neighbor_prev: 3, neighbor_next: 3, dry_run: false, corpus_model_id: '',
|
||||
});
|
||||
const [running, setRunning] = useState(false);
|
||||
const [embModels, setEmbModels] = useState([]);
|
||||
@@ -1660,7 +1783,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatF
|
||||
const [inspectLoadingId, setInspectLoadingId] = useState(null);
|
||||
|
||||
const fetchExperiments = useCallback(() => {
|
||||
api('/experiments').then(d => setExperiments(d?.items || (Array.isArray(d) ? d : []))).catch(() => {});
|
||||
api('/experiments?limit=500').then(d => setExperiments(d?.items || (Array.isArray(d) ? d : []))).catch(() => {});
|
||||
}, []);
|
||||
|
||||
useEffect(() => { fetchExperiments(); }, [fetchExperiments]);
|
||||
@@ -2267,6 +2390,648 @@ function PdfWorkspaceTab({ documents, setDocuments, strategies, addToast, questi
|
||||
);
|
||||
}
|
||||
|
||||
// -- Tab: Decision Board (ADR-0026) --------------------------
|
||||
const DECISION_DOC_FILENAMES = [
|
||||
'bazresi.docx',
|
||||
'customer1.docx',
|
||||
'fire.docx',
|
||||
'general-havades-individuals.doc',
|
||||
'havades.docx',
|
||||
'life-time-individual.docx',
|
||||
'moavenin.docx',
|
||||
'Refah.docx',
|
||||
'website.docx',
|
||||
'lifetime-compensation.docx',
|
||||
];
|
||||
|
||||
function decisionComposite(m) {
|
||||
if (!m) return null;
|
||||
return ((m.avg_context_relevance || 0) * 0.3
|
||||
+ (m.avg_answer_similarity || 0) * 0.4
|
||||
+ (m.avg_faithfulness || 0) * 0.3) * (1 - (m.hallucination_rate || 0));
|
||||
}
|
||||
|
||||
function decisionFmt(v, digits) {
|
||||
if (v == null || Number.isNaN(v)) return '—';
|
||||
return Number(v).toFixed(digits == null ? 2 : digits);
|
||||
}
|
||||
|
||||
function decisionPct(v) {
|
||||
if (v == null || Number.isNaN(v)) return '—';
|
||||
return `${(Number(v) * 100).toFixed(0)}%`;
|
||||
}
|
||||
|
||||
function decisionHeatBin(score) {
|
||||
if (score == null || Number.isNaN(score)) {
|
||||
return {
|
||||
key: 'missing', label: '—',
|
||||
bg: 'transparent', fg: 'var(--text-muted)', accent: 'transparent',
|
||||
};
|
||||
}
|
||||
const s = Number(score);
|
||||
// Modern dark-dashboard scale: translucent wash + luminous text + left accent rail
|
||||
// (no solid primary blocks — avoids Windows-98 / crayon look)
|
||||
if (s < 8.0) {
|
||||
return {
|
||||
key: 'lt80', label: '<8.0',
|
||||
bg: 'rgba(244, 63, 94, 0.14)', fg: '#FB7185', accent: '#F43F5E',
|
||||
};
|
||||
}
|
||||
if (s < 8.5) {
|
||||
return {
|
||||
key: '80_85', label: '8.0–8.5',
|
||||
bg: 'rgba(251, 146, 60, 0.13)', fg: '#FB923C', accent: '#F97316',
|
||||
};
|
||||
}
|
||||
if (s < 9.0) {
|
||||
return {
|
||||
key: '85_90', label: '8.5–9.0',
|
||||
bg: 'rgba(250, 204, 21, 0.14)', fg: '#FDE047', accent: '#EAB308',
|
||||
};
|
||||
}
|
||||
if (s < 9.5) {
|
||||
return {
|
||||
key: '90_95', label: '9.0–9.5',
|
||||
bg: 'rgba(45, 212, 191, 0.13)', fg: '#2DD4BF', accent: '#14B8A6',
|
||||
};
|
||||
}
|
||||
return {
|
||||
key: 'ge95', label: '≥9.5',
|
||||
bg: 'rgba(52, 211, 153, 0.16)', fg: '#34D399', accent: '#10B981',
|
||||
};
|
||||
}
|
||||
|
||||
function decisionHeatStyle(score) {
|
||||
const bin = decisionHeatBin(score);
|
||||
const style = { background: bin.bg, color: bin.fg };
|
||||
if (bin.accent && bin.accent !== 'transparent') {
|
||||
style.boxShadow = `inset 3px 0 0 ${bin.accent}`;
|
||||
}
|
||||
return style;
|
||||
}
|
||||
|
||||
/** Per-row (or mean-row) ids of best fixed_size and best semantic Candidate. */
|
||||
function decisionFamilyBestIds(candidates, scoreOf) {
|
||||
let bestFs = null, bestFsScore = -Infinity;
|
||||
let bestSem = null, bestSemScore = -Infinity;
|
||||
(candidates || []).forEach(c => {
|
||||
const s = scoreOf(c);
|
||||
if (s == null || Number.isNaN(s)) return;
|
||||
if (c.family === 'fixed_size' && s > bestFsScore) {
|
||||
bestFsScore = s;
|
||||
bestFs = c.id;
|
||||
}
|
||||
if (c.family === 'semantic' && s > bestSemScore) {
|
||||
bestSemScore = s;
|
||||
bestSem = c.id;
|
||||
}
|
||||
});
|
||||
return { fixed: bestFs, semantic: bestSem };
|
||||
}
|
||||
|
||||
const DECISION_HEAT_LEGEND = [
|
||||
decisionHeatBin(null),
|
||||
decisionHeatBin(7.9),
|
||||
decisionHeatBin(8.2),
|
||||
decisionHeatBin(8.7),
|
||||
decisionHeatBin(9.2),
|
||||
decisionHeatBin(9.6),
|
||||
];
|
||||
|
||||
function DecisionHeatLegend() {
|
||||
return React.createElement('div', { className: 'decision-legend', role: 'list', 'aria-label': 'Composite score color guide' },
|
||||
React.createElement('span', { style: { fontWeight: 600, color: 'var(--text-body)' } }, 'Score guide'),
|
||||
DECISION_HEAT_LEGEND.map(bin =>
|
||||
React.createElement('span', { key: bin.key, className: 'decision-legend-item', role: 'listitem' },
|
||||
React.createElement('span', {
|
||||
className: 'decision-legend-swatch',
|
||||
style: {
|
||||
background: bin.bg === 'transparent' ? 'rgba(255,255,255,0.04)' : bin.bg,
|
||||
boxShadow: bin.accent && bin.accent !== 'transparent' ? `inset 3px 0 0 ${bin.accent}` : undefined,
|
||||
},
|
||||
'aria-hidden': true,
|
||||
}),
|
||||
bin.label
|
||||
)
|
||||
),
|
||||
React.createElement('span', {
|
||||
style: { width: 1, height: 14, background: 'rgba(255,255,255,0.1)', margin: '0 4px' },
|
||||
'aria-hidden': true,
|
||||
}),
|
||||
React.createElement('span', { className: 'decision-legend-item' },
|
||||
React.createElement('span', { className: 'best-mark fs' }, 'F'),
|
||||
'best fixed_size'
|
||||
),
|
||||
React.createElement('span', { className: 'decision-legend-item' },
|
||||
React.createElement('span', { className: 'best-mark sem' }, 'S'),
|
||||
'best semantic'
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function meanOf(nums) {
|
||||
const vals = nums.filter(v => v != null && !Number.isNaN(v));
|
||||
if (!vals.length) return null;
|
||||
return vals.reduce((a, b) => a + b, 0) / vals.length;
|
||||
}
|
||||
|
||||
function buildDecisionBoard(experiments, corpusId, excludedIds) {
|
||||
const excluded = new Set(excludedIds || []);
|
||||
const pool = (experiments || []).filter(e => {
|
||||
if (excluded.has(e.id)) return false;
|
||||
const fn = e.document_filename || '';
|
||||
if (!DECISION_DOC_FILENAMES.includes(fn)) return false;
|
||||
if (corpusIdOf(e) !== corpusId) return false;
|
||||
const strats = e.strategies_used || [];
|
||||
return strats.length === 1;
|
||||
});
|
||||
|
||||
// Newest first assumed from API; keep first hit per cell
|
||||
const cellMap = {}; // key: `${fn}||${candId}` -> exp
|
||||
const fsLevels = new Set([0, 1, 2, 3]);
|
||||
const semBounds = new Set();
|
||||
|
||||
pool.forEach(e => {
|
||||
const strat = (e.strategies_used || [])[0];
|
||||
const fn = e.document_filename;
|
||||
if (strat === 'fixed_size') {
|
||||
const { prev, next } = neighborCounts(e);
|
||||
if (prev !== next || !fsLevels.has(prev)) return;
|
||||
const candId = `fixed_size:±${prev}`;
|
||||
const key = `${fn}||${candId}`;
|
||||
if (!cellMap[key]) cellMap[key] = e;
|
||||
} else if (strat === 'semantic') {
|
||||
const b = boundaryIdOf(e);
|
||||
if (!b) return;
|
||||
semBounds.add(b);
|
||||
const candId = `semantic:${b}`;
|
||||
const key = `${fn}||${candId}`;
|
||||
if (!cellMap[key]) cellMap[key] = e;
|
||||
}
|
||||
});
|
||||
|
||||
const fixedCands = [0, 1, 2, 3].map(n => ({
|
||||
id: `fixed_size:±${n}`,
|
||||
family: 'fixed_size',
|
||||
label: `fixed_size ±${n}/${n}`,
|
||||
short: `±${n}`,
|
||||
level: n,
|
||||
}));
|
||||
const semanticCands = [...semBounds].sort().map(b => ({
|
||||
id: `semantic:${b}`,
|
||||
family: 'semantic',
|
||||
label: `semantic @ ${b}`,
|
||||
short: b,
|
||||
boundary: b,
|
||||
}));
|
||||
|
||||
function metricsFor(exp, family) {
|
||||
if (!exp) return null;
|
||||
return (exp.aggregate_metrics || {})[family] || null;
|
||||
}
|
||||
|
||||
function summarize(cands) {
|
||||
return cands.map(c => {
|
||||
const perDoc = {};
|
||||
const composites = [];
|
||||
const metricBags = {
|
||||
avg_context_relevance: [],
|
||||
avg_answer_similarity: [],
|
||||
avg_faithfulness: [],
|
||||
hallucination_rate: [],
|
||||
};
|
||||
let filled = 0;
|
||||
DECISION_DOC_FILENAMES.forEach(fn => {
|
||||
const exp = cellMap[`${fn}||${c.id}`];
|
||||
const m = metricsFor(exp, c.family);
|
||||
const score = decisionComposite(m);
|
||||
perDoc[fn] = { exp, metrics: m, composite: score };
|
||||
if (score != null) {
|
||||
filled += 1;
|
||||
composites.push(score);
|
||||
Object.keys(metricBags).forEach(k => {
|
||||
if (m && m[k] != null) metricBags[k].push(m[k]);
|
||||
});
|
||||
}
|
||||
});
|
||||
return {
|
||||
...c,
|
||||
perDoc,
|
||||
filled,
|
||||
totalDocs: DECISION_DOC_FILENAMES.length,
|
||||
meanComposite: meanOf(composites),
|
||||
meanMetrics: {
|
||||
avg_context_relevance: meanOf(metricBags.avg_context_relevance),
|
||||
avg_answer_similarity: meanOf(metricBags.avg_answer_similarity),
|
||||
avg_faithfulness: meanOf(metricBags.avg_faithfulness),
|
||||
hallucination_rate: meanOf(metricBags.hallucination_rate),
|
||||
},
|
||||
wins: 0,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
const fixedSummaries = summarize(fixedCands);
|
||||
const semanticSummaries = summarize(semanticCands);
|
||||
|
||||
function assignWins(summaries) {
|
||||
DECISION_DOC_FILENAMES.forEach(fn => {
|
||||
let best = null;
|
||||
let bestScore = -Infinity;
|
||||
summaries.forEach(s => {
|
||||
const sc = s.perDoc[fn]?.composite;
|
||||
if (sc == null) return;
|
||||
if (sc > bestScore) {
|
||||
bestScore = sc;
|
||||
best = s;
|
||||
}
|
||||
});
|
||||
if (best) best.wins += 1;
|
||||
});
|
||||
}
|
||||
assignWins(fixedSummaries);
|
||||
assignWins(semanticSummaries);
|
||||
|
||||
function autoPick(summaries) {
|
||||
if (!summaries.length) return null;
|
||||
return [...summaries].sort((a, b) => {
|
||||
const ma = a.meanComposite == null ? -1 : a.meanComposite;
|
||||
const mb = b.meanComposite == null ? -1 : b.meanComposite;
|
||||
if (mb !== ma) return mb - ma;
|
||||
if (b.wins !== a.wins) return b.wins - a.wins;
|
||||
return a.id.localeCompare(b.id);
|
||||
})[0];
|
||||
}
|
||||
|
||||
return {
|
||||
cellMap,
|
||||
fixedSummaries,
|
||||
semanticSummaries,
|
||||
autoFixed: autoPick(fixedSummaries),
|
||||
autoSemantic: autoPick(semanticSummaries),
|
||||
allCandidates: [...fixedSummaries, ...semanticSummaries],
|
||||
};
|
||||
}
|
||||
|
||||
function DecisionTab({ addToast }) {
|
||||
const [experiments, setExperiments] = useState([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [corpusId, setCorpusId] = useState('text-embedding-3-large');
|
||||
const [embModels, setEmbModels] = useState([]);
|
||||
const [excludedIds, setExcludedIds] = useState([]);
|
||||
const [overrideFixed, setOverrideFixed] = useState(null);
|
||||
const [overrideSemantic, setOverrideSemantic] = useState(null);
|
||||
|
||||
const refresh = useCallback(() => {
|
||||
setLoading(true);
|
||||
Promise.all([
|
||||
api('/experiments?limit=500'),
|
||||
api('/admin/embedding-models'),
|
||||
]).then(([ex, emb]) => {
|
||||
setExperiments(ex?.items || []);
|
||||
setEmbModels(emb?.models || []);
|
||||
if (emb?.corpus_id) {
|
||||
setCorpusId(prev => prev || emb.corpus_id);
|
||||
}
|
||||
}).catch(() => addToast('Failed to load Decision Board data', 'error'))
|
||||
.finally(() => setLoading(false));
|
||||
}, [addToast]);
|
||||
|
||||
useEffect(() => { refresh(); }, [refresh]);
|
||||
|
||||
const board = React.useMemo(
|
||||
() => buildDecisionBoard(experiments, corpusId, excludedIds),
|
||||
[experiments, corpusId, excludedIds]
|
||||
);
|
||||
|
||||
const winnerFixed = (overrideFixed && board.fixedSummaries.find(c => c.id === overrideFixed))
|
||||
|| board.autoFixed;
|
||||
const winnerSemantic = (overrideSemantic && board.semanticSummaries.find(c => c.id === overrideSemantic))
|
||||
|| board.autoSemantic;
|
||||
|
||||
const excludeId = (id) => {
|
||||
if (!id) return;
|
||||
setExcludedIds(prev => prev.includes(id) ? prev : [...prev, id]);
|
||||
};
|
||||
const unexcludeId = (id) => setExcludedIds(prev => prev.filter(x => x !== id));
|
||||
|
||||
const fsFilled = board.fixedSummaries.reduce((a, c) => a + c.filled, 0);
|
||||
const fsTotal = board.fixedSummaries.length * DECISION_DOC_FILENAMES.length;
|
||||
const semFilled = board.semanticSummaries.reduce((a, c) => a + c.filled, 0);
|
||||
const semTotal = board.semanticSummaries.length * DECISION_DOC_FILENAMES.length;
|
||||
|
||||
// Stage-2 head-to-head
|
||||
let duel = null;
|
||||
if (winnerFixed && winnerSemantic) {
|
||||
let fsWins = 0, semWins = 0, ties = 0;
|
||||
const perDoc = DECISION_DOC_FILENAMES.map(fn => {
|
||||
const a = winnerFixed.perDoc[fn]?.composite;
|
||||
const b = winnerSemantic.perDoc[fn]?.composite;
|
||||
let winner = '—';
|
||||
if (a != null && b != null) {
|
||||
if (a > b) { fsWins += 1; winner = 'fixed'; }
|
||||
else if (b > a) { semWins += 1; winner = 'semantic'; }
|
||||
else { ties += 1; winner = 'tie'; }
|
||||
}
|
||||
return { fn, a, b, winner };
|
||||
});
|
||||
const recommend = (winnerFixed.meanComposite || 0) >= (winnerSemantic.meanComposite || 0)
|
||||
? winnerFixed : winnerSemantic;
|
||||
// Prefer win-count if mean close? stick to mean primary with wins as display
|
||||
duel = { fsWins, semWins, ties, perDoc, recommend };
|
||||
}
|
||||
|
||||
const renderCandCard = (c, selected, onSelect, familyAutoId) =>
|
||||
React.createElement('div', {
|
||||
key: c.id,
|
||||
className: `decision-cand${selected ? ' selected' : ''}`,
|
||||
onClick: () => onSelect(selected ? null : (c.id === familyAutoId ? null : c.id)),
|
||||
title: 'Click to override family winner; click selected again to return to auto',
|
||||
},
|
||||
React.createElement('div', { className: 'cand-title', style: { display: 'flex', justifyContent: 'space-between', gap: 8 } },
|
||||
React.createElement('span', null,
|
||||
React.createElement('input', {
|
||||
type: 'radio',
|
||||
checked: !!selected,
|
||||
readOnly: true,
|
||||
style: { marginRight: 8 },
|
||||
}),
|
||||
c.label,
|
||||
c.id === familyAutoId
|
||||
? React.createElement('span', { className: 'badge badge-accent', style: { marginLeft: 8 } }, 'auto')
|
||||
: null
|
||||
),
|
||||
React.createElement('span', { style: { color: 'var(--accent)', fontVariantNumeric: 'tabular-nums' } },
|
||||
decisionFmt(c.meanComposite))
|
||||
),
|
||||
React.createElement('div', { className: 'decision-metrics' },
|
||||
React.createElement('span', null, 'Wins ', React.createElement('strong', null, `${c.wins}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Coverage ', React.createElement('strong', null, `${c.filled}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Context ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_context_relevance))),
|
||||
React.createElement('span', null, 'Similarity ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_answer_similarity))),
|
||||
React.createElement('span', null, 'Faithfulness ', React.createElement('strong', null, decisionFmt(c.meanMetrics.avg_faithfulness))),
|
||||
React.createElement('span', null, 'Halluc. ', React.createElement('strong', null, decisionPct(c.meanMetrics.hallucination_rate)))
|
||||
)
|
||||
);
|
||||
|
||||
const duelPanel = (c, side) => {
|
||||
if (!c) {
|
||||
return React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'text-muted' }, `No ${side} winner yet`));
|
||||
}
|
||||
const isRec = duel && duel.recommend && duel.recommend.id === c.id;
|
||||
return React.createElement('div', {
|
||||
className: 'card',
|
||||
style: {
|
||||
margin: 0,
|
||||
borderColor: isRec ? 'var(--accent)' : undefined,
|
||||
boxShadow: isRec ? '0 0 0 1px rgba(234,179,8,0.35)' : undefined,
|
||||
},
|
||||
},
|
||||
React.createElement('div', { className: 'flex-between mb-2' },
|
||||
React.createElement('div', { className: 'card-title mb-0' }, c.label),
|
||||
isRec ? React.createElement('span', { className: 'badge badge-success' }, 'Recommended') : null
|
||||
),
|
||||
React.createElement('div', { style: { fontSize: 28, fontWeight: 700, color: 'var(--accent)', marginBottom: 8 } },
|
||||
decisionFmt(c.meanComposite)),
|
||||
React.createElement('div', { className: 'decision-metrics', style: { marginBottom: 8 } },
|
||||
React.createElement('span', null, 'Doc wins vs other ', React.createElement('strong', null,
|
||||
side === 'fixed' ? (duel ? duel.fsWins : '—') : (duel ? duel.semWins : '—'))),
|
||||
React.createElement('span', null, 'Family wins ', React.createElement('strong', null, `${c.wins}/${c.totalDocs}`)),
|
||||
React.createElement('span', null, 'Coverage ', React.createElement('strong', null, `${c.filled}/${c.totalDocs}`))
|
||||
),
|
||||
React.createElement('table', { className: 'cmp-table' },
|
||||
React.createElement('tbody', null,
|
||||
[['Context Relevance', c.meanMetrics.avg_context_relevance, false],
|
||||
['Answer Similarity', c.meanMetrics.avg_answer_similarity, false],
|
||||
['Faithfulness', c.meanMetrics.avg_faithfulness, false],
|
||||
['Hallucination Rate', c.meanMetrics.hallucination_rate, true]].map(([label, val, isPct]) =>
|
||||
React.createElement('tr', { key: label },
|
||||
React.createElement('td', null, label),
|
||||
React.createElement('td', { style: { fontWeight: 600 } }, isPct ? decisionPct(val) : decisionFmt(val))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
};
|
||||
|
||||
return React.createElement('div', null,
|
||||
React.createElement('div', { className: 'flex-between mb-2' },
|
||||
React.createElement('h1', { style: { color: 'var(--text-heading)', margin: 0, fontSize: '22px' } }, 'Decision Board'),
|
||||
React.createElement('button', { className: 'btn btn-secondary btn-sm', onClick: refresh, disabled: loading },
|
||||
loading ? React.createElement('span', { className: 'spinner' }) : '↻ Refresh')
|
||||
),
|
||||
React.createElement('p', { className: 'text-sm text-muted', style: { marginBottom: 16, maxWidth: 720 } },
|
||||
'Two-stage final selection: pick the best fixed_size Neighbor level and best semantic Boundary, then compare those winners across the 10-doc evaluation set. Mean composite ranks stage 1; win-counts are shown; click a Candidate to override.'
|
||||
),
|
||||
|
||||
// Header controls
|
||||
React.createElement('div', { className: 'card mb-4' },
|
||||
React.createElement('div', { className: 'row mb-0', style: { alignItems: 'flex-end' } },
|
||||
React.createElement(EmbeddingModelSelect, {
|
||||
label: 'Corpus filter',
|
||||
value: corpusId,
|
||||
onChange: (v) => { setCorpusId(v); setOverrideFixed(null); setOverrideSemantic(null); },
|
||||
models: embModels,
|
||||
role: 'corpus',
|
||||
title: 'Only single-strategy Experiments under this Corpus Embedding Model',
|
||||
style: { maxWidth: 360 },
|
||||
}),
|
||||
React.createElement('div', { className: 'col' },
|
||||
React.createElement('div', { className: 'text-sm text-muted' }, 'Coverage'),
|
||||
React.createElement('div', { style: { fontWeight: 600, color: 'var(--text-heading)', marginTop: 6 } },
|
||||
`fixed_size ${fsFilled}/${fsTotal || 40} · semantic ${semFilled}/${semTotal || '—'}`)
|
||||
)
|
||||
),
|
||||
excludedIds.length > 0 && React.createElement('div', { style: { marginTop: 12 } },
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-1' }, 'Excluded Experiments (next-newest fills the cell)'),
|
||||
React.createElement('div', { style: { display: 'flex', flexWrap: 'wrap', gap: 6 } },
|
||||
excludedIds.map(id =>
|
||||
React.createElement('button', {
|
||||
key: id,
|
||||
className: 'btn btn-secondary btn-sm',
|
||||
onClick: () => unexcludeId(id),
|
||||
title: 'Click to restore',
|
||||
}, `✕ ${id.substring(0, 8)}…`)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
|
||||
loading && React.createElement('div', { style: { textAlign: 'center', padding: 24 } },
|
||||
React.createElement('span', { className: 'spinner' })),
|
||||
|
||||
!loading && React.createElement(React.Fragment, null,
|
||||
// Stage 1
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Stage 1 — Best Candidate per family'),
|
||||
React.createElement('div', { className: 'decision-grid mb-4' },
|
||||
React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'card-title' }, 'fixed_size (Neighbor Expansion)'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
'Auto winner: ', board.autoFixed ? board.autoFixed.label : '—',
|
||||
overrideFixed ? ' · override active' : ''),
|
||||
board.fixedSummaries.map(c => renderCandCard(
|
||||
c,
|
||||
winnerFixed && winnerFixed.id === c.id,
|
||||
setOverrideFixed,
|
||||
board.autoFixed?.id
|
||||
))
|
||||
),
|
||||
React.createElement('div', { className: 'card', style: { margin: 0 } },
|
||||
React.createElement('div', { className: 'card-title' }, 'semantic (Boundary Embedding Model)'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
'Auto winner: ', board.autoSemantic ? board.autoSemantic.label : '—',
|
||||
overrideSemantic ? ' · override active' : ''),
|
||||
board.semanticSummaries.length === 0
|
||||
? React.createElement('div', { className: 'empty-state' }, 'No semantic single-strategy Experiments for this Corpus')
|
||||
: board.semanticSummaries.map(c => renderCandCard(
|
||||
c,
|
||||
winnerSemantic && winnerSemantic.id === c.id,
|
||||
setOverrideSemantic,
|
||||
board.autoSemantic?.id
|
||||
))
|
||||
)
|
||||
),
|
||||
|
||||
// Stage 2
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Stage 2 — Family showdown'),
|
||||
React.createElement('div', { className: 'decision-duel mb-4' },
|
||||
duelPanel(winnerFixed, 'fixed'),
|
||||
React.createElement('div', { className: 'decision-vs' }, 'vs'),
|
||||
duelPanel(winnerSemantic, 'semantic')
|
||||
),
|
||||
duel && React.createElement('div', { className: 'card mb-4' },
|
||||
React.createElement('div', { className: 'card-title' }, 'Head-to-head by document'),
|
||||
React.createElement('div', { className: 'text-sm text-muted mb-2' },
|
||||
`fixed_size wins ${duel.fsWins} · semantic wins ${duel.semWins} · ties ${duel.ties}`),
|
||||
React.createElement('table', { className: 'cmp-table' },
|
||||
React.createElement('thead', null,
|
||||
React.createElement('tr', null,
|
||||
React.createElement('th', null, 'Document'),
|
||||
React.createElement('th', null, winnerFixed?.short || 'fixed'),
|
||||
React.createElement('th', null, winnerSemantic?.short || 'semantic'),
|
||||
React.createElement('th', null, 'Winner')
|
||||
)
|
||||
),
|
||||
React.createElement('tbody', null,
|
||||
duel.perDoc.map(row =>
|
||||
React.createElement('tr', { key: row.fn },
|
||||
React.createElement('td', null, row.fn),
|
||||
React.createElement('td', { style: { textAlign: 'center', fontVariantNumeric: 'tabular-nums' } }, decisionFmt(row.a)),
|
||||
React.createElement('td', { style: { textAlign: 'center', fontVariantNumeric: 'tabular-nums' } }, decisionFmt(row.b)),
|
||||
React.createElement('td', null,
|
||||
row.winner === 'fixed' ? React.createElement('span', { className: 'badge badge-accent' }, 'fixed_size')
|
||||
: row.winner === 'semantic' ? React.createElement('span', { className: 'badge badge-success' }, 'semantic')
|
||||
: row.winner === 'tie' ? React.createElement('span', { className: 'badge' }, 'tie')
|
||||
: React.createElement('span', { className: 'text-muted' }, '—')
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
|
||||
// Matrix
|
||||
React.createElement('h2', { style: { fontSize: 16, color: 'var(--text-heading)', marginBottom: 10 } }, 'Per-document matrix'),
|
||||
React.createElement('div', { className: 'card', style: { paddingBottom: 14 } },
|
||||
React.createElement(DecisionHeatLegend),
|
||||
React.createElement('div', { className: 'decision-matrix-wrap' },
|
||||
React.createElement('table', { className: 'decision-matrix' },
|
||||
React.createElement('thead', null,
|
||||
React.createElement('tr', null,
|
||||
React.createElement('th', { className: 'sticky-col' }, 'Document'),
|
||||
board.allCandidates.map(c =>
|
||||
React.createElement('th', {
|
||||
key: c.id,
|
||||
className: (winnerFixed && c.id === winnerFixed.id) || (winnerSemantic && c.id === winnerSemantic.id)
|
||||
? 'col-duel' : undefined,
|
||||
title: c.label,
|
||||
}, c.family === 'fixed_size' ? c.short : (c.short || '').substring(0, 14))
|
||||
)
|
||||
)
|
||||
),
|
||||
React.createElement('tbody', null,
|
||||
DECISION_DOC_FILENAMES.map(fn => {
|
||||
const rowBest = decisionFamilyBestIds(board.allCandidates, c => c.perDoc[fn]?.composite);
|
||||
return React.createElement('tr', { key: fn },
|
||||
React.createElement('td', { className: 'sticky-col' }, fn),
|
||||
board.allCandidates.map(c => {
|
||||
const cell = c.perDoc[fn];
|
||||
const score = cell?.composite;
|
||||
const exp = cell?.exp;
|
||||
const mark = c.id === rowBest.fixed ? 'fs' : (c.id === rowBest.semantic ? 'sem' : null);
|
||||
return React.createElement('td', {
|
||||
key: c.id,
|
||||
className: 'decision-heat',
|
||||
style: { ...decisionHeatStyle(score), cursor: exp ? 'pointer' : 'default' },
|
||||
title: exp
|
||||
? `${c.label} · ${fn}\ncomposite=${decisionFmt(score)}${mark ? `\nbest ${mark === 'fs' ? 'fixed_size' : 'semantic'} in row` : ''}\nid=${exp.id}\nClick: report · Shift+click: exclude`
|
||||
: `${c.label} · ${fn}: missing`,
|
||||
onClick: (e) => {
|
||||
if (!exp) return;
|
||||
if (e.shiftKey) {
|
||||
excludeId(exp.id);
|
||||
addToast(`Excluded ${exp.id.substring(0, 8)}…`, 'success');
|
||||
return;
|
||||
}
|
||||
window.open(`/benchmarks/${exp.id}/report`, '_blank');
|
||||
},
|
||||
},
|
||||
mark
|
||||
? React.createElement('span', {
|
||||
className: `best-mark ${mark}`,
|
||||
'aria-label': mark === 'fs' ? 'Best fixed_size in row' : 'Best semantic in row',
|
||||
}, mark === 'fs' ? 'F' : 'S')
|
||||
: null,
|
||||
decisionFmt(score)
|
||||
);
|
||||
})
|
||||
);
|
||||
}),
|
||||
(() => {
|
||||
const meanBest = decisionFamilyBestIds(board.allCandidates, c => c.meanComposite);
|
||||
return React.createElement('tr', { key: '_mean', className: 'mean-row' },
|
||||
React.createElement('td', { className: 'sticky-col' }, 'Mean'),
|
||||
board.allCandidates.map(c => {
|
||||
const mark = c.id === meanBest.fixed ? 'fs' : (c.id === meanBest.semantic ? 'sem' : null);
|
||||
return React.createElement('td', {
|
||||
key: c.id,
|
||||
className: 'decision-heat',
|
||||
style: decisionHeatStyle(c.meanComposite),
|
||||
},
|
||||
mark
|
||||
? React.createElement('span', {
|
||||
className: `best-mark ${mark}`,
|
||||
'aria-label': mark === 'fs' ? 'Best fixed_size mean' : 'Best semantic mean',
|
||||
}, mark === 'fs' ? 'F' : 'S')
|
||||
: null,
|
||||
decisionFmt(c.meanComposite)
|
||||
);
|
||||
})
|
||||
);
|
||||
})(),
|
||||
React.createElement('tr', { key: '_wins', className: 'wins-row' },
|
||||
React.createElement('td', { className: 'sticky-col' }, 'Wins'),
|
||||
board.allCandidates.map(c =>
|
||||
React.createElement('td', { key: c.id }, `${c.wins}/${c.totalDocs}`)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
React.createElement('div', { className: 'text-sm text-muted', style: { marginTop: 12 } },
|
||||
'Click a cell to open its Experiment report. Shift+click to exclude. ',
|
||||
React.createElement('span', { className: 'best-mark fs' }, 'F'),
|
||||
' / ',
|
||||
React.createElement('span', { className: 'best-mark sem' }, 'S'),
|
||||
' mark the best score in each family per row.'
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// -- Tab: Admin ---------------------------------------------
|
||||
function AdminTab({ addToast, documents, strategies, setActiveTab, setQuestionsFile }) {
|
||||
// Collapse state for each section
|
||||
@@ -2804,6 +3569,7 @@ function App() {
|
||||
{ id: 'pdf', label: 'PDF' },
|
||||
{ id: 'query', label: 'Query' },
|
||||
{ id: 'benchmarks', label: 'Benchmarks' },
|
||||
{ id: 'decision', label: 'Decision' },
|
||||
{ id: 'admin', label: 'Admin' },
|
||||
];
|
||||
|
||||
@@ -2818,6 +3584,7 @@ function App() {
|
||||
documents, strategies, addToast, formatFilter: 'word' });
|
||||
case 'benchmarks': return React.createElement(BenchmarksTab, {
|
||||
documents, strategies, addToast, questionsFile, formatFilter: 'word' });
|
||||
case 'decision': return React.createElement(DecisionTab, { addToast });
|
||||
case 'admin': return React.createElement(AdminTab, {
|
||||
addToast, documents, strategies, setActiveTab, setQuestionsFile });
|
||||
default: return null;
|
||||
|
||||
@@ -88,6 +88,19 @@ CREATE TABLE IF NOT EXISTS app_settings (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
job_type TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
payload TEXT NOT NULL DEFAULT '{}',
|
||||
result TEXT,
|
||||
error TEXT,
|
||||
progress TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL,
|
||||
started_at TEXT,
|
||||
completed_at TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
@@ -149,6 +162,21 @@ def _migrate_schema(conn: sqlite3.Connection) -> None:
|
||||
(LEGACY_CLOUD_MODEL_ID, Provider.CLOUD.value),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
job_type TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
payload TEXT NOT NULL DEFAULT '{}',
|
||||
result TEXT,
|
||||
error TEXT,
|
||||
progress TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL,
|
||||
started_at TEXT,
|
||||
completed_at TEXT
|
||||
)"""
|
||||
)
|
||||
|
||||
|
||||
def get_app_setting(key: str) -> str | None:
|
||||
"""Read a persisted app setting value."""
|
||||
@@ -408,9 +436,11 @@ def get_experiment(experiment_id: str) -> dict[str, Any] | None:
|
||||
|
||||
|
||||
def list_experiments(
|
||||
*, document_id: str | None = None, offset: int = 0, limit: int = 50
|
||||
*, document_id: str | None = None, offset: int = 0, limit: int = 200
|
||||
) -> dict[str, Any]:
|
||||
"""List experiments, optionally filtered by document."""
|
||||
limit = max(1, min(int(limit), 500))
|
||||
offset = max(0, int(offset))
|
||||
conn = _connect()
|
||||
try:
|
||||
if document_id:
|
||||
@@ -443,12 +473,114 @@ def delete_experiment(experiment_id: str) -> bool:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ── Job CRUD ───────────────────────────────────────────────────────
|
||||
|
||||
def create_job(*, job_type: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Insert a pending background job."""
|
||||
job_id = _new_id()
|
||||
conn = _connect()
|
||||
try:
|
||||
conn.execute(
|
||||
"""INSERT INTO jobs
|
||||
(id, job_type, status, payload, progress, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
job_id,
|
||||
job_type,
|
||||
"pending",
|
||||
json.dumps(payload),
|
||||
json.dumps({}),
|
||||
_now(),
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
return get_job(job_id) # type: ignore[return-value]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_job(job_id: str) -> dict[str, Any] | None:
|
||||
"""Fetch a job by ID."""
|
||||
conn = _connect()
|
||||
try:
|
||||
row = conn.execute("SELECT * FROM jobs WHERE id = ?", (job_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_dict(row)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def update_job(job_id: str, **fields: Any) -> None:
|
||||
"""Update job fields (status, result, error, progress, timestamps)."""
|
||||
if not fields:
|
||||
return
|
||||
|
||||
json_fields = {"payload", "result", "progress"}
|
||||
sets: list[str] = []
|
||||
values: list[Any] = []
|
||||
for key, value in fields.items():
|
||||
if key in json_fields and value is not None and not isinstance(value, str):
|
||||
value = json.dumps(value)
|
||||
sets.append(f"{key} = ?")
|
||||
values.append(value)
|
||||
|
||||
values.append(job_id)
|
||||
conn = _connect()
|
||||
try:
|
||||
conn.execute(
|
||||
f"UPDATE jobs SET {', '.join(sets)} WHERE id = ?",
|
||||
values,
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def list_jobs(
|
||||
*,
|
||||
job_type: str | None = None,
|
||||
status: str | None = None,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
) -> dict[str, Any]:
|
||||
"""List jobs with optional filters, newest first."""
|
||||
limit = max(1, min(int(limit), 200))
|
||||
offset = max(0, int(offset))
|
||||
conn = _connect()
|
||||
try:
|
||||
clauses: list[str] = []
|
||||
params: list[Any] = []
|
||||
if job_type:
|
||||
clauses.append("job_type = ?")
|
||||
params.append(job_type)
|
||||
if status:
|
||||
clauses.append("status = ?")
|
||||
params.append(status)
|
||||
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
|
||||
total = conn.execute(
|
||||
f"SELECT COUNT(*) FROM jobs {where}", params
|
||||
).fetchone()[0]
|
||||
rows = conn.execute(
|
||||
f"SELECT * FROM jobs {where} ORDER BY created_at DESC LIMIT ? OFFSET ?",
|
||||
[*params, limit, offset],
|
||||
).fetchall()
|
||||
return {
|
||||
"items": [_row_to_dict(r) for r in rows],
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ── Internal helpers ───────────────────────────────────────────────
|
||||
|
||||
_JSON_FIELDS = {"chunk_counts", "retrieved_chunks", "expansion_tree", "latency_breakdown",
|
||||
"token_usage", "document_tree", "benchmark_config",
|
||||
"questions", "per_question", "aggregate_metrics",
|
||||
"strategies_used"}
|
||||
"strategies_used", "payload", "result", "progress"}
|
||||
|
||||
|
||||
def _row_to_dict(row: sqlite3.Row) -> dict[str, Any]:
|
||||
|
||||
Reference in New Issue
Block a user