docs: add high- and low-level design

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/HLD.md

@@ -0,0 +1,280 @@
+# High-Level Design (HLD)
+
+RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
+
+**Audience:** architects, tech leads, new engineers  
+**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
+
+---
+
+## 1. Purpose
+
+Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
+
+Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
+
+---
+
+## 2. Goals and Non-Goals
+
+### Goals
+
+| Goal | Notes |
+|------|--------|
+| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
+| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
+| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
+| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
+| Failure isolation | One Strategy failing during process does not discard others |
+
+### Non-Goals (v1)
+
+| Out of scope | See |
+|--------------|-----|
+| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
+| Multi-tenant auth | Single-operator local tool |
+| Production-scale concurrency | SQLite + single process |
+| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
+
+---
+
+## 3. System Context
+
+```
+┌──────────────┐     HTTP/REST      ┌─────────────────────────────────────┐
+│  Operator    │◄──────────────────►│  FastAPI App (RAG Chunking Benchmarker)│
+│  (Browser)   │   Dashboard /app   │                                     │
+└──────────────┘                    │  Documents · Chunking · Benchmarking │
+                                    │  Admin · Static Dashboard           │
+                                    └───────────┬─────────────────────────┘
+                                                │
+              ┌─────────────────────────────────┼─────────────────────────────────┐
+              ▼                                 ▼                                 ▼
+     ┌────────────────┐              ┌────────────────┐              ┌────────────────┐
+     │     SQLite     │              │     Qdrant     │              │  LLM / Embed   │
+     │  metadata,     │              │  Model Corpus  │              │  OpenAI Cloud  │
+     │  Experiments,  │              │  vectors +     │              │  + Ollama Local│
+     │  app_settings  │              │  chunk payload │              └────────────────┘
+     └────────────────┘              └────────────────┘
+```
+
+**External dependencies**
+
+| System | Role |
+|--------|------|
+| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
+| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
+| Qdrant | Vector search per Strategy × Corpus Embedding Model |
+| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
+
+---
+
+## 4. Logical Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                         Presentation Layer                               │
+│  src/static/index.html — single-file React Dashboard (CDN, no build)     │
+│  Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin    │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │ REST
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                         API Layer (FastAPI)                              │
+│  /documents*   /queries*   /benchmarks*   /experiments*   /admin/*       │
+│  /strategies   /app (StaticFiles)                                        │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                       Domain Services                                    │
+│  documents/service   chunking/service   query_service   benchmark_service│
+│  evaluation          admin/service      report                           │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                       Foundation                                         │
+│  core/config · models · exceptions · dependencies                        │
+│  chunking/base · embedding · embedding_models · strategies/*             │
+│  documents/parser · pdf_parser                                           │
+│  storage/sqlite · storage/qdrant                                         │
+└──────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 5. Major Components
+
+| Component | Responsibility | Key modules |
+|-----------|----------------|-------------|
+| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
+| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
+| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
+| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
+| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
+| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
+| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
+| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
+
+---
+
+## 6. Domain Concepts (summary)
+
+Full glossary: [CONTEXT.md](../CONTEXT.md).
+
+| Term | Meaning |
+|------|---------|
+| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
+| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
+| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
+| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
+| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
+| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
+| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
+| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
+| **Decision Board** | Two-stage selection over the fixed evaluation document set |
+
+---
+
+## 7. End-to-End Flows
+
+### 7.1 Ingest and process
+
+```
+Upload (.docx/.doc/.pdf)
+  → Text-layer Gate (PDF) / parse DOCX
+  → Heading Reconstruction (PDF) → DocumentTree + markdown
+  → SQLite documents
+
+Process(strategies, boundary?, corpus?)
+  → For each Strategy (isolated failures):
+       Boundary embeds (semantic*) → Strategy.chunk()
+       → Corpus embeds → Qdrant Model Corpus upsert
+  → Update chunk_counts + last_* embedding provenance on document
+```
+
+### 7.2 Query
+
+```
+POST /queries
+  → Resolve Corpus Embedding Model
+  → Embed question (Corpus)
+  → Qdrant top-k (document filter)
+  → Neighbor Expansion if fixed_size (±P/N)
+  → Build context (parent fetch for semantic_parent_child)
+  → LLM answer → SQLite queries (+ expansion_tree)
+```
+
+### 7.3 Experiment
+
+```
+POST /benchmarks
+  → Load questions JSON
+  → For each question × Strategy: Query + evaluate_single
+  → Aggregate metrics → SQLite experiments
+  → Optional HTML report (managerial / technical)
+```
+
+### 7.4 Decision
+
+```
+Dashboard Decision Tab
+  → Load Experiments filtered by Corpus Embedding Model
+  → Discover Strategy Candidates (newest per doc × Candidate)
+  → Stage 1: best fixed_size ±N vs best semantic@Boundary
+  → Stage 2: compare winners + per-document breakdown
+```
+
+---
+
+## 8. Data Architecture
+
+### 8.1 Responsibility split
+
+| Store | Owns |
+|-------|------|
+| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
+| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
+| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
+
+### 8.2 Model Corpus naming
+
+```
+{strategy}__{sanitized_model_id}_collection
+```
+
+Example: `fixed_size__text-embedding-3-large_collection`
+
+Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
+
+### 8.3 Provenance
+
+Experiments store:
+
+- `embedding_model_id` / `embedding_provider` — Corpus
+- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
+- Neighbor Expansion levels in `benchmark_config`
+
+Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
+
+---
+
+## 9. Deployment View
+
+```
+┌─────────────────────────────────────────┐
+│  Host process                           │
+│  uvicorn src.main:app                   │
+│  └── FastAPI + Dashboard static         │
+│                                         │
+│  data/chunking_benchmark.db             │
+│  .env (secrets + knobs)                 │
+└───────────────┬─────────────────────────┘
+                │
+    ┌───────────┴───────────┐
+    ▼                       ▼
+ Qdrant (:6333)      OpenAI / Ollama
+```
+
+Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
+
+---
+
+## 10. Cross-Cutting Concerns
+
+| Concern | Approach |
+|---------|----------|
+| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
+| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
+| Logging | Request middleware + structured stage logs in query/benchmark |
+| CORS | Permissive (`*`) for local Dashboard development |
+| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
+| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
+
+---
+
+## 11. Key Architectural Decisions
+
+| ADR | Decision |
+|-----|----------|
+| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
+| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
+| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
+| 0021 | Always scope Qdrant collections by Embedding Model id |
+| 0022 | Per-model `semantic_threshold` |
+| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
+| 0024 | Boundary vs Corpus Embedding Model roles |
+| 0025 | Retrieval Inspect in Benchmarks Tab |
+| 0026 | Decision Board two-stage Candidate selection |
+
+---
+
+## 12. Related Documents
+
+| Doc | Role |
+|-----|------|
+| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
+| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
+| [data-flow.md](data-flow.md) | Pipeline detail |
+| [api-reference.md](api-reference.md) | HTTP contracts |
+| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
+| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
+| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |

docs/LLD.md

@@ -0,0 +1,571 @@
+# Low-Level Design (LLD)
+
+RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.
+
+**Audience:** implementers  
+**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)
+
+---
+
+## 1. Package Map
+
+```
+src/
+├── main.py                      # App factory, middleware, router mount
+├── static/index.html            # Dashboard (React + Babel CDN)
+├── core/
+│   ├── config.py                # Settings (.env)
+│   ├── models.py                # Chunk, DocumentTree, StrategyName
+│   ├── exceptions.py            # Domain errors + handlers
+│   └── dependencies.py          # OpenAI / Ollama / Qdrant singletons
+├── storage/
+│   ├── sqlite.py                # documents, queries, experiments, app_settings
+│   └── qdrant.py                # Model Corpus CRUD + search
+├── documents/
+│   ├── parser.py                # DOCX / .doc (+ LibreOffice)
+│   ├── pdf_parser.py            # Text PDF + Text-layer Gate
+│   ├── heading_heuristics.py    # PDF Heading Reconstruction helpers
+│   ├── service.py               # Upload / process / delete
+│   ├── routes.py
+│   └── models.py                # ProcessRequest, DocumentResponse, …
+├── chunking/
+│   ├── base.py                  # ChunkingStrategy ABC, tokens, sentences
+│   ├── embedding.py             # Boundary/Corpus resolve + embed_texts
+│   ├── embedding_models.py      # Registry, thresholds, task prefixes
+│   ├── service.py               # run_strategies orchestration
+│   └── strategies/
+│       ├── fixed_size.py
+│       ├── recursive.py
+│       ├── semantic.py
+│       ├── contextual_retrieval.py
+│       └── semantic_parent_child.py
+├── benchmarking/
+│   ├── query_service.py         # Query + Neighbor Expansion
+│   ├── benchmark_service.py     # Experiment runner + cost estimate
+│   ├── evaluation.py            # LLM-as-Judge
+│   ├── report.py                # HTML reports
+│   ├── routes.py
+│   └── models.py
+└── admin/
+    ├── routes.py                # /admin/*
+    └── service.py
+```
+
+---
+
+## 2. Application Bootstrap
+
+**Entry:** `create_app()` in `src/main.py`
+
+1. `init_db()` — create/migrate SQLite schema  
+2. Register CORS + `RequestLoggingMiddleware`  
+3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`  
+4. Routers: documents, benchmarking, admin  
+5. Mount `StaticFiles` at `/app` → `src/static/`
+
+```python
+# Singletons (lru_cache)
+get_openai_client()   # Cloud LLM + cloud embeds
+get_ollama_client()   # OpenAI-compatible client → Ollama base URL
+get_qdrant_client()
+```
+
+---
+
+## 3. Configuration
+
+`Settings` (`src/core/config.py`) — selected fields:
+
+| Field | Default | Use |
+|-------|---------|-----|
+| `openai_api_key` | required | Cloud Provider |
+| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
+| `llm_model` | `gpt-4o-mini` | Answer + judge |
+| `ollama_base_url` | host URL | Local embeds |
+| `qdrant_url` | `http://localhost:6333` | Vectors |
+| `top_k` | `5` | Retrieval |
+| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults |
+| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
+| `semantic_threshold` | `0.3` | Fallback if model default missing |
+| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
+| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
+| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |
+
+---
+
+## 4. Core Domain Models
+
+### 4.1 StrategyName
+
+```python
+class StrategyName(str, Enum):
+    RECURSIVE = "recursive"
+    FIXED_SIZE = "fixed_size"
+    SEMANTIC = "semantic"
+    CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
+    SEMANTIC_PARENT_CHILD = "semantic_parent_child"
+```
+
+### 4.2 Chunk (unified)
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `document_name` | str | Source filename |
+| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
+| `strategy_name` | StrategyName | |
+| `chunk_index` | int | Document order (used by Neighbor Expansion) |
+| `text` | str | Stored in Qdrant; used for LLM context |
+| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
+| `parent_id` | str \| None | Parent-child Strategy |
+| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |
+
+`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).
+
+### 4.3 DocumentTree
+
+```
+DocumentTree
+  └── root: DocumentTreeNode
+        node_type: document | section | article | paragraph
+        text, heading, heading_level, children[]
+```
+
+Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.
+
+---
+
+## 5. Storage LLD
+
+### 5.1 SQLite schema
+
+**documents**
+
+| Column | Type | Purpose |
+|--------|------|---------|
+| `id` | TEXT PK | UUID hex |
+| `filename` | TEXT | Original name |
+| `parsed_text` | TEXT | Markdown for chunking |
+| `document_tree` | TEXT | JSON tree |
+| `chunk_counts` | TEXT | JSON `{strategy: count}` |
+| `last_corpus_embedding_model_id` | TEXT | Provenance |
+| `last_boundary_embedding_model_id` | TEXT | Provenance |
+| `created_at` | TEXT | ISO UTC |
+
+**queries**
+
+| Column | Purpose |
+|--------|---------|
+| `id`, `document_id`, `strategy_name` | Identity |
+| `question`, `answer` | Content |
+| `retrieved_chunks` | Flat LLM/eval list (JSON) |
+| `expansion_tree` | Per-hit neighbors (JSON) |
+| `latency_breakdown`, `token_usage` | Observability JSON |
+| `created_at` | |
+
+**experiments**
+
+| Column | Purpose |
+|--------|---------|
+| `id`, `document_id` | Identity |
+| `benchmark_config` | top_k, neighbor_prev/next, etc. |
+| `questions`, `per_question` | Inputs + per-cell results |
+| `aggregate_metrics` | Per-Strategy averages |
+| `strategies_used` | JSON list |
+| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
+| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
+| `created_at` | |
+
+**app_settings** — key/value for:
+
+- `corpus_embedding_model_id`, `boundary_embedding_model_id`
+- Legacy `active_embedding_model_id` (migrated into both roles)
+- `semantic_threshold:{model_id}` overrides
+
+Connection: WAL mode, foreign keys ON, one connection per call (no pool).
+
+### 5.2 Qdrant
+
+**Collection name**
+
+```python
+def collection_name(strategy, model_id) -> str:
+    return f"{strategy}__{sanitize(model_id)}_collection"
+# sanitize: replace : and / with -
+```
+
+**Point**
+
+| Field | Value |
+|-------|-------|
+| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
+| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
+| `payload` | ChunkMetadata fields |
+
+**Search**
+
+- Cosine distance  
+- Optional filter: `document_name == filename`  
+- Returns payload + score  
+
+**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.
+
+**Admin:** list/create/delete collections; wipe points.
+
+---
+
+## 6. Embedding Subsystem
+
+### 6.1 Registry (`embedding_models.py`)
+
+| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
+|----|----------|-----------|----------------------------|---------------|
+| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
+| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
+| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |
+
+`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.
+
+### 6.2 Role resolution (`embedding.py`)
+
+```
+resolve_corpus_model(optional_id)   → snapshot for process/query/experiment
+resolve_boundary_model(optional_id) → snapshot for semantic cuts
+
+get_corpus_embedding_model() / get_boundary_embedding_model()
+  → app_settings → registry → DEFAULT_CLOUD_MODEL_ID
+```
+
+Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.
+
+### 6.3 embed_texts / embed_single
+
+1. `apply_task_prefixes(texts, model, purpose)` if needed  
+2. Batch: OpenAI 2048 / Ollama 64  
+3. Route to `get_openai_client()` or `get_ollama_client()` by Provider  
+4. Raise `EmbeddingError` on failure  
+
+**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).
+
+---
+
+## 7. Documents LLD
+
+### 7.1 Upload
+
+```
+upload_document(filename, bytes)
+  → temp file → parse_document(path)
+  → db.save_document(parsed_text=markdown, document_tree=JSON)
+```
+
+**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).
+
+**PDF path (`pdf_parser.py`):**
+
+1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF  
+2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)  
+3. Table Flattening to sequential plain text  
+4. Emit markdown `#` / `##` + DocumentTree  
+
+### 7.2 Process
+
+```
+ProcessRequest:
+  strategies: list[StrategyName]  # default all five
+  boundary_model_id: str | None
+  corpus_model_id: str | None
+
+process_document → chunking.service.run_strategies(...)
+  → ProcessResponse(completed[], failed[], corpus_*, boundary_*)
+```
+
+### 7.3 Delete
+
+Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).
+
+---
+
+## 8. Chunking LLD
+
+### 8.1 Interface
+
+```python
+class ChunkingStrategy(ABC):
+    name: StrategyName
+
+    @abstractmethod
+    def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
+        ...
+```
+
+Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).
+
+### 8.2 Orchestrator (`run_strategies`)
+
+```
+corpus = resolve_corpus_model(corpus_model_id)
+boundary = resolve_boundary_model(...) if any(semantic*) else unused
+
+for strategy in strategies:
+  try:
+    ensure_collection(strategy, corpus.id, corpus.dimension)
+    chunks = _chunk_document(...)   # inject Boundary embeds if needed
+    texts = enriched_content or text
+    embeddings = embed_texts(texts, model=corpus, purpose="document")
+    upsert_chunks(chunks, embeddings, model_id=corpus.id)
+  except → failed[] (others continue)
+
+update_chunk_counts + update_process_embedding_provenance
+```
+
+### 8.3 Strategy algorithms (summary)
+
+| Strategy | Input units | Boundary | Output |
+|----------|-------------|----------|--------|
+| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
+| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
+| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
+| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
+| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |
+
+Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.
+
+**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.
+
+Deep dive: [strategy-technical-details.md](strategy-technical-details.md).
+
+---
+
+## 9. Query LLD
+
+### 9.1 `run_query` pipeline
+
+```
+1. Load document (filename for filter)
+2. Resolve Corpus model
+3. embed_single(question, purpose="query")
+4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
+5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
+6. _build_context (parent scroll for semantic_parent_child)
+7. _generate_answer (settings.llm_model, temp 0)
+8. db.save_query(...)
+```
+
+### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)
+
+**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.
+
+1. Build Expansion Tree nodes for each top-k hit (score order)  
+2. Collect needed `chunk_index` values: hit±prev/next (skip < 0)  
+3. Fetch missing indices via Qdrant  
+4. Fill `neighbors_prev` / `neighbors_next` per hit  
+5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`  
+6. Neighbors have `role="neighbor"`, `score=None`  
+
+Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).
+
+### 9.3 Parent-child context
+
+For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.
+
+---
+
+## 10. Benchmark LLD
+
+### 10.1 Questions format
+
+```json
+{
+  "questions": [
+    {
+      "id": "q1",
+      "question": "...",
+      "expected_answer": "...",
+      "category": "...",
+      "difficulty": "..."
+    }
+  ]
+}
+```
+
+Load from `questions_file` path or inline `questions` list.
+
+### 10.2 `run_benchmark`
+
+```
+embedding_model = resolve_corpus_model(corpus_model_id)
+boundary_id = doc.last_boundary_embedding_model_id  # if semantic in strategies
+
+for question in questions:
+  for strategy in strategies:
+    result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
+    scores = evaluate_single(question, retrieved_chunks, expected, generated)
+    append per_question row
+
+aggregate_metrics[strategy] = averages + hallucination_rate
+best_strategy = argmax (e.g. answer_similarity / composite — see service)
+save experiment with provenance + benchmark_config
+```
+
+### 10.3 Evaluation (`evaluate_single`)
+
+LLM returns JSON:
+
+```json
+{
+  "context_relevance": 1-10,
+  "answer_similarity": 1-10,
+  "faithfulness": 1-10,
+  "hallucination": true|false,
+  "reasoning": "..."
+}
+```
+
+`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).
+
+### 10.4 Reports
+
+`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).
+
+### 10.5 Cost estimate
+
+`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.
+
+---
+
+## 11. Admin LLD
+
+| Endpoint | Behavior |
+|----------|----------|
+| `GET /admin/health` | App + Qdrant + SQLite ping |
+| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
+| `PUT .../corpus` · `.../boundary` | Set role defaults |
+| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
+| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
+| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
+| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
+| `POST /admin/cost-estimate` | Same heuristic as dry-run |
+
+Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).
+
+---
+
+## 12. Dashboard LLD (behavioral)
+
+| Concern | Design |
+|---------|--------|
+| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
+| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
+| Theme | Dark `#111113`… + amber accent (ADR-0008) |
+| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
+| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
+| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
+| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |
+
+No client router or global store — props from root state.
+
+---
+
+## 13. API Surface (concise)
+
+| Method | Path | Service |
+|--------|------|---------|
+| GET/POST | `/documents` | list / upload |
+| POST | `/documents/{id}/process` | Strategies |
+| DELETE | `/documents/{id}` | delete |
+| GET | `/strategies` | catalog |
+| POST | `/queries` | query |
+| GET | `/queries/{id}` | history |
+| POST | `/benchmarks` | Experiment or dry_run |
+| GET | `/benchmarks/{id}` | Experiment detail |
+| GET | `/benchmarks/{id}/report` | HTML |
+| GET | `/experiments` | list (+ filters used by Decision Board) |
+| * | `/admin/*` | ops |
+| GET | `/app/` | Dashboard |
+
+Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.
+
+**Request highlights**
+
+```python
+# Process
+boundary_model_id: Optional[str]
+corpus_model_id: Optional[str]
+
+# Query / Benchmark
+top_k: int = 5
+neighbor_prev / neighbor_next: int = 0..5
+corpus_model_id: Optional[str]
+```
+
+---
+
+## 14. Error Model
+
+| Exception | Typical cause | HTTP |
+|-----------|---------------|------|
+| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
+| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
+| `EmbeddingError` | Provider/API failure | 400 |
+| `QdrantError` | Collection/upsert/search failure | 400 |
+| `QueryError` | Missing doc, LLM answer failure | 400 |
+| `BenchmarkError` | Bad questions file, eval failure | 400 |
+
+Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.
+
+---
+
+## 15. Sequence Diagrams
+
+### Process (one Strategy)
+
+```
+Client → DocumentsAPI → DocumentService → ChunkingService
+  ChunkingService → resolve Boundary/Corpus
+  ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
+  ChunkingService → embed_texts (Corpus)
+  ChunkingService → Qdrant.upsert
+  ChunkingService → SQLite.update counts + provenance
+Client ← ProcessResponse
+```
+
+### Query with Neighbor Expansion
+
+```
+Client → QueryAPI → run_query
+  → embed_single (Corpus)
+  → Qdrant.search top-k
+  → apply_neighbor_expansion → get_chunks_by_indices
+  → _build_context → OpenAI chat
+  → SQLite.save_query
+Client ← QueryResponse (retrieved_chunks + expansion_tree)
+```
+
+---
+
+## 16. Extension Points
+
+| Extension | Steps |
+|-----------|-------|
+| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
+| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
+| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
+| New Admin op | Prefer `/admin` only when domain routers lack the capability |
+
+---
+
+## 17. Scripts
+
+`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).
+
+---
+
+## 18. Related Documents
+
+| Doc | Role |
+|-----|------|
+| [HLD.md](HLD.md) | System context and component view |
+| [data-flow.md](data-flow.md) | Narrative pipelines |
+| [configuration.md](configuration.md) | Env knobs |
+| [adr/](adr/) | Decision records |
+| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |

docs/README.md

@@ -6,12 +6,12 @@ Complete documentation for the RAG Chunking Benchmarker.
 
 ## Quick Start
 
-1. **New to the project?** Start with [Architecture Overview](architecture.md)
+1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail)
-2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
+2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md)
-3. **Need to use the API?** Check [API Reference](api-reference.md)
+3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
-4. **Configuring the system?** See [Configuration Guide](configuration.md)
+4. **Need to use the API?** Check [API Reference](api-reference.md)
-5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
+5. **Configuring the system?** See [Configuration Guide](configuration.md)
-6. **Curious about data flow?** See [Data Flow](data-flow.md)
+6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
 
 ---
 
@@ -19,13 +19,17 @@ Complete documentation for the RAG Chunking Benchmarker.
 
 | File | Purpose | Audience |
 |------|---------|----------|
-| [architecture.md](architecture.md) | System structure and design | New team members |
+| [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members |
+| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers |
+| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members |
 | [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
 | [api-reference.md](api-reference.md) | All endpoints documented | Developers |
 | [configuration.md](configuration.md) | Settings and environment variables | DevOps |
 | [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
 | [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
 | [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
+| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators |
+| [adr/](adr/) | Architectural Decision Records | Everyone |
 | [phases.md](phases.md) | Implementation phases | Project managers |
 | [tasks.md](tasks.md) | Task tracking | Developers |