docs: add high- and low-level design

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-17 14:22:35 +03:30
parent 2ac69c9944
commit 6261bc0a23
3 changed files with 862 additions and 7 deletions
--- a/docs/HLD.md
+++ b/docs/HLD.md
@@ -0,0 +1,280 @@
+# High-Level Design (HLD)
+
+RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
+
+**Audience:** architects, tech leads, new engineers  
+**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
+
+---
+
+## 1. Purpose
+
+Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
+
+Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
+
+---
+
+## 2. Goals and Non-Goals
+
+### Goals
+
+| Goal | Notes |
+|------|--------|
+| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
+| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
+| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
+| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
+| Failure isolation | One Strategy failing during process does not discard others |
+
+### Non-Goals (v1)
+
+| Out of scope | See |
+|--------------|-----|
+| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
+| Multi-tenant auth | Single-operator local tool |
+| Production-scale concurrency | SQLite + single process |
+| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
+
+---
+
+## 3. System Context
+
+```
+┌──────────────┐     HTTP/REST      ┌─────────────────────────────────────┐
+│  Operator    │◄──────────────────►│  FastAPI App (RAG Chunking Benchmarker)│
+│  (Browser)   │   Dashboard /app   │                                     │
+└──────────────┘                    │  Documents · Chunking · Benchmarking │
+                                    │  Admin · Static Dashboard           │
+                                    └───────────┬─────────────────────────┘
+                                                │
+              ┌─────────────────────────────────┼─────────────────────────────────┐
+              ▼                                 ▼                                 ▼
+     ┌────────────────┐              ┌────────────────┐              ┌────────────────┐
+     │     SQLite     │              │     Qdrant     │              │  LLM / Embed   │
+     │  metadata,     │              │  Model Corpus  │              │  OpenAI Cloud  │
+     │  Experiments,  │              │  vectors +     │              │  + Ollama Local│
+     │  app_settings  │              │  chunk payload │              └────────────────┘
+     └────────────────┘              └────────────────┘
+```
+
+**External dependencies**
+
+| System | Role |
+|--------|------|
+| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
+| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
+| Qdrant | Vector search per Strategy × Corpus Embedding Model |
+| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
+
+---
+
+## 4. Logical Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                         Presentation Layer                               │
+│  src/static/index.html — single-file React Dashboard (CDN, no build)     │
+│  Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin    │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │ REST
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                         API Layer (FastAPI)                              │
+│  /documents*   /queries*   /benchmarks*   /experiments*   /admin/*       │
+│  /strategies   /app (StaticFiles)                                        │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                       Domain Services                                    │
+│  documents/service   chunking/service   query_service   benchmark_service│
+│  evaluation          admin/service      report                           │
+└────────────────────────────────┬─────────────────────────────────────────┘
+                                 │
+┌────────────────────────────────▼─────────────────────────────────────────┐
+│                       Foundation                                         │
+│  core/config · models · exceptions · dependencies                        │
+│  chunking/base · embedding · embedding_models · strategies/*             │
+│  documents/parser · pdf_parser                                           │
+│  storage/sqlite · storage/qdrant                                         │
+└──────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 5. Major Components
+
+| Component | Responsibility | Key modules |
+|-----------|----------------|-------------|
+| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
+| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
+| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
+| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
+| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
+| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
+| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
+| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
+
+---
+
+## 6. Domain Concepts (summary)
+
+Full glossary: [CONTEXT.md](../CONTEXT.md).
+
+| Term | Meaning |
+|------|---------|
+| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
+| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
+| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
+| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
+| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
+| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
+| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
+| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
+| **Decision Board** | Two-stage selection over the fixed evaluation document set |
+
+---
+
+## 7. End-to-End Flows
+
+### 7.1 Ingest and process
+
+```
+Upload (.docx/.doc/.pdf)
+  → Text-layer Gate (PDF) / parse DOCX
+  → Heading Reconstruction (PDF) → DocumentTree + markdown
+  → SQLite documents
+
+Process(strategies, boundary?, corpus?)
+  → For each Strategy (isolated failures):
+       Boundary embeds (semantic*) → Strategy.chunk()
+       → Corpus embeds → Qdrant Model Corpus upsert
+  → Update chunk_counts + last_* embedding provenance on document
+```
+
+### 7.2 Query
+
+```
+POST /queries
+  → Resolve Corpus Embedding Model
+  → Embed question (Corpus)
+  → Qdrant top-k (document filter)
+  → Neighbor Expansion if fixed_size (±P/N)
+  → Build context (parent fetch for semantic_parent_child)
+  → LLM answer → SQLite queries (+ expansion_tree)
+```
+
+### 7.3 Experiment
+
+```
+POST /benchmarks
+  → Load questions JSON
+  → For each question × Strategy: Query + evaluate_single
+  → Aggregate metrics → SQLite experiments
+  → Optional HTML report (managerial / technical)
+```
+
+### 7.4 Decision
+
+```
+Dashboard Decision Tab
+  → Load Experiments filtered by Corpus Embedding Model
+  → Discover Strategy Candidates (newest per doc × Candidate)
+  → Stage 1: best fixed_size ±N vs best semantic@Boundary
+  → Stage 2: compare winners + per-document breakdown
+```
+
+---
+
+## 8. Data Architecture
+
+### 8.1 Responsibility split
+
+| Store | Owns |
+|-------|------|
+| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
+| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
+| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
+
+### 8.2 Model Corpus naming
+
+```
+{strategy}__{sanitized_model_id}_collection
+```
+
+Example: `fixed_size__text-embedding-3-large_collection`
+
+Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
+
+### 8.3 Provenance
+
+Experiments store:
+
+- `embedding_model_id` / `embedding_provider` — Corpus
+- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
+- Neighbor Expansion levels in `benchmark_config`
+
+Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
+
+---
+
+## 9. Deployment View
+
+```
+┌─────────────────────────────────────────┐
+│  Host process                           │
+│  uvicorn src.main:app                   │
+│  └── FastAPI + Dashboard static         │
+│                                         │
+│  data/chunking_benchmark.db             │
+│  .env (secrets + knobs)                 │
+└───────────────┬─────────────────────────┘
+                │
+    ┌───────────┴───────────┐
+    ▼                       ▼
+ Qdrant (:6333)      OpenAI / Ollama
+```
+
+Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
+
+---
+
+## 10. Cross-Cutting Concerns
+
+| Concern | Approach |
+|---------|----------|
+| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
+| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
+| Logging | Request middleware + structured stage logs in query/benchmark |
+| CORS | Permissive (`*`) for local Dashboard development |
+| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
+| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
+
+---
+
+## 11. Key Architectural Decisions
+
+| ADR | Decision |
+|-----|----------|
+| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
+| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
+| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
+| 0021 | Always scope Qdrant collections by Embedding Model id |
+| 0022 | Per-model `semantic_threshold` |
+| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
+| 0024 | Boundary vs Corpus Embedding Model roles |
+| 0025 | Retrieval Inspect in Benchmarks Tab |
+| 0026 | Decision Board two-stage Candidate selection |
+
+---
+
+## 12. Related Documents
+
+| Doc | Role |
+|-----|------|
+| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
+| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
+| [data-flow.md](data-flow.md) | Pipeline detail |
+| [api-reference.md](api-reference.md) | HTTP contracts |
+| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
+| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
+| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |