281 lines
14 KiB
Markdown
281 lines
14 KiB
Markdown
# High-Level Design (HLD)
|
||
|
||
RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
|
||
|
||
**Audience:** architects, tech leads, new engineers
|
||
**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
|
||
|
||
Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
|
||
|
||
---
|
||
|
||
## 2. Goals and Non-Goals
|
||
|
||
### Goals
|
||
|
||
| Goal | Notes |
|
||
|------|--------|
|
||
| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
|
||
| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
|
||
| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
|
||
| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
|
||
| Failure isolation | One Strategy failing during process does not discard others |
|
||
|
||
### Non-Goals (v1)
|
||
|
||
| Out of scope | See |
|
||
|--------------|-----|
|
||
| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
|
||
| Multi-tenant auth | Single-operator local tool |
|
||
| Production-scale concurrency | SQLite + single process |
|
||
| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
|
||
|
||
---
|
||
|
||
## 3. System Context
|
||
|
||
```
|
||
┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐
|
||
│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│
|
||
│ (Browser) │ Dashboard /app │ │
|
||
└──────────────┘ │ Documents · Chunking · Benchmarking │
|
||
│ Admin · Static Dashboard │
|
||
└───────────┬─────────────────────────┘
|
||
│
|
||
┌─────────────────────────────────┼─────────────────────────────────┐
|
||
▼ ▼ ▼
|
||
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
|
||
│ SQLite │ │ Qdrant │ │ LLM / Embed │
|
||
│ metadata, │ │ Model Corpus │ │ OpenAI Cloud │
|
||
│ Experiments, │ │ vectors + │ │ + Ollama Local│
|
||
│ app_settings │ │ chunk payload │ └────────────────┘
|
||
└────────────────┘ └────────────────┘
|
||
```
|
||
|
||
**External dependencies**
|
||
|
||
| System | Role |
|
||
|--------|------|
|
||
| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
|
||
| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
|
||
| Qdrant | Vector search per Strategy × Corpus Embedding Model |
|
||
| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
|
||
|
||
---
|
||
|
||
## 4. Logical Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ Presentation Layer │
|
||
│ src/static/index.html — single-file React Dashboard (CDN, no build) │
|
||
│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │
|
||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||
│ REST
|
||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||
│ API Layer (FastAPI) │
|
||
│ /documents* /queries* /benchmarks* /experiments* /admin/* │
|
||
│ /strategies /app (StaticFiles) │
|
||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||
│ Domain Services │
|
||
│ documents/service chunking/service query_service benchmark_service│
|
||
│ evaluation admin/service report │
|
||
└────────────────────────────────┬─────────────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────────▼─────────────────────────────────────────┐
|
||
│ Foundation │
|
||
│ core/config · models · exceptions · dependencies │
|
||
│ chunking/base · embedding · embedding_models · strategies/* │
|
||
│ documents/parser · pdf_parser │
|
||
│ storage/sqlite · storage/qdrant │
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Major Components
|
||
|
||
| Component | Responsibility | Key modules |
|
||
|-----------|----------------|-------------|
|
||
| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
|
||
| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
|
||
| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
|
||
| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
|
||
| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
|
||
| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
|
||
| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
|
||
| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
|
||
|
||
---
|
||
|
||
## 6. Domain Concepts (summary)
|
||
|
||
Full glossary: [CONTEXT.md](../CONTEXT.md).
|
||
|
||
| Term | Meaning |
|
||
|------|---------|
|
||
| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
|
||
| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
|
||
| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
|
||
| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
|
||
| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
|
||
| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
|
||
| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
|
||
| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
|
||
| **Decision Board** | Two-stage selection over the fixed evaluation document set |
|
||
|
||
---
|
||
|
||
## 7. End-to-End Flows
|
||
|
||
### 7.1 Ingest and process
|
||
|
||
```
|
||
Upload (.docx/.doc/.pdf)
|
||
→ Text-layer Gate (PDF) / parse DOCX
|
||
→ Heading Reconstruction (PDF) → DocumentTree + markdown
|
||
→ SQLite documents
|
||
|
||
Process(strategies, boundary?, corpus?)
|
||
→ For each Strategy (isolated failures):
|
||
Boundary embeds (semantic*) → Strategy.chunk()
|
||
→ Corpus embeds → Qdrant Model Corpus upsert
|
||
→ Update chunk_counts + last_* embedding provenance on document
|
||
```
|
||
|
||
### 7.2 Query
|
||
|
||
```
|
||
POST /queries
|
||
→ Resolve Corpus Embedding Model
|
||
→ Embed question (Corpus)
|
||
→ Qdrant top-k (document filter)
|
||
→ Neighbor Expansion if fixed_size (±P/N)
|
||
→ Build context (parent fetch for semantic_parent_child)
|
||
→ LLM answer → SQLite queries (+ expansion_tree)
|
||
```
|
||
|
||
### 7.3 Experiment
|
||
|
||
```
|
||
POST /benchmarks
|
||
→ Load questions JSON
|
||
→ For each question × Strategy: Query + evaluate_single
|
||
→ Aggregate metrics → SQLite experiments
|
||
→ Optional HTML report (managerial / technical)
|
||
```
|
||
|
||
### 7.4 Decision
|
||
|
||
```
|
||
Dashboard Decision Tab
|
||
→ Load Experiments filtered by Corpus Embedding Model
|
||
→ Discover Strategy Candidates (newest per doc × Candidate)
|
||
→ Stage 1: best fixed_size ±N vs best semantic@Boundary
|
||
→ Stage 2: compare winners + per-document breakdown
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Data Architecture
|
||
|
||
### 8.1 Responsibility split
|
||
|
||
| Store | Owns |
|
||
|-------|------|
|
||
| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
|
||
| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
|
||
| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
|
||
|
||
### 8.2 Model Corpus naming
|
||
|
||
```
|
||
{strategy}__{sanitized_model_id}_collection
|
||
```
|
||
|
||
Example: `fixed_size__text-embedding-3-large_collection`
|
||
|
||
Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
|
||
|
||
### 8.3 Provenance
|
||
|
||
Experiments store:
|
||
|
||
- `embedding_model_id` / `embedding_provider` — Corpus
|
||
- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
|
||
- Neighbor Expansion levels in `benchmark_config`
|
||
|
||
Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
|
||
|
||
---
|
||
|
||
## 9. Deployment View
|
||
|
||
```
|
||
┌─────────────────────────────────────────┐
|
||
│ Host process │
|
||
│ uvicorn src.main:app │
|
||
│ └── FastAPI + Dashboard static │
|
||
│ │
|
||
│ data/chunking_benchmark.db │
|
||
│ .env (secrets + knobs) │
|
||
└───────────────┬─────────────────────────┘
|
||
│
|
||
┌───────────┴───────────┐
|
||
▼ ▼
|
||
Qdrant (:6333) OpenAI / Ollama
|
||
```
|
||
|
||
Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
|
||
|
||
---
|
||
|
||
## 10. Cross-Cutting Concerns
|
||
|
||
| Concern | Approach |
|
||
|---------|----------|
|
||
| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
|
||
| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
|
||
| Logging | Request middleware + structured stage logs in query/benchmark |
|
||
| CORS | Permissive (`*`) for local Dashboard development |
|
||
| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
|
||
| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
|
||
|
||
---
|
||
|
||
## 11. Key Architectural Decisions
|
||
|
||
| ADR | Decision |
|
||
|-----|----------|
|
||
| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
|
||
| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
|
||
| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
|
||
| 0021 | Always scope Qdrant collections by Embedding Model id |
|
||
| 0022 | Per-model `semantic_threshold` |
|
||
| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
|
||
| 0024 | Boundary vs Corpus Embedding Model roles |
|
||
| 0025 | Retrieval Inspect in Benchmarks Tab |
|
||
| 0026 | Decision Board two-stage Candidate selection |
|
||
|
||
---
|
||
|
||
## 12. Related Documents
|
||
|
||
| Doc | Role |
|
||
|-----|------|
|
||
| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
|
||
| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
|
||
| [data-flow.md](data-flow.md) | Pipeline detail |
|
||
| [api-reference.md](api-reference.md) | HTTP contracts |
|
||
| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
|
||
| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
|
||
| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |
|