14 KiB
14 KiB
High-Level Design (HLD)
RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
Audience: architects, tech leads, new engineers
Companion: LLD · ADRs · CONTEXT.md
1. Purpose
Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
2. Goals and Non-Goals
Goals
| Goal | Notes |
|---|---|
| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
| Operator UX | Dashboard at /app replaces Swagger as primary UI |
| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
| Failure isolation | One Strategy failing during process does not discard others |
Non-Goals (v1)
| Out of scope | See |
|---|---|
| Scanned PDF / OCR | out-of-scope-v1.md, backlog |
| Multi-tenant auth | Single-operator local tool |
| Production-scale concurrency | SQLite + single process |
| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
3. System Context
┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐
│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│
│ (Browser) │ Dashboard /app │ │
└──────────────┘ │ Documents · Chunking · Benchmarking │
│ Admin · Static Dashboard │
└───────────┬─────────────────────────┘
│
┌─────────────────────────────────┼─────────────────────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ SQLite │ │ Qdrant │ │ LLM / Embed │
│ metadata, │ │ Model Corpus │ │ OpenAI Cloud │
│ Experiments, │ │ vectors + │ │ + Ollama Local│
│ app_settings │ │ chunk payload │ └────────────────┘
└────────────────┘ └────────────────┘
External dependencies
| System | Role |
|---|---|
| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
| Ollama | Local Embedding Model Provider (nomic-embed-text-v2-moe) |
| Qdrant | Vector search per Strategy × Corpus Embedding Model |
| LibreOffice (optional) | Legacy .doc → .docx conversion path in parser |
4. Logical Architecture
┌──────────────────────────────────────────────────────────────────────────┐
│ Presentation Layer │
│ src/static/index.html — single-file React Dashboard (CDN, no build) │
│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │
└────────────────────────────────┬─────────────────────────────────────────┘
│ REST
┌────────────────────────────────▼─────────────────────────────────────────┐
│ API Layer (FastAPI) │
│ /documents* /queries* /benchmarks* /experiments* /admin/* │
│ /strategies /app (StaticFiles) │
└────────────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────────────▼─────────────────────────────────────────┐
│ Domain Services │
│ documents/service chunking/service query_service benchmark_service│
│ evaluation admin/service report │
└────────────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────────────▼─────────────────────────────────────────┐
│ Foundation │
│ core/config · models · exceptions · dependencies │
│ chunking/base · embedding · embedding_models · strategies/* │
│ documents/parser · pdf_parser │
│ storage/sqlite · storage/qdrant │
└──────────────────────────────────────────────────────────────────────────┘
5. Major Components
| Component | Responsibility | Key modules |
|---|---|---|
| Dashboard | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | src/static/index.html |
| Documents | Upload/parse Word & Text PDF; process Strategies; delete | documents/* |
| Chunking | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | chunking/* |
| Query | Embed question → search → Neighbor Expansion → LLM answer | benchmarking/query_service.py |
| Benchmark | Question × Strategy Experiment; aggregate metrics; HTML reports | benchmarking/benchmark_service.py, report.py |
| Evaluation | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | benchmarking/evaluation.py |
| Admin | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | admin/* |
| Storage | SQLite metadata; Qdrant Model Corpus | storage/* |
6. Domain Concepts (summary)
Full glossary: CONTEXT.md.
| Term | Meaning |
|---|---|
| Strategy | One of: fixed_size, recursive, semantic, contextual_retrieval, semantic_parent_child |
| Experiment | Completed benchmark run for one document × N Strategies × M questions, with provenance |
| Boundary Embedding Model | Used only for Semantic Boundary Detection |
| Corpus Embedding Model | Embeds finished chunks and queries; scopes Qdrant collections |
| Model Corpus | Collections {strategy}__{model_id}_collection for one Corpus model |
| Neighbor Expansion | Query-time prev/next chunks for fixed_size only (±P/N) |
| Expansion Tree | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
| Strategy Candidate | Comparable config for Decision Board (fixed_size±N or semantic@Boundary) |
| Decision Board | Two-stage selection over the fixed evaluation document set |
7. End-to-End Flows
7.1 Ingest and process
Upload (.docx/.doc/.pdf)
→ Text-layer Gate (PDF) / parse DOCX
→ Heading Reconstruction (PDF) → DocumentTree + markdown
→ SQLite documents
Process(strategies, boundary?, corpus?)
→ For each Strategy (isolated failures):
Boundary embeds (semantic*) → Strategy.chunk()
→ Corpus embeds → Qdrant Model Corpus upsert
→ Update chunk_counts + last_* embedding provenance on document
7.2 Query
POST /queries
→ Resolve Corpus Embedding Model
→ Embed question (Corpus)
→ Qdrant top-k (document filter)
→ Neighbor Expansion if fixed_size (±P/N)
→ Build context (parent fetch for semantic_parent_child)
→ LLM answer → SQLite queries (+ expansion_tree)
7.3 Experiment
POST /benchmarks
→ Load questions JSON
→ For each question × Strategy: Query + evaluate_single
→ Aggregate metrics → SQLite experiments
→ Optional HTML report (managerial / technical)
7.4 Decision
Dashboard Decision Tab
→ Load Experiments filtered by Corpus Embedding Model
→ Discover Strategy Candidates (newest per doc × Candidate)
→ Stage 1: best fixed_size ±N vs best semantic@Boundary
→ Stage 2: compare winners + per-document breakdown
8. Data Architecture
8.1 Responsibility split
| Store | Owns |
|---|---|
| SQLite | Documents (tree + markdown), queries, Experiments, app_settings (Embedding defaults, thresholds) |
| Qdrant | Vectors + chunk text payload (no vectors in SQLite) |
| Filesystem | Question JSON under files/; uploaded bytes are not retained after parse |
8.2 Model Corpus naming
{strategy}__{sanitized_model_id}_collection
Example: fixed_size__text-embedding-3-large_collection
Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
8.3 Provenance
Experiments store:
embedding_model_id/embedding_provider— Corpusboundary_embedding_model_id— when semantic Strategies were on the process path for that document- Neighbor Expansion levels in
benchmark_config
Documents store last_corpus_embedding_model_id and last_boundary_embedding_model_id after process.
9. Deployment View
┌─────────────────────────────────────────┐
│ Host process │
│ uvicorn src.main:app │
│ └── FastAPI + Dashboard static │
│ │
│ data/chunking_benchmark.db │
│ .env (secrets + knobs) │
└───────────────┬─────────────────────────┘
│
┌───────────┴───────────┐
▼ ▼
Qdrant (:6333) OpenAI / Ollama
Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
10. Cross-Cutting Concerns
| Concern | Approach |
|---|---|
| Config | pydantic-settings from .env (configuration.md) |
| Errors | Domain exceptions → HTTP 400 handlers (ChunkingError, BenchmarkError, QueryError, …) |
| Logging | Request middleware + structured stage logs in query/benchmark |
| CORS | Permissive (*) for local Dashboard development |
| Cost | Dry-run /benchmarks and /admin/cost-estimate; Local embeds = $0 embedding side |
| Extensibility | New Strategy = implement ChunkingStrategy + registry; new Embedding Model = registry entry |
11. Key Architectural Decisions
| ADR | Decision |
|---|---|
| 0001–0009 | CDN React Dashboard; /admin for ops gaps; top Tabs; dark amber theme |
| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
| 0021 | Always scope Qdrant collections by Embedding Model id |
| 0022 | Per-model semantic_threshold |
| 0023 | Neighbor Expansion + Expansion Tree for fixed_size |
| 0024 | Boundary vs Corpus Embedding Model roles |
| 0025 | Retrieval Inspect in Benchmarks Tab |
| 0026 | Decision Board two-stage Candidate selection |
12. Related Documents
| Doc | Role |
|---|---|
| LLD.md | Module interfaces, schemas, algorithms |
| architecture.md | Older overview (prefer HLD for current shape) |
| data-flow.md | Pipeline detail |
| api-reference.md | HTTP contracts |
| strategy-technical-details.md | Per-Strategy algorithms |
| evaluation-metrics.md | Scoring definitions |
| CONTEXT.md | Ubiquitous language |