From 6261bc0a2324e7ca77a41283ad3383445baf7a92 Mon Sep 17 00:00:00 2001 From: Mahdi Bazrafshan Date: Mon, 17 Aug 2026 14:22:35 +0330 Subject: [PATCH] docs: add high- and low-level design Co-authored-by: Cursor --- docs/HLD.md | 280 ++++++++++++++++++++++++ docs/LLD.md | 571 +++++++++++++++++++++++++++++++++++++++++++++++++ docs/README.md | 18 +- 3 files changed, 862 insertions(+), 7 deletions(-) create mode 100644 docs/HLD.md create mode 100644 docs/LLD.md diff --git a/docs/HLD.md b/docs/HLD.md new file mode 100644 index 0000000..af55253 --- /dev/null +++ b/docs/HLD.md @@ -0,0 +1,280 @@ +# High-Level Design (HLD) + +RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows. + +**Audience:** architects, tech leads, new engineers +**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md) + +--- + +## 1. Purpose + +Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation. + +Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board. + +--- + +## 2. Goals and Non-Goals + +### Goals + +| Goal | Notes | +|------|--------| +| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment | +| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) | +| Operator UX | Dashboard at `/app` replaces Swagger as primary UI | +| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry | +| Failure isolation | One Strategy failing during process does not discard others | + +### Non-Goals (v1) + +| Out of scope | See | +|--------------|-----| +| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog | +| Multi-tenant auth | Single-operator local tool | +| Production-scale concurrency | SQLite + single process | +| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus | + +--- + +## 3. System Context + +``` +┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐ +│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│ +│ (Browser) │ Dashboard /app │ │ +└──────────────┘ │ Documents · Chunking · Benchmarking │ + │ Admin · Static Dashboard │ + └───────────┬─────────────────────────┘ + │ + ┌─────────────────────────────────┼─────────────────────────────────┐ + ▼ ▼ ▼ + ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ + │ SQLite │ │ Qdrant │ │ LLM / Embed │ + │ metadata, │ │ Model Corpus │ │ OpenAI Cloud │ + │ Experiments, │ │ vectors + │ │ + Ollama Local│ + │ app_settings │ │ chunk payload │ └────────────────┘ + └────────────────┘ └────────────────┘ +``` + +**External dependencies** + +| System | Role | +|--------|------| +| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge | +| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) | +| Qdrant | Vector search per Strategy × Corpus Embedding Model | +| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser | + +--- + +## 4. Logical Architecture + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Presentation Layer │ +│ src/static/index.html — single-file React Dashboard (CDN, no build) │ +│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │ +└────────────────────────────────┬─────────────────────────────────────────┘ + │ REST +┌────────────────────────────────▼─────────────────────────────────────────┐ +│ API Layer (FastAPI) │ +│ /documents* /queries* /benchmarks* /experiments* /admin/* │ +│ /strategies /app (StaticFiles) │ +└────────────────────────────────┬─────────────────────────────────────────┘ + │ +┌────────────────────────────────▼─────────────────────────────────────────┐ +│ Domain Services │ +│ documents/service chunking/service query_service benchmark_service│ +│ evaluation admin/service report │ +└────────────────────────────────┬─────────────────────────────────────────┘ + │ +┌────────────────────────────────▼─────────────────────────────────────────┐ +│ Foundation │ +│ core/config · models · exceptions · dependencies │ +│ chunking/base · embedding · embedding_models · strategies/* │ +│ documents/parser · pdf_parser │ +│ storage/sqlite · storage/qdrant │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 5. Major Components + +| Component | Responsibility | Key modules | +|-----------|----------------|-------------| +| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` | +| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` | +| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` | +| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` | +| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` | +| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` | +| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` | +| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` | + +--- + +## 6. Domain Concepts (summary) + +Full glossary: [CONTEXT.md](../CONTEXT.md). + +| Term | Meaning | +|------|---------| +| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` | +| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance | +| **Boundary Embedding Model** | Used only for Semantic Boundary Detection | +| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections | +| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model | +| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) | +| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) | +| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) | +| **Decision Board** | Two-stage selection over the fixed evaluation document set | + +--- + +## 7. End-to-End Flows + +### 7.1 Ingest and process + +``` +Upload (.docx/.doc/.pdf) + → Text-layer Gate (PDF) / parse DOCX + → Heading Reconstruction (PDF) → DocumentTree + markdown + → SQLite documents + +Process(strategies, boundary?, corpus?) + → For each Strategy (isolated failures): + Boundary embeds (semantic*) → Strategy.chunk() + → Corpus embeds → Qdrant Model Corpus upsert + → Update chunk_counts + last_* embedding provenance on document +``` + +### 7.2 Query + +``` +POST /queries + → Resolve Corpus Embedding Model + → Embed question (Corpus) + → Qdrant top-k (document filter) + → Neighbor Expansion if fixed_size (±P/N) + → Build context (parent fetch for semantic_parent_child) + → LLM answer → SQLite queries (+ expansion_tree) +``` + +### 7.3 Experiment + +``` +POST /benchmarks + → Load questions JSON + → For each question × Strategy: Query + evaluate_single + → Aggregate metrics → SQLite experiments + → Optional HTML report (managerial / technical) +``` + +### 7.4 Decision + +``` +Dashboard Decision Tab + → Load Experiments filtered by Corpus Embedding Model + → Discover Strategy Candidates (newest per doc × Candidate) + → Stage 1: best fixed_size ±N vs best semantic@Boundary + → Stage 2: compare winners + per-document breakdown +``` + +--- + +## 8. Data Architecture + +### 8.1 Responsibility split + +| Store | Owns | +|-------|------| +| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) | +| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) | +| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse | + +### 8.2 Model Corpus naming + +``` +{strategy}__{sanitized_model_id}_collection +``` + +Example: `fixed_size__text-embedding-3-large_collection` + +Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched. + +### 8.3 Provenance + +Experiments store: + +- `embedding_model_id` / `embedding_provider` — Corpus +- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document +- Neighbor Expansion levels in `benchmark_config` + +Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process. + +--- + +## 9. Deployment View + +``` +┌─────────────────────────────────────────┐ +│ Host process │ +│ uvicorn src.main:app │ +│ └── FastAPI + Dashboard static │ +│ │ +│ data/chunking_benchmark.db │ +│ .env (secrets + knobs) │ +└───────────────┬─────────────────────────┘ + │ + ┌───────────┴───────────┐ + ▼ ▼ + Qdrant (:6333) OpenAI / Ollama +``` + +Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard. + +--- + +## 10. Cross-Cutting Concerns + +| Concern | Approach | +|---------|----------| +| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) | +| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) | +| Logging | Request middleware + structured stage logs in query/benchmark | +| CORS | Permissive (`*`) for local Dashboard development | +| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side | +| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry | + +--- + +## 11. Key Architectural Decisions + +| ADR | Decision | +|-----|----------| +| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme | +| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections | +| 0020 | Semantic Boundary Detection required (no fixed-count fallback) | +| 0021 | Always scope Qdrant collections by Embedding Model id | +| 0022 | Per-model `semantic_threshold` | +| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` | +| 0024 | Boundary vs Corpus Embedding Model roles | +| 0025 | Retrieval Inspect in Benchmarks Tab | +| 0026 | Decision Board two-stage Candidate selection | + +--- + +## 12. Related Documents + +| Doc | Role | +|-----|------| +| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms | +| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) | +| [data-flow.md](data-flow.md) | Pipeline detail | +| [api-reference.md](api-reference.md) | HTTP contracts | +| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms | +| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions | +| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language | diff --git a/docs/LLD.md b/docs/LLD.md new file mode 100644 index 0000000..2127788 --- /dev/null +++ b/docs/LLD.md @@ -0,0 +1,571 @@ +# Low-Level Design (LLD) + +RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts. + +**Audience:** implementers +**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md) + +--- + +## 1. Package Map + +``` +src/ +├── main.py # App factory, middleware, router mount +├── static/index.html # Dashboard (React + Babel CDN) +├── core/ +│ ├── config.py # Settings (.env) +│ ├── models.py # Chunk, DocumentTree, StrategyName +│ ├── exceptions.py # Domain errors + handlers +│ └── dependencies.py # OpenAI / Ollama / Qdrant singletons +├── storage/ +│ ├── sqlite.py # documents, queries, experiments, app_settings +│ └── qdrant.py # Model Corpus CRUD + search +├── documents/ +│ ├── parser.py # DOCX / .doc (+ LibreOffice) +│ ├── pdf_parser.py # Text PDF + Text-layer Gate +│ ├── heading_heuristics.py # PDF Heading Reconstruction helpers +│ ├── service.py # Upload / process / delete +│ ├── routes.py +│ └── models.py # ProcessRequest, DocumentResponse, … +├── chunking/ +│ ├── base.py # ChunkingStrategy ABC, tokens, sentences +│ ├── embedding.py # Boundary/Corpus resolve + embed_texts +│ ├── embedding_models.py # Registry, thresholds, task prefixes +│ ├── service.py # run_strategies orchestration +│ └── strategies/ +│ ├── fixed_size.py +│ ├── recursive.py +│ ├── semantic.py +│ ├── contextual_retrieval.py +│ └── semantic_parent_child.py +├── benchmarking/ +│ ├── query_service.py # Query + Neighbor Expansion +│ ├── benchmark_service.py # Experiment runner + cost estimate +│ ├── evaluation.py # LLM-as-Judge +│ ├── report.py # HTML reports +│ ├── routes.py +│ └── models.py +└── admin/ + ├── routes.py # /admin/* + └── service.py +``` + +--- + +## 2. Application Bootstrap + +**Entry:** `create_app()` in `src/main.py` + +1. `init_db()` — create/migrate SQLite schema +2. Register CORS + `RequestLoggingMiddleware` +3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError` +4. Routers: documents, benchmarking, admin +5. Mount `StaticFiles` at `/app` → `src/static/` + +```python +# Singletons (lru_cache) +get_openai_client() # Cloud LLM + cloud embeds +get_ollama_client() # OpenAI-compatible client → Ollama base URL +get_qdrant_client() +``` + +--- + +## 3. Configuration + +`Settings` (`src/core/config.py`) — selected fields: + +| Field | Default | Use | +|-------|---------|-----| +| `openai_api_key` | required | Cloud Provider | +| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin | +| `llm_model` | `gpt-4o-mini` | Answer + judge | +| `ollama_base_url` | host URL | Local embeds | +| `qdrant_url` | `http://localhost:6333` | Vectors | +| `top_k` | `5` | Retrieval | +| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults | +| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets | +| `semantic_threshold` | `0.3` | Fallback if model default missing | +| `semantic_min_chunk_size` | `3` | Min units per semantic chunk | +| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata | +| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate | + +--- + +## 4. Core Domain Models + +### 4.1 StrategyName + +```python +class StrategyName(str, Enum): + RECURSIVE = "recursive" + FIXED_SIZE = "fixed_size" + SEMANTIC = "semantic" + CONTEXTUAL_RETRIEVAL = "contextual_retrieval" + SEMANTIC_PARENT_CHILD = "semantic_parent_child" +``` + +### 4.2 Chunk (unified) + +| Field | Type | Notes | +|-------|------|-------| +| `document_name` | str | Source filename | +| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` | +| `strategy_name` | StrategyName | | +| `chunk_index` | int | Document order (used by Neighbor Expansion) | +| `text` | str | Stored in Qdrant; used for LLM context | +| `token_count` / `character_count` | int | tiktoken `cl100k_base` | +| `parent_id` | str \| None | Parent-child Strategy | +| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) | + +`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`). + +### 4.3 DocumentTree + +``` +DocumentTree + └── root: DocumentTreeNode + node_type: document | section | article | paragraph + text, heading, heading_level, children[] +``` + +Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview. + +--- + +## 5. Storage LLD + +### 5.1 SQLite schema + +**documents** + +| Column | Type | Purpose | +|--------|------|---------| +| `id` | TEXT PK | UUID hex | +| `filename` | TEXT | Original name | +| `parsed_text` | TEXT | Markdown for chunking | +| `document_tree` | TEXT | JSON tree | +| `chunk_counts` | TEXT | JSON `{strategy: count}` | +| `last_corpus_embedding_model_id` | TEXT | Provenance | +| `last_boundary_embedding_model_id` | TEXT | Provenance | +| `created_at` | TEXT | ISO UTC | + +**queries** + +| Column | Purpose | +|--------|---------| +| `id`, `document_id`, `strategy_name` | Identity | +| `question`, `answer` | Content | +| `retrieved_chunks` | Flat LLM/eval list (JSON) | +| `expansion_tree` | Per-hit neighbors (JSON) | +| `latency_breakdown`, `token_usage` | Observability JSON | +| `created_at` | | + +**experiments** + +| Column | Purpose | +|--------|---------| +| `id`, `document_id` | Identity | +| `benchmark_config` | top_k, neighbor_prev/next, etc. | +| `questions`, `per_question` | Inputs + per-cell results | +| `aggregate_metrics` | Per-Strategy averages | +| `strategies_used` | JSON list | +| `embedding_model_id`, `embedding_provider` | Corpus snapshot | +| `boundary_embedding_model_id` | Boundary snapshot (nullable) | +| `created_at` | | + +**app_settings** — key/value for: + +- `corpus_embedding_model_id`, `boundary_embedding_model_id` +- Legacy `active_embedding_model_id` (migrated into both roles) +- `semantic_threshold:{model_id}` overrides + +Connection: WAL mode, foreign keys ON, one connection per call (no pool). + +### 5.2 Qdrant + +**Collection name** + +```python +def collection_name(strategy, model_id) -> str: + return f"{strategy}__{sanitize(model_id)}_collection" +# sanitize: replace : and / with - +``` + +**Point** + +| Field | Value | +|-------|-------| +| `id` | `uuid5(NAMESPACE_URL, chunk_id)` | +| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) | +| `payload` | ChunkMetadata fields | + +**Search** + +- Cosine distance +- Optional filter: `document_name == filename` +- Returns payload + score + +**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion. + +**Admin:** list/create/delete collections; wipe points. + +--- + +## 6. Embedding Subsystem + +### 6.1 Registry (`embedding_models.py`) + +| id | Provider | Dimension | Default semantic_threshold | Task prefixes | +|----|----------|-----------|----------------------------|---------------| +| `text-embedding-3-small` | cloud | 1536 | 0.3 | no | +| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) | +| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) | + +`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`. + +### 6.2 Role resolution (`embedding.py`) + +``` +resolve_corpus_model(optional_id) → snapshot for process/query/experiment +resolve_boundary_model(optional_id) → snapshot for semantic cuts + +get_corpus_embedding_model() / get_boundary_embedding_model() + → app_settings → registry → DEFAULT_CLOUD_MODEL_ID +``` + +Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots. + +### 6.3 embed_texts / embed_single + +1. `apply_task_prefixes(texts, model, purpose)` if needed +2. Batch: OpenAI 2048 / Ollama 64 +3. Route to `get_openai_client()` or `get_ollama_client()` by Provider +4. Raise `EmbeddingError` on failure + +**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus). + +--- + +## 7. Documents LLD + +### 7.1 Upload + +``` +upload_document(filename, bytes) + → temp file → parse_document(path) + → db.save_document(parsed_text=markdown, document_tree=JSON) +``` + +**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`). + +**PDF path (`pdf_parser.py`):** + +1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF +2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics) +3. Table Flattening to sequential plain text +4. Emit markdown `#` / `##` + DocumentTree + +### 7.2 Process + +``` +ProcessRequest: + strategies: list[StrategyName] # default all five + boundary_model_id: str | None + corpus_model_id: str | None + +process_document → chunking.service.run_strategies(...) + → ProcessResponse(completed[], failed[], corpus_*, boundary_*) +``` + +### 7.3 Delete + +Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`). + +--- + +## 8. Chunking LLD + +### 8.1 Interface + +```python +class ChunkingStrategy(ABC): + name: StrategyName + + @abstractmethod + def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]: + ... +``` + +Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`). + +### 8.2 Orchestrator (`run_strategies`) + +``` +corpus = resolve_corpus_model(corpus_model_id) +boundary = resolve_boundary_model(...) if any(semantic*) else unused + +for strategy in strategies: + try: + ensure_collection(strategy, corpus.id, corpus.dimension) + chunks = _chunk_document(...) # inject Boundary embeds if needed + texts = enriched_content or text + embeddings = embed_texts(texts, model=corpus, purpose="document") + upsert_chunks(chunks, embeddings, model_id=corpus.id) + except → failed[] (others continue) + +update_chunk_counts + update_process_embedding_provenance +``` + +### 8.3 Strategy algorithms (summary) + +| Strategy | Input units | Boundary | Output | +|----------|-------------|----------|--------| +| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` | +| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size | +| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups | +| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed | +| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` | + +Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback. + +**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit. + +Deep dive: [strategy-technical-details.md](strategy-technical-details.md). + +--- + +## 9. Query LLD + +### 9.1 `run_query` pipeline + +``` +1. Load document (filename for filter) +2. Resolve Corpus model +3. embed_single(question, purpose="query") +4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id) +5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree +6. _build_context (parent scroll for semantic_parent_child) +7. _generate_answer (settings.llm_model, temp 0) +8. db.save_query(...) +``` + +### 9.2 Neighbor Expansion (`apply_neighbor_expansion`) + +**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`. + +1. Build Expansion Tree nodes for each top-k hit (score order) +2. Collect needed `chunk_index` values: hit±prev/next (skip < 0) +3. Fetch missing indices via Qdrant +4. Fill `neighbors_prev` / `neighbors_next` per hit +5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index` +6. Neighbors have `role="neighbor"`, `score=None` + +Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays). + +### 9.3 Parent-child context + +For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context. + +--- + +## 10. Benchmark LLD + +### 10.1 Questions format + +```json +{ + "questions": [ + { + "id": "q1", + "question": "...", + "expected_answer": "...", + "category": "...", + "difficulty": "..." + } + ] +} +``` + +Load from `questions_file` path or inline `questions` list. + +### 10.2 `run_benchmark` + +``` +embedding_model = resolve_corpus_model(corpus_model_id) +boundary_id = doc.last_boundary_embedding_model_id # if semantic in strategies + +for question in questions: + for strategy in strategies: + result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...) + scores = evaluate_single(question, retrieved_chunks, expected, generated) + append per_question row + +aggregate_metrics[strategy] = averages + hallucination_rate +best_strategy = argmax (e.g. answer_similarity / composite — see service) +save experiment with provenance + benchmark_config +``` + +### 10.3 Evaluation (`evaluate_single`) + +LLM returns JSON: + +```json +{ + "context_relevance": 1-10, + "answer_similarity": 1-10, + "faithfulness": 1-10, + "hallucination": true|false, + "reasoning": "..." +} +``` + +`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md). + +### 10.4 Reports + +`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage). + +### 10.5 Cost estimate + +`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0. + +--- + +## 11. Admin LLD + +| Endpoint | Behavior | +|----------|----------| +| `GET /admin/health` | App + Qdrant + SQLite ping | +| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds | +| `PUT .../corpus` · `.../boundary` | Set role defaults | +| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` | +| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points | +| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant | +| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` | +| `POST /admin/cost-estimate` | Same heuristic as dry-run | + +Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006). + +--- + +## 12. Dashboard LLD (behavioral) + +| Concern | Design | +|---------|--------| +| Delivery | One `index.html`; React + ReactDOM + Babel from CDN | +| Navigation | Top Tabs; components stay mounted (`useState` on `App`) | +| Theme | Dark `#111113`… + amber accent (ADR-0008) | +| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter | +| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree | +| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison | +| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor | + +No client router or global store — props from root state. + +--- + +## 13. API Surface (concise) + +| Method | Path | Service | +|--------|------|---------| +| GET/POST | `/documents` | list / upload | +| POST | `/documents/{id}/process` | Strategies | +| DELETE | `/documents/{id}` | delete | +| GET | `/strategies` | catalog | +| POST | `/queries` | query | +| GET | `/queries/{id}` | history | +| POST | `/benchmarks` | Experiment or dry_run | +| GET | `/benchmarks/{id}` | Experiment detail | +| GET | `/benchmarks/{id}/report` | HTML | +| GET | `/experiments` | list (+ filters used by Decision Board) | +| * | `/admin/*` | ops | +| GET | `/app/` | Dashboard | + +Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`. + +**Request highlights** + +```python +# Process +boundary_model_id: Optional[str] +corpus_model_id: Optional[str] + +# Query / Benchmark +top_k: int = 5 +neighbor_prev / neighbor_next: int = 0..5 +corpus_model_id: Optional[str] +``` + +--- + +## 14. Error Model + +| Exception | Typical cause | HTTP | +|-----------|---------------|------| +| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 | +| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 | +| `EmbeddingError` | Provider/API failure | 400 | +| `QdrantError` | Collection/upsert/search failure | 400 | +| `QueryError` | Missing doc, LLM answer failure | 400 | +| `BenchmarkError` | Bad questions file, eval failure | 400 | + +Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request. + +--- + +## 15. Sequence Diagrams + +### Process (one Strategy) + +``` +Client → DocumentsAPI → DocumentService → ChunkingService + ChunkingService → resolve Boundary/Corpus + ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds) + ChunkingService → embed_texts (Corpus) + ChunkingService → Qdrant.upsert + ChunkingService → SQLite.update counts + provenance +Client ← ProcessResponse +``` + +### Query with Neighbor Expansion + +``` +Client → QueryAPI → run_query + → embed_single (Corpus) + → Qdrant.search top-k + → apply_neighbor_expansion → get_chunks_by_indices + → _build_context → OpenAI chat + → SQLite.save_query +Client ← QueryResponse (retrieved_chunks + expansion_tree) +``` + +--- + +## 16. Extension Points + +| Extension | Steps | +|-----------|-------| +| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels | +| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections | +| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates | +| New Admin op | Prefer `/admin` only when domain routers lack the capability | + +--- + +## 17. Scripts + +`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment). + +--- + +## 18. Related Documents + +| Doc | Role | +|-----|------| +| [HLD.md](HLD.md) | System context and component view | +| [data-flow.md](data-flow.md) | Narrative pipelines | +| [configuration.md](configuration.md) | Env knobs | +| [adr/](adr/) | Decision records | +| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics | diff --git a/docs/README.md b/docs/README.md index f514de4..9b58a1f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,12 +6,12 @@ Complete documentation for the RAG Chunking Benchmarker. ## Quick Start -1. **New to the project?** Start with [Architecture Overview](architecture.md) -2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md) -3. **Need to use the API?** Check [API Reference](api-reference.md) -4. **Configuring the system?** See [Configuration Guide](configuration.md) -5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md) -6. **Curious about data flow?** See [Data Flow](data-flow.md) +1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail) +2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md) +3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md) +4. **Need to use the API?** Check [API Reference](api-reference.md) +5. **Configuring the system?** See [Configuration Guide](configuration.md) +6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md) --- @@ -19,13 +19,17 @@ Complete documentation for the RAG Chunking Benchmarker. | File | Purpose | Audience | |------|---------|----------| -| [architecture.md](architecture.md) | System structure and design | New team members | +| [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members | +| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers | +| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members | | [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers | | [api-reference.md](api-reference.md) | All endpoints documented | Developers | | [configuration.md](configuration.md) | Settings and environment variables | DevOps | | [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists | | [data-flow.md](data-flow.md) | How data moves through the system | Engineers | | [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone | +| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators | +| [adr/](adr/) | Architectural Decision Records | Everyone | | [phases.md](phases.md) | Implementation phases | Project managers | | [tasks.md](tasks.md) | Task tracking | Developers |