mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add high- and low-level design

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Mahdi Bazrafshan
2026-08-17 14:22:35 +03:30
parent 2ac69c9944
commit 6261bc0a23
3 changed files with 862 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

280
docs/HLD.md Normal file
View File

@@ -0,0 +1,280 @@
# High-Level Design (HLD)
RAG Chunking Benchmarker — system purpose, boundaries, components, and major flows.
**Audience:** architects, tech leads, new engineers
**Companion:** [LLD](LLD.md) · [ADRs](adr/) · [CONTEXT.md](../CONTEXT.md)
---
## 1. Purpose
Compare five chunking Strategies on regulatory (and similar) documents under a controlled RAG pipeline: ingest → chunk → embed → retrieve → answer → LLM-as-Judge evaluation.
Operators use a Dashboard to upload documents, process Strategies, run Queries and Experiments, inspect retrieval, and choose a final Strategy Candidate on the Decision Board.
---
## 2. Goals and Non-Goals
### Goals
| Goal | Notes |
|------|--------|
| Fair Strategy comparison | Same document, questions, Corpus Embedding Model, and Neighbor Expansion knobs per Experiment |
| Auditable provenance | Experiments record Corpus (+ Boundary when semantic Strategies were processed) |
| Operator UX | Dashboard at `/app` replaces Swagger as primary UI |
| Embedding flexibility | Cloud (OpenAI) and Local (Ollama) via Embedding Model Registry |
| Failure isolation | One Strategy failing during process does not discard others |
### Non-Goals (v1)
| Out of scope | See |
|--------------|-----|
| Scanned PDF / OCR | [out-of-scope-v1.md](out-of-scope-v1.md), backlog |
| Multi-tenant auth | Single-operator local tool |
| Production-scale concurrency | SQLite + single process |
| Comparing Experiments across different Boundary/Corpus/Neighbor settings without warning | Compare UI warns; Decision Board filters by Corpus |
---
## 3. System Context
```
┌──────────────┐ HTTP/REST ┌─────────────────────────────────────┐
│ Operator │◄──────────────────►│ FastAPI App (RAG Chunking Benchmarker)│
│ (Browser) │ Dashboard /app │ │
└──────────────┘ │ Documents · Chunking · Benchmarking │
│ Admin · Static Dashboard │
└───────────┬─────────────────────────┘
│
┌─────────────────────────────────┼─────────────────────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ SQLite │ │ Qdrant │ │ LLM / Embed │
│ metadata, │ │ Model Corpus │ │ OpenAI Cloud │
│ Experiments, │ │ vectors + │ │ + Ollama Local│
│ app_settings │ │ chunk payload │ └────────────────┘
└────────────────┘ └────────────────┘
```
**External dependencies**
| System | Role |
|--------|------|
| OpenAI API | Corpus/Boundary embeddings (cloud models); answer generation; LLM-as-Judge |
| Ollama | Local Embedding Model Provider (`nomic-embed-text-v2-moe`) |
| Qdrant | Vector search per Strategy × Corpus Embedding Model |
| LibreOffice (optional) | Legacy `.doc` → `.docx` conversion path in parser |
---
## 4. Logical Architecture
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Presentation Layer │
│ src/static/index.html — single-file React Dashboard (CDN, no build) │
│ Tabs: Home · Documents · Query · Benchmarks · Decision · PDF · Admin │
└────────────────────────────────┬─────────────────────────────────────────┘
│ REST
┌────────────────────────────────▼─────────────────────────────────────────┐
│ API Layer (FastAPI) │
│ /documents* /queries* /benchmarks* /experiments* /admin/* │
│ /strategies /app (StaticFiles) │
└────────────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────────────▼─────────────────────────────────────────┐
│ Domain Services │
│ documents/service chunking/service query_service benchmark_service│
│ evaluation admin/service report │
└────────────────────────────────┬─────────────────────────────────────────┘
│
┌────────────────────────────────▼─────────────────────────────────────────┐
│ Foundation │
│ core/config · models · exceptions · dependencies │
│ chunking/base · embedding · embedding_models · strategies/* │
│ documents/parser · pdf_parser │
│ storage/sqlite · storage/qdrant │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## 5. Major Components
| Component | Responsibility | Key modules |
|-----------|----------------|-------------|
| **Dashboard** | Operator UI; persistent Tabs; Decision Board; Retrieval Inspect; PDF Workspace | `src/static/index.html` |
| **Documents** | Upload/parse Word & Text PDF; process Strategies; delete | `documents/*` |
| **Chunking** | Strategy registry; Boundary vs Corpus embeds; chunk → embed → Qdrant | `chunking/*` |
| **Query** | Embed question → search → Neighbor Expansion → LLM answer | `benchmarking/query_service.py` |
| **Benchmark** | Question × Strategy Experiment; aggregate metrics; HTML reports | `benchmarking/benchmark_service.py`, `report.py` |
| **Evaluation** | LLM-as-Judge (context relevance, answer similarity, faithfulness, hallucination) | `benchmarking/evaluation.py` |
| **Admin** | Health, Embedding Model defaults/thresholds, Qdrant CRUD, Chunk Preview, questions files, cost estimate | `admin/*` |
| **Storage** | SQLite metadata; Qdrant Model Corpus | `storage/*` |
---
## 6. Domain Concepts (summary)
Full glossary: [CONTEXT.md](../CONTEXT.md).
| Term | Meaning |
|------|---------|
| **Strategy** | One of: `fixed_size`, `recursive`, `semantic`, `contextual_retrieval`, `semantic_parent_child` |
| **Experiment** | Completed benchmark run for one document × N Strategies × M questions, with provenance |
| **Boundary Embedding Model** | Used only for Semantic Boundary Detection |
| **Corpus Embedding Model** | Embeds finished chunks and queries; scopes Qdrant collections |
| **Model Corpus** | Collections `{strategy}__{model_id}_collection` for one Corpus model |
| **Neighbor Expansion** | Query-time prev/next chunks for `fixed_size` only (`±P/N`) |
| **Expansion Tree** | Per-hit neighbor grouping for operator audit (vs flat LLM context) |
| **Strategy Candidate** | Comparable config for Decision Board (`fixed_size±N` or `semantic@Boundary`) |
| **Decision Board** | Two-stage selection over the fixed evaluation document set |
---
## 7. End-to-End Flows
### 7.1 Ingest and process
```
Upload (.docx/.doc/.pdf)
→ Text-layer Gate (PDF) / parse DOCX
→ Heading Reconstruction (PDF) → DocumentTree + markdown
→ SQLite documents
Process(strategies, boundary?, corpus?)
→ For each Strategy (isolated failures):
Boundary embeds (semantic*) → Strategy.chunk()
→ Corpus embeds → Qdrant Model Corpus upsert
→ Update chunk_counts + last_* embedding provenance on document
```
### 7.2 Query
```
POST /queries
→ Resolve Corpus Embedding Model
→ Embed question (Corpus)
→ Qdrant top-k (document filter)
→ Neighbor Expansion if fixed_size (±P/N)
→ Build context (parent fetch for semantic_parent_child)
→ LLM answer → SQLite queries (+ expansion_tree)
```
### 7.3 Experiment
```
POST /benchmarks
→ Load questions JSON
→ For each question × Strategy: Query + evaluate_single
→ Aggregate metrics → SQLite experiments
→ Optional HTML report (managerial / technical)
```
### 7.4 Decision
```
Dashboard Decision Tab
→ Load Experiments filtered by Corpus Embedding Model
→ Discover Strategy Candidates (newest per doc × Candidate)
→ Stage 1: best fixed_size ±N vs best semantic@Boundary
→ Stage 2: compare winners + per-document breakdown
```
---
## 8. Data Architecture
### 8.1 Responsibility split
| Store | Owns |
|-------|------|
| **SQLite** | Documents (tree + markdown), queries, Experiments, `app_settings` (Embedding defaults, thresholds) |
| **Qdrant** | Vectors + chunk text payload (no vectors in SQLite) |
| **Filesystem** | Question JSON under `files/`; uploaded bytes are not retained after parse |
### 8.2 Model Corpus naming
```
{strategy}__{sanitized_model_id}_collection
```
Example: `fixed_size__text-embedding-3-large_collection`
Process, Query, and Experiment only read/write the Corpus Embedding Model in force. Other corpora remain untouched.
### 8.3 Provenance
Experiments store:
- `embedding_model_id` / `embedding_provider` — Corpus
- `boundary_embedding_model_id` — when semantic Strategies were on the process path for that document
- Neighbor Expansion levels in `benchmark_config`
Documents store `last_corpus_embedding_model_id` and `last_boundary_embedding_model_id` after process.
---
## 9. Deployment View
```
┌─────────────────────────────────────────┐
│ Host process │
│ uvicorn src.main:app │
│ └── FastAPI + Dashboard static │
│ │
│ data/chunking_benchmark.db │
│ .env (secrets + knobs) │
└───────────────┬─────────────────────────┘
│
┌───────────┴───────────┐
▼ ▼
Qdrant (:6333) OpenAI / Ollama
```
Typical local stack: app + Qdrant (Docker or native) + optional Ollama host. No build step for the Dashboard.
---
## 10. Cross-Cutting Concerns
| Concern | Approach |
|---------|----------|
| Config | `pydantic-settings` from `.env` ([configuration.md](configuration.md)) |
| Errors | Domain exceptions → HTTP 400 handlers (`ChunkingError`, `BenchmarkError`, `QueryError`, …) |
| Logging | Request middleware + structured stage logs in query/benchmark |
| CORS | Permissive (`*`) for local Dashboard development |
| Cost | Dry-run `/benchmarks` and `/admin/cost-estimate`; Local embeds = $0 embedding side |
| Extensibility | New Strategy = implement `ChunkingStrategy` + registry; new Embedding Model = registry entry |
---
## 11. Key Architectural Decisions
| ADR | Decision |
|-----|----------|
| 0001–0009 | CDN React Dashboard; `/admin` for ops gaps; top Tabs; dark amber theme |
| 0016–0017 | Text PDF via PyMuPDF; PDF Workspace via format-filtered shared sections |
| 0020 | Semantic Boundary Detection required (no fixed-count fallback) |
| 0021 | Always scope Qdrant collections by Embedding Model id |
| 0022 | Per-model `semantic_threshold` |
| 0023 | Neighbor Expansion + Expansion Tree for `fixed_size` |
| 0024 | Boundary vs Corpus Embedding Model roles |
| 0025 | Retrieval Inspect in Benchmarks Tab |
| 0026 | Decision Board two-stage Candidate selection |
---
## 12. Related Documents
| Doc | Role |
|-----|------|
| [LLD.md](LLD.md) | Module interfaces, schemas, algorithms |
| [architecture.md](architecture.md) | Older overview (prefer HLD for current shape) |
| [data-flow.md](data-flow.md) | Pipeline detail |
| [api-reference.md](api-reference.md) | HTTP contracts |
| [strategy-technical-details.md](strategy-technical-details.md) | Per-Strategy algorithms |
| [evaluation-metrics.md](evaluation-metrics.md) | Scoring definitions |
| [CONTEXT.md](../CONTEXT.md) | Ubiquitous language |

571
docs/LLD.md Normal file
View File

@@ -0,0 +1,571 @@
# Low-Level Design (LLD)
RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.
**Audience:** implementers
**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)
---
## 1. Package Map
```
src/
├── main.py # App factory, middleware, router mount
├── static/index.html # Dashboard (React + Babel CDN)
├── core/
│ ├── config.py # Settings (.env)
│ ├── models.py # Chunk, DocumentTree, StrategyName
│ ├── exceptions.py # Domain errors + handlers
│ └── dependencies.py # OpenAI / Ollama / Qdrant singletons
├── storage/
│ ├── sqlite.py # documents, queries, experiments, app_settings
│ └── qdrant.py # Model Corpus CRUD + search
├── documents/
│ ├── parser.py # DOCX / .doc (+ LibreOffice)
│ ├── pdf_parser.py # Text PDF + Text-layer Gate
│ ├── heading_heuristics.py # PDF Heading Reconstruction helpers
│ ├── service.py # Upload / process / delete
│ ├── routes.py
│ └── models.py # ProcessRequest, DocumentResponse, …
├── chunking/
│ ├── base.py # ChunkingStrategy ABC, tokens, sentences
│ ├── embedding.py # Boundary/Corpus resolve + embed_texts
│ ├── embedding_models.py # Registry, thresholds, task prefixes
│ ├── service.py # run_strategies orchestration
│ └── strategies/
│ ├── fixed_size.py
│ ├── recursive.py
│ ├── semantic.py
│ ├── contextual_retrieval.py
│ └── semantic_parent_child.py
├── benchmarking/
│ ├── query_service.py # Query + Neighbor Expansion
│ ├── benchmark_service.py # Experiment runner + cost estimate
│ ├── evaluation.py # LLM-as-Judge
│ ├── report.py # HTML reports
│ ├── routes.py
│ └── models.py
└── admin/
├── routes.py # /admin/*
└── service.py
```
---
## 2. Application Bootstrap
**Entry:** `create_app()` in `src/main.py`
1. `init_db()` — create/migrate SQLite schema
2. Register CORS + `RequestLoggingMiddleware`
3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`
4. Routers: documents, benchmarking, admin
5. Mount `StaticFiles` at `/app` → `src/static/`
```python
# Singletons (lru_cache)
get_openai_client() # Cloud LLM + cloud embeds
get_ollama_client() # OpenAI-compatible client → Ollama base URL
get_qdrant_client()
```
---
## 3. Configuration
`Settings` (`src/core/config.py`) — selected fields:
| Field | Default | Use |
|-------|---------|-----|
| `openai_api_key` | required | Cloud Provider |
| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
| `llm_model` | `gpt-4o-mini` | Answer + judge |
| `ollama_base_url` | host URL | Local embeds |
| `qdrant_url` | `http://localhost:6333` | Vectors |
| `top_k` | `5` | Retrieval |
| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults |
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
| `semantic_threshold` | `0.3` | Fallback if model default missing |
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |
---
## 4. Core Domain Models
### 4.1 StrategyName
```python
class StrategyName(str, Enum):
RECURSIVE = "recursive"
FIXED_SIZE = "fixed_size"
SEMANTIC = "semantic"
CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
SEMANTIC_PARENT_CHILD = "semantic_parent_child"
```
### 4.2 Chunk (unified)
| Field | Type | Notes |
|-------|------|-------|
| `document_name` | str | Source filename |
| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
| `strategy_name` | StrategyName | |
| `chunk_index` | int | Document order (used by Neighbor Expansion) |
| `text` | str | Stored in Qdrant; used for LLM context |
| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
| `parent_id` | str \| None | Parent-child Strategy |
| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |
`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).
### 4.3 DocumentTree
```
DocumentTree
└── root: DocumentTreeNode
node_type: document | section | article | paragraph
text, heading, heading_level, children[]
```
Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.
---
## 5. Storage LLD
### 5.1 SQLite schema
**documents**
| Column | Type | Purpose |
|--------|------|---------|
| `id` | TEXT PK | UUID hex |
| `filename` | TEXT | Original name |
| `parsed_text` | TEXT | Markdown for chunking |
| `document_tree` | TEXT | JSON tree |
| `chunk_counts` | TEXT | JSON `{strategy: count}` |
| `last_corpus_embedding_model_id` | TEXT | Provenance |
| `last_boundary_embedding_model_id` | TEXT | Provenance |
| `created_at` | TEXT | ISO UTC |
**queries**
| Column | Purpose |
|--------|---------|
| `id`, `document_id`, `strategy_name` | Identity |
| `question`, `answer` | Content |
| `retrieved_chunks` | Flat LLM/eval list (JSON) |
| `expansion_tree` | Per-hit neighbors (JSON) |
| `latency_breakdown`, `token_usage` | Observability JSON |
| `created_at` | |
**experiments**
| Column | Purpose |
|--------|---------|
| `id`, `document_id` | Identity |
| `benchmark_config` | top_k, neighbor_prev/next, etc. |
| `questions`, `per_question` | Inputs + per-cell results |
| `aggregate_metrics` | Per-Strategy averages |
| `strategies_used` | JSON list |
| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
| `created_at` | |
**app_settings** — key/value for:
- `corpus_embedding_model_id`, `boundary_embedding_model_id`
- Legacy `active_embedding_model_id` (migrated into both roles)
- `semantic_threshold:{model_id}` overrides
Connection: WAL mode, foreign keys ON, one connection per call (no pool).
### 5.2 Qdrant
**Collection name**
```python
def collection_name(strategy, model_id) -> str:
return f"{strategy}__{sanitize(model_id)}_collection"
# sanitize: replace : and / with -
```
**Point**
| Field | Value |
|-------|-------|
| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
| `payload` | ChunkMetadata fields |
**Search**
- Cosine distance
- Optional filter: `document_name == filename`
- Returns payload + score
**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.
**Admin:** list/create/delete collections; wipe points.
---
## 6. Embedding Subsystem
### 6.1 Registry (`embedding_models.py`)
| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
|----|----------|-----------|----------------------------|---------------|
| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |
`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.
### 6.2 Role resolution (`embedding.py`)
```
resolve_corpus_model(optional_id) → snapshot for process/query/experiment
resolve_boundary_model(optional_id) → snapshot for semantic cuts
get_corpus_embedding_model() / get_boundary_embedding_model()
→ app_settings → registry → DEFAULT_CLOUD_MODEL_ID
```
Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.
### 6.3 embed_texts / embed_single
1. `apply_task_prefixes(texts, model, purpose)` if needed
2. Batch: OpenAI 2048 / Ollama 64
3. Route to `get_openai_client()` or `get_ollama_client()` by Provider
4. Raise `EmbeddingError` on failure
**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).
---
## 7. Documents LLD
### 7.1 Upload
```
upload_document(filename, bytes)
→ temp file → parse_document(path)
→ db.save_document(parsed_text=markdown, document_tree=JSON)
```
**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).
**PDF path (`pdf_parser.py`):**
1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF
2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)
3. Table Flattening to sequential plain text
4. Emit markdown `#` / `##` + DocumentTree
### 7.2 Process
```
ProcessRequest:
strategies: list[StrategyName] # default all five
boundary_model_id: str | None
corpus_model_id: str | None
process_document → chunking.service.run_strategies(...)
→ ProcessResponse(completed[], failed[], corpus_*, boundary_*)
```
### 7.3 Delete
Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).
---
## 8. Chunking LLD
### 8.1 Interface
```python
class ChunkingStrategy(ABC):
name: StrategyName
@abstractmethod
def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
...
```
Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).
### 8.2 Orchestrator (`run_strategies`)
```
corpus = resolve_corpus_model(corpus_model_id)
boundary = resolve_boundary_model(...) if any(semantic*) else unused
for strategy in strategies:
try:
ensure_collection(strategy, corpus.id, corpus.dimension)
chunks = _chunk_document(...) # inject Boundary embeds if needed
texts = enriched_content or text
embeddings = embed_texts(texts, model=corpus, purpose="document")
upsert_chunks(chunks, embeddings, model_id=corpus.id)
except → failed[] (others continue)
update_chunk_counts + update_process_embedding_provenance
```
### 8.3 Strategy algorithms (summary)
| Strategy | Input units | Boundary | Output |
|----------|-------------|----------|--------|
| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |
Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.
**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.
Deep dive: [strategy-technical-details.md](strategy-technical-details.md).
---
## 9. Query LLD
### 9.1 `run_query` pipeline
```
1. Load document (filename for filter)
2. Resolve Corpus model
3. embed_single(question, purpose="query")
4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
6. _build_context (parent scroll for semantic_parent_child)
7. _generate_answer (settings.llm_model, temp 0)
8. db.save_query(...)
```
### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)
**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.
1. Build Expansion Tree nodes for each top-k hit (score order)
2. Collect needed `chunk_index` values: hit±prev/next (skip &lt; 0)
3. Fetch missing indices via Qdrant
4. Fill `neighbors_prev` / `neighbors_next` per hit
5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`
6. Neighbors have `role="neighbor"`, `score=None`
Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).
### 9.3 Parent-child context
For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.
---
## 10. Benchmark LLD
### 10.1 Questions format
```json
{
"questions": [
{
"id": "q1",
"question": "...",
"expected_answer": "...",
"category": "...",
"difficulty": "..."
}
]
}
```
Load from `questions_file` path or inline `questions` list.
### 10.2 `run_benchmark`
```
embedding_model = resolve_corpus_model(corpus_model_id)
boundary_id = doc.last_boundary_embedding_model_id # if semantic in strategies
for question in questions:
for strategy in strategies:
result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
scores = evaluate_single(question, retrieved_chunks, expected, generated)
append per_question row
aggregate_metrics[strategy] = averages + hallucination_rate
best_strategy = argmax (e.g. answer_similarity / composite — see service)
save experiment with provenance + benchmark_config
```
### 10.3 Evaluation (`evaluate_single`)
LLM returns JSON:
```json
{
"context_relevance": 1-10,
"answer_similarity": 1-10,
"faithfulness": 1-10,
"hallucination": true|false,
"reasoning": "..."
}
```
`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).
### 10.4 Reports
`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).
### 10.5 Cost estimate
`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.
---
## 11. Admin LLD
| Endpoint | Behavior |
|----------|----------|
| `GET /admin/health` | App + Qdrant + SQLite ping |
| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
| `PUT .../corpus` · `.../boundary` | Set role defaults |
| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
| `POST /admin/cost-estimate` | Same heuristic as dry-run |
Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).
---
## 12. Dashboard LLD (behavioral)
| Concern | Design |
|---------|--------|
| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
| Theme | Dark `#111113`… + amber accent (ADR-0008) |
| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |
No client router or global store — props from root state.
---
## 13. API Surface (concise)
| Method | Path | Service |
|--------|------|---------|
| GET/POST | `/documents` | list / upload |
| POST | `/documents/{id}/process` | Strategies |
| DELETE | `/documents/{id}` | delete |
| GET | `/strategies` | catalog |
| POST | `/queries` | query |
| GET | `/queries/{id}` | history |
| POST | `/benchmarks` | Experiment or dry_run |
| GET | `/benchmarks/{id}` | Experiment detail |
| GET | `/benchmarks/{id}/report` | HTML |
| GET | `/experiments` | list (+ filters used by Decision Board) |
| * | `/admin/*` | ops |
| GET | `/app/` | Dashboard |
Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.
**Request highlights**
```python
# Process
boundary_model_id: Optional[str]
corpus_model_id: Optional[str]
# Query / Benchmark
top_k: int = 5
neighbor_prev / neighbor_next: int = 0..5
corpus_model_id: Optional[str]
```
---
## 14. Error Model
| Exception | Typical cause | HTTP |
|-----------|---------------|------|
| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
| `EmbeddingError` | Provider/API failure | 400 |
| `QdrantError` | Collection/upsert/search failure | 400 |
| `QueryError` | Missing doc, LLM answer failure | 400 |
| `BenchmarkError` | Bad questions file, eval failure | 400 |
Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.
---
## 15. Sequence Diagrams
### Process (one Strategy)
```
Client → DocumentsAPI → DocumentService → ChunkingService
ChunkingService → resolve Boundary/Corpus
ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
ChunkingService → embed_texts (Corpus)
ChunkingService → Qdrant.upsert
ChunkingService → SQLite.update counts + provenance
Client ← ProcessResponse
```
### Query with Neighbor Expansion
```
Client → QueryAPI → run_query
→ embed_single (Corpus)
→ Qdrant.search top-k
→ apply_neighbor_expansion → get_chunks_by_indices
→ _build_context → OpenAI chat
→ SQLite.save_query
Client ← QueryResponse (retrieved_chunks + expansion_tree)
```
---
## 16. Extension Points
| Extension | Steps |
|-----------|-------|
| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
| New Admin op | Prefer `/admin` only when domain routers lack the capability |
---
## 17. Scripts
`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).
---
## 18. Related Documents
| Doc | Role |
|-----|------|
| [HLD.md](HLD.md) | System context and component view |
| [data-flow.md](data-flow.md) | Narrative pipelines |
| [configuration.md](configuration.md) | Env knobs |
| [adr/](adr/) | Decision records |
| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |

18
docs/README.md
View File

@@ -6,12 +6,12 @@ Complete documentation for the RAG Chunking Benchmarker.
## Quick Start ## Quick Start
1. **New to the project?** Start with [Architecture Overview](architecture.md) 1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail)
2. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md) 2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md)
3. **Need to use the API?** Check [API Reference](api-reference.md) 3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
4. **Configuring the system?** See [Configuration Guide](configuration.md) 4. **Need to use the API?** Check [API Reference](api-reference.md)
5. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md) 5. **Configuring the system?** See [Configuration Guide](configuration.md)
6. **Curious about data flow?** See [Data Flow](data-flow.md) 6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)
--- ---
@@ -19,13 +19,17 @@ Complete documentation for the RAG Chunking Benchmarker.
| File | Purpose | Audience | | File | Purpose | Audience |
|------|---------|----------| |------|---------|----------|
| [architecture.md](architecture.md) | System structure and design | New team members | | [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members |
| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers |
| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members |
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers | | [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
| [api-reference.md](api-reference.md) | All endpoints documented | Developers | | [api-reference.md](api-reference.md) | All endpoints documented | Developers |
| [configuration.md](configuration.md) | Settings and environment variables | DevOps | | [configuration.md](configuration.md) | Settings and environment variables | DevOps |
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists | | [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers | | [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone | | [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators |
| [adr/](adr/) | Architectural Decision Records | Everyone |
| [phases.md](phases.md) | Implementation phases | Project managers | | [phases.md](phases.md) | Implementation phases | Project managers |
| [tasks.md](tasks.md) | Task tracking | Developers | | [tasks.md](tasks.md) | Task tracking | Developers |