chunking_strategies_evaluation/docs/LLD.md

# Low-Level Design (LLD)

RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.

**Audience:** implementers  
**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)

---

## 1. Package Map

```
src/
├── main.py                      # App factory, middleware, router mount
├── static/index.html            # Dashboard (React + Babel CDN)
├── core/
│   ├── config.py                # Settings (.env)
│   ├── models.py                # Chunk, DocumentTree, StrategyName
│   ├── exceptions.py            # Domain errors + handlers
│   └── dependencies.py          # OpenAI / Ollama / Qdrant singletons
├── storage/
│   ├── sqlite.py                # documents, queries, experiments, app_settings
│   └── qdrant.py                # Model Corpus CRUD + search
├── documents/
│   ├── parser.py                # DOCX / .doc (+ LibreOffice)
│   ├── pdf_parser.py            # Text PDF + Text-layer Gate
│   ├── heading_heuristics.py    # PDF Heading Reconstruction helpers
│   ├── service.py               # Upload / process / delete
│   ├── routes.py
│   └── models.py                # ProcessRequest, DocumentResponse, …
├── chunking/
│   ├── base.py                  # ChunkingStrategy ABC, tokens, sentences
│   ├── embedding.py             # Boundary/Corpus resolve + embed_texts
│   ├── embedding_models.py      # Registry, thresholds, task prefixes
│   ├── service.py               # run_strategies orchestration
│   └── strategies/
│       ├── fixed_size.py
│       ├── recursive.py
│       ├── semantic.py
│       ├── contextual_retrieval.py
│       └── semantic_parent_child.py
├── benchmarking/
│   ├── query_service.py         # Query + Neighbor Expansion
│   ├── benchmark_service.py     # Experiment runner + cost estimate
│   ├── evaluation.py            # LLM-as-Judge
│   ├── report.py                # HTML reports
│   ├── routes.py
│   └── models.py
└── admin/
    ├── routes.py                # /admin/*
    └── service.py
```

---

## 2. Application Bootstrap

**Entry:** `create_app()` in `src/main.py`

1. `init_db()` — create/migrate SQLite schema  
2. Register CORS + `RequestLoggingMiddleware`  
3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`  
4. Routers: documents, benchmarking, admin  
5. Mount `StaticFiles` at `/app` → `src/static/`

```python
# Singletons (lru_cache)
get_openai_client()   # Cloud LLM + cloud embeds
get_ollama_client()   # OpenAI-compatible client → Ollama base URL
get_qdrant_client()
```

---

## 3. Configuration

`Settings` (`src/core/config.py`) — selected fields:

| Field | Default | Use |
|-------|---------|-----|
| `openai_api_key` | required | Cloud Provider |
| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
| `llm_model` | `gpt-4o-mini` | Answer + judge |
| `ollama_base_url` | host URL | Local embeds |
| `qdrant_url` | `http://localhost:6333` | Vectors |
| `top_k` | `5` | Retrieval |
| `neighbor_prev` / `neighbor_next` | `3` / `3` | Expansion defaults (fixed_size ±3) |
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
| `semantic_threshold` | `0.3` | Fallback if model default missing |
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |

---

## 4. Core Domain Models

### 4.1 StrategyName

```python
class StrategyName(str, Enum):
    RECURSIVE = "recursive"
    FIXED_SIZE = "fixed_size"
    SEMANTIC = "semantic"
    CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
    SEMANTIC_PARENT_CHILD = "semantic_parent_child"
```

### 4.2 Chunk (unified)

| Field | Type | Notes |
|-------|------|-------|
| `document_name` | str | Source filename |
| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
| `strategy_name` | StrategyName | |
| `chunk_index` | int | Document order (used by Neighbor Expansion) |
| `text` | str | Stored in Qdrant; used for LLM context |
| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
| `parent_id` | str \| None | Parent-child Strategy |
| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |

`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).

### 4.3 DocumentTree

```
DocumentTree
  └── root: DocumentTreeNode
        node_type: document | section | article | paragraph
        text, heading, heading_level, children[]
```

Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.

---

## 5. Storage LLD

### 5.1 SQLite schema

**documents**

| Column | Type | Purpose |
|--------|------|---------|
| `id` | TEXT PK | UUID hex |
| `filename` | TEXT | Original name |
| `parsed_text` | TEXT | Markdown for chunking |
| `document_tree` | TEXT | JSON tree |
| `chunk_counts` | TEXT | JSON `{strategy: count}` |
| `last_corpus_embedding_model_id` | TEXT | Provenance |
| `last_boundary_embedding_model_id` | TEXT | Provenance |
| `created_at` | TEXT | ISO UTC |

**queries**

| Column | Purpose |
|--------|---------|
| `id`, `document_id`, `strategy_name` | Identity |
| `question`, `answer` | Content |
| `retrieved_chunks` | Flat LLM/eval list (JSON) |
| `expansion_tree` | Per-hit neighbors (JSON) |
| `latency_breakdown`, `token_usage` | Observability JSON |
| `created_at` | |

**experiments**

| Column | Purpose |
|--------|---------|
| `id`, `document_id` | Identity |
| `benchmark_config` | top_k, neighbor_prev/next, etc. |
| `questions`, `per_question` | Inputs + per-cell results |
| `aggregate_metrics` | Per-Strategy averages |
| `strategies_used` | JSON list |
| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
| `created_at` | |

**app_settings** — key/value for:

- `corpus_embedding_model_id`, `boundary_embedding_model_id`
- Legacy `active_embedding_model_id` (migrated into both roles)
- `semantic_threshold:{model_id}` overrides

Connection: WAL mode, foreign keys ON, one connection per call (no pool).

### 5.2 Qdrant

**Collection name**

```python
def collection_name(strategy, model_id) -> str:
    return f"{strategy}__{sanitize(model_id)}_collection"
# sanitize: replace : and / with -
```

**Point**

| Field | Value |
|-------|-------|
| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
| `payload` | ChunkMetadata fields |

**Search**

- Cosine distance  
- Optional filter: `document_name == filename`  
- Returns payload + score  

**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.

**Admin:** list/create/delete collections; wipe points.

---

## 6. Embedding Subsystem

### 6.1 Registry (`embedding_models.py`)

| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
|----|----------|-----------|----------------------------|---------------|
| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |

`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.

### 6.2 Role resolution (`embedding.py`)

```
resolve_corpus_model(optional_id)   → snapshot for process/query/experiment
resolve_boundary_model(optional_id) → snapshot for semantic cuts

get_corpus_embedding_model() / get_boundary_embedding_model()
  → app_settings → registry → DEFAULT_CLOUD_MODEL_ID
```

Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.

### 6.3 embed_texts / embed_single

1. `apply_task_prefixes(texts, model, purpose)` if needed  
2. Batch: OpenAI 2048 / Ollama 64  
3. Route to `get_openai_client()` or `get_ollama_client()` by Provider  
4. Raise `EmbeddingError` on failure  

**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).

---

## 7. Documents LLD

### 7.1 Upload

```
upload_document(filename, bytes)
  → temp file → parse_document(path)
  → db.save_document(parsed_text=markdown, document_tree=JSON)
```

**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).

**PDF path (`pdf_parser.py`):**

1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF  
2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)  
3. Table Flattening to sequential plain text  
4. Emit markdown `#` / `##` + DocumentTree  

### 7.2 Process

```
ProcessRequest:
  strategies: list[StrategyName]  # default all five
  boundary_model_id: str | None
  corpus_model_id: str | None

process_document → chunking.service.run_strategies(...)
  → ProcessResponse(completed[], failed[], corpus_*, boundary_*)
```

### 7.3 Delete

Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).

---

## 8. Chunking LLD

### 8.1 Interface

```python
class ChunkingStrategy(ABC):
    name: StrategyName

    @abstractmethod
    def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
        ...
```

Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).

### 8.2 Orchestrator (`run_strategies`)

```
corpus = resolve_corpus_model(corpus_model_id)
boundary = resolve_boundary_model(...) if any(semantic*) else unused

for strategy in strategies:
  try:
    ensure_collection(strategy, corpus.id, corpus.dimension)
    chunks = _chunk_document(...)   # inject Boundary embeds if needed
    texts = enriched_content or text
    embeddings = embed_texts(texts, model=corpus, purpose="document")
    upsert_chunks(chunks, embeddings, model_id=corpus.id)
  except → failed[] (others continue)

update_chunk_counts + update_process_embedding_provenance
```

### 8.3 Strategy algorithms (summary)

| Strategy | Input units | Boundary | Output |
|----------|-------------|----------|--------|
| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |

Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.

**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.

Deep dive: [strategy-technical-details.md](strategy-technical-details.md).

---

## 9. Query LLD

### 9.1 `run_query` pipeline

```
1. Load document (filename for filter)
2. Resolve Corpus model
3. embed_single(question, purpose="query")
4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
6. _build_context (parent scroll for semantic_parent_child)
7. _generate_answer (settings.llm_model, temp 0)
8. db.save_query(...)
```

### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)

**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.

1. Build Expansion Tree nodes for each top-k hit (score order)  
2. Collect needed `chunk_index` values: hit±prev/next (skip < 0)  
3. Fetch missing indices via Qdrant  
4. Fill `neighbors_prev` / `neighbors_next` per hit  
5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`  
6. Neighbors have `role="neighbor"`, `score=None`  

Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).

### 9.3 Parent-child context

For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.

---

## 10. Benchmark LLD

### 10.1 Questions format

```json
{
  "questions": [
    {
      "id": "q1",
      "question": "...",
      "expected_answer": "...",
      "category": "...",
      "difficulty": "..."
    }
  ]
}
```

Load from `questions_file` path or inline `questions` list.

### 10.2 `run_benchmark`

```
embedding_model = resolve_corpus_model(corpus_model_id)
boundary_id = doc.last_boundary_embedding_model_id  # if semantic in strategies

for question in questions:
  for strategy in strategies:
    result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
    scores = evaluate_single(question, retrieved_chunks, expected, generated)
    append per_question row

aggregate_metrics[strategy] = averages + hallucination_rate
best_strategy = argmax (e.g. answer_similarity / composite — see service)
save experiment with provenance + benchmark_config
```

### 10.3 Evaluation (`evaluate_single`)

LLM returns JSON:

```json
{
  "context_relevance": 1-10,
  "answer_similarity": 1-10,
  "faithfulness": 1-10,
  "hallucination": true|false,
  "reasoning": "..."
}
```

`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).

### 10.4 Reports

`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).

### 10.5 Cost estimate

`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.

---

## 11. Admin LLD

| Endpoint | Behavior |
|----------|----------|
| `GET /admin/health` | App + Qdrant + SQLite ping |
| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
| `PUT .../corpus` · `.../boundary` | Set role defaults |
| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
| `POST /admin/cost-estimate` | Same heuristic as dry-run |

Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).

---

## 12. Dashboard LLD (behavioral)

| Concern | Design |
|---------|--------|
| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
| Theme | Dark `#111113`… + amber accent (ADR-0008) |
| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |

No client router or global store — props from root state.

---

## 13. API Surface (concise)

| Method | Path | Service |
|--------|------|---------|
| GET/POST | `/documents` | list / upload |
| POST | `/documents/{id}/process` | Strategies |
| DELETE | `/documents/{id}` | delete |
| GET | `/strategies` | catalog |
| POST | `/queries` | query |
| GET | `/queries/{id}` | history |
| POST | `/benchmarks` | Experiment or dry_run |
| GET | `/benchmarks/{id}` | Experiment detail |
| GET | `/benchmarks/{id}/report` | HTML |
| GET | `/experiments` | list (+ filters used by Decision Board) |
| * | `/admin/*` | ops |
| GET | `/app/` | Dashboard |

Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.

**Request highlights**

```python
# Process
boundary_model_id: Optional[str]
corpus_model_id: Optional[str]

# Query / Benchmark
top_k: int = 5
neighbor_prev / neighbor_next: int = 0..5
corpus_model_id: Optional[str]
```

---

## 14. Error Model

| Exception | Typical cause | HTTP |
|-----------|---------------|------|
| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
| `EmbeddingError` | Provider/API failure | 400 |
| `QdrantError` | Collection/upsert/search failure | 400 |
| `QueryError` | Missing doc, LLM answer failure | 400 |
| `BenchmarkError` | Bad questions file, eval failure | 400 |

Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.

---

## 15. Sequence Diagrams

### Process (one Strategy)

```
Client → DocumentsAPI → DocumentService → ChunkingService
  ChunkingService → resolve Boundary/Corpus
  ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
  ChunkingService → embed_texts (Corpus)
  ChunkingService → Qdrant.upsert
  ChunkingService → SQLite.update counts + provenance
Client ← ProcessResponse
```

### Query with Neighbor Expansion

```
Client → QueryAPI → run_query
  → embed_single (Corpus)
  → Qdrant.search top-k
  → apply_neighbor_expansion → get_chunks_by_indices
  → _build_context → OpenAI chat
  → SQLite.save_query
Client ← QueryResponse (retrieved_chunks + expansion_tree)
```

---

## 16. Extension Points

| Extension | Steps |
|-----------|-------|
| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
| New Admin op | Prefer `/admin` only when domain routers lack the capability |

---

## 17. Scripts

`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).

---

## 18. Related Documents

| Doc | Role |
|-----|------|
| [HLD.md](HLD.md) | System context and component view |
| [data-flow.md](data-flow.md) | Narrative pipelines |
| [configuration.md](configuration.md) | Env knobs |
| [adr/](adr/) | Decision records |
| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |