docs: add high- and low-level design
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
571
docs/LLD.md
Normal file
571
docs/LLD.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# Low-Level Design (LLD)
|
||||
|
||||
RAG Chunking Benchmarker — module interfaces, data schemas, algorithms, and API contracts.
|
||||
|
||||
**Audience:** implementers
|
||||
**Companion:** [HLD](HLD.md) · [API Reference](api-reference.md) · [CONTEXT.md](../CONTEXT.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Package Map
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.py # App factory, middleware, router mount
|
||||
├── static/index.html # Dashboard (React + Babel CDN)
|
||||
├── core/
|
||||
│ ├── config.py # Settings (.env)
|
||||
│ ├── models.py # Chunk, DocumentTree, StrategyName
|
||||
│ ├── exceptions.py # Domain errors + handlers
|
||||
│ └── dependencies.py # OpenAI / Ollama / Qdrant singletons
|
||||
├── storage/
|
||||
│ ├── sqlite.py # documents, queries, experiments, app_settings
|
||||
│ └── qdrant.py # Model Corpus CRUD + search
|
||||
├── documents/
|
||||
│ ├── parser.py # DOCX / .doc (+ LibreOffice)
|
||||
│ ├── pdf_parser.py # Text PDF + Text-layer Gate
|
||||
│ ├── heading_heuristics.py # PDF Heading Reconstruction helpers
|
||||
│ ├── service.py # Upload / process / delete
|
||||
│ ├── routes.py
|
||||
│ └── models.py # ProcessRequest, DocumentResponse, …
|
||||
├── chunking/
|
||||
│ ├── base.py # ChunkingStrategy ABC, tokens, sentences
|
||||
│ ├── embedding.py # Boundary/Corpus resolve + embed_texts
|
||||
│ ├── embedding_models.py # Registry, thresholds, task prefixes
|
||||
│ ├── service.py # run_strategies orchestration
|
||||
│ └── strategies/
|
||||
│ ├── fixed_size.py
|
||||
│ ├── recursive.py
|
||||
│ ├── semantic.py
|
||||
│ ├── contextual_retrieval.py
|
||||
│ └── semantic_parent_child.py
|
||||
├── benchmarking/
|
||||
│ ├── query_service.py # Query + Neighbor Expansion
|
||||
│ ├── benchmark_service.py # Experiment runner + cost estimate
|
||||
│ ├── evaluation.py # LLM-as-Judge
|
||||
│ ├── report.py # HTML reports
|
||||
│ ├── routes.py
|
||||
│ └── models.py
|
||||
└── admin/
|
||||
├── routes.py # /admin/*
|
||||
└── service.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Application Bootstrap
|
||||
|
||||
**Entry:** `create_app()` in `src/main.py`
|
||||
|
||||
1. `init_db()` — create/migrate SQLite schema
|
||||
2. Register CORS + `RequestLoggingMiddleware`
|
||||
3. Exception handlers: `ChunkingError`, `BenchmarkError`, `QueryError`
|
||||
4. Routers: documents, benchmarking, admin
|
||||
5. Mount `StaticFiles` at `/app` → `src/static/`
|
||||
|
||||
```python
|
||||
# Singletons (lru_cache)
|
||||
get_openai_client() # Cloud LLM + cloud embeds
|
||||
get_ollama_client() # OpenAI-compatible client → Ollama base URL
|
||||
get_qdrant_client()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
`Settings` (`src/core/config.py`) — selected fields:
|
||||
|
||||
| Field | Default | Use |
|
||||
|-------|---------|-----|
|
||||
| `openai_api_key` | required | Cloud Provider |
|
||||
| `embedding_model` | `text-embedding-3-large` | Config legacy; runtime uses Registry + Admin |
|
||||
| `llm_model` | `gpt-4o-mini` | Answer + judge |
|
||||
| `ollama_base_url` | host URL | Local embeds |
|
||||
| `qdrant_url` | `http://localhost:6333` | Vectors |
|
||||
| `top_k` | `5` | Retrieval |
|
||||
| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults |
|
||||
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
|
||||
| `semantic_threshold` | `0.3` | Fallback if model default missing |
|
||||
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
|
||||
| `database_url` | `sqlite:///./data/chunking_benchmark.db` | Metadata |
|
||||
| `pdf_min_total_chars` / `pdf_min_median_chars_per_page` | `100` / `40` | Text-layer Gate |
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Domain Models
|
||||
|
||||
### 4.1 StrategyName
|
||||
|
||||
```python
|
||||
class StrategyName(str, Enum):
|
||||
RECURSIVE = "recursive"
|
||||
FIXED_SIZE = "fixed_size"
|
||||
SEMANTIC = "semantic"
|
||||
CONTEXTUAL_RETRIEVAL = "contextual_retrieval"
|
||||
SEMANTIC_PARENT_CHILD = "semantic_parent_child"
|
||||
```
|
||||
|
||||
### 4.2 Chunk (unified)
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `document_name` | str | Source filename |
|
||||
| `chunk_id` | str | `{strategy}_{safe_doc}_{index:06d}` |
|
||||
| `strategy_name` | StrategyName | |
|
||||
| `chunk_index` | int | Document order (used by Neighbor Expansion) |
|
||||
| `text` | str | Stored in Qdrant; used for LLM context |
|
||||
| `token_count` / `character_count` | int | tiktoken `cl100k_base` |
|
||||
| `parent_id` | str \| None | Parent-child Strategy |
|
||||
| `enriched_content` | str \| None | Contextual retrieval embed text (not always in payload) |
|
||||
|
||||
`chunk_to_metadata()` drops `enriched_content` for Qdrant payload (`ChunkMetadata`).
|
||||
|
||||
### 4.3 DocumentTree
|
||||
|
||||
```
|
||||
DocumentTree
|
||||
└── root: DocumentTreeNode
|
||||
node_type: document | section | article | paragraph
|
||||
text, heading, heading_level, children[]
|
||||
```
|
||||
|
||||
Serialized as JSON in SQLite `documents.document_tree`. Chunking primarily uses `parsed_text` (markdown); tree supports structure-aware Strategies and preview.
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage LLD
|
||||
|
||||
### 5.1 SQLite schema
|
||||
|
||||
**documents**
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|--------|------|---------|
|
||||
| `id` | TEXT PK | UUID hex |
|
||||
| `filename` | TEXT | Original name |
|
||||
| `parsed_text` | TEXT | Markdown for chunking |
|
||||
| `document_tree` | TEXT | JSON tree |
|
||||
| `chunk_counts` | TEXT | JSON `{strategy: count}` |
|
||||
| `last_corpus_embedding_model_id` | TEXT | Provenance |
|
||||
| `last_boundary_embedding_model_id` | TEXT | Provenance |
|
||||
| `created_at` | TEXT | ISO UTC |
|
||||
|
||||
**queries**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id`, `strategy_name` | Identity |
|
||||
| `question`, `answer` | Content |
|
||||
| `retrieved_chunks` | Flat LLM/eval list (JSON) |
|
||||
| `expansion_tree` | Per-hit neighbors (JSON) |
|
||||
| `latency_breakdown`, `token_usage` | Observability JSON |
|
||||
| `created_at` | |
|
||||
|
||||
**experiments**
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `id`, `document_id` | Identity |
|
||||
| `benchmark_config` | top_k, neighbor_prev/next, etc. |
|
||||
| `questions`, `per_question` | Inputs + per-cell results |
|
||||
| `aggregate_metrics` | Per-Strategy averages |
|
||||
| `strategies_used` | JSON list |
|
||||
| `embedding_model_id`, `embedding_provider` | Corpus snapshot |
|
||||
| `boundary_embedding_model_id` | Boundary snapshot (nullable) |
|
||||
| `created_at` | |
|
||||
|
||||
**app_settings** — key/value for:
|
||||
|
||||
- `corpus_embedding_model_id`, `boundary_embedding_model_id`
|
||||
- Legacy `active_embedding_model_id` (migrated into both roles)
|
||||
- `semantic_threshold:{model_id}` overrides
|
||||
|
||||
Connection: WAL mode, foreign keys ON, one connection per call (no pool).
|
||||
|
||||
### 5.2 Qdrant
|
||||
|
||||
**Collection name**
|
||||
|
||||
```python
|
||||
def collection_name(strategy, model_id) -> str:
|
||||
return f"{strategy}__{sanitize(model_id)}_collection"
|
||||
# sanitize: replace : and / with -
|
||||
```
|
||||
|
||||
**Point**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `id` | `uuid5(NAMESPACE_URL, chunk_id)` |
|
||||
| `vector` | Corpus embedding (dim from model: 1536 / 3072 / 768) |
|
||||
| `payload` | ChunkMetadata fields |
|
||||
|
||||
**Search**
|
||||
|
||||
- Cosine distance
|
||||
- Optional filter: `document_name == filename`
|
||||
- Returns payload + score
|
||||
|
||||
**Neighbor fetch:** `get_chunks_by_indices(strategy, document_name, indices, model_id)` for Expansion.
|
||||
|
||||
**Admin:** list/create/delete collections; wipe points.
|
||||
|
||||
---
|
||||
|
||||
## 6. Embedding Subsystem
|
||||
|
||||
### 6.1 Registry (`embedding_models.py`)
|
||||
|
||||
| id | Provider | Dimension | Default semantic_threshold | Task prefixes |
|
||||
|----|----------|-----------|----------------------------|---------------|
|
||||
| `text-embedding-3-small` | cloud | 1536 | 0.3 | no |
|
||||
| `text-embedding-3-large` | cloud | 3072 | 0.3 | no (default Admin) |
|
||||
| `nomic-embed-text-v2-moe` | local | 768 | 0.6 | yes (`search_document:` / `search_query:`) |
|
||||
|
||||
`EmbeddingModelSpec`: `id`, `provider`, `model_name`, `dimension`, `display_name`, `task_prefixes`, `default_semantic_threshold`.
|
||||
|
||||
### 6.2 Role resolution (`embedding.py`)
|
||||
|
||||
```
|
||||
resolve_corpus_model(optional_id) → snapshot for process/query/experiment
|
||||
resolve_boundary_model(optional_id) → snapshot for semantic cuts
|
||||
|
||||
get_corpus_embedding_model() / get_boundary_embedding_model()
|
||||
→ app_settings → registry → DEFAULT_CLOUD_MODEL_ID
|
||||
```
|
||||
|
||||
Admin switches persist to SQLite; mid-flight Admin changes do not affect an in-progress operation that already resolved snapshots.
|
||||
|
||||
### 6.3 embed_texts / embed_single
|
||||
|
||||
1. `apply_task_prefixes(texts, model, purpose)` if needed
|
||||
2. Batch: OpenAI 2048 / Ollama 64
|
||||
3. Route to `get_openai_client()` or `get_ollama_client()` by Provider
|
||||
4. Raise `EmbeddingError` on failure
|
||||
|
||||
**Threshold:** `get_semantic_threshold(model_id)` = Admin override → registry default. Boundary Strategy cuts **always** use Boundary model's threshold (never Corpus).
|
||||
|
||||
---
|
||||
|
||||
## 7. Documents LLD
|
||||
|
||||
### 7.1 Upload
|
||||
|
||||
```
|
||||
upload_document(filename, bytes)
|
||||
→ temp file → parse_document(path)
|
||||
→ db.save_document(parsed_text=markdown, document_tree=JSON)
|
||||
```
|
||||
|
||||
**Supported suffixes:** `.docx`, `.doc`, `.pdf` (`SUPPORTED_SUFFIXES`).
|
||||
|
||||
**PDF path (`pdf_parser.py`):**
|
||||
|
||||
1. Text-layer Gate (`pdf_min_total_chars`, `pdf_min_median_chars_per_page`) — reject Scanned PDF
|
||||
2. Extract text + Heading Reconstruction (outline → font → Farsi/English heuristics)
|
||||
3. Table Flattening to sequential plain text
|
||||
4. Emit markdown `#` / `##` + DocumentTree
|
||||
|
||||
### 7.2 Process
|
||||
|
||||
```
|
||||
ProcessRequest:
|
||||
strategies: list[StrategyName] # default all five
|
||||
boundary_model_id: str | None
|
||||
corpus_model_id: str | None
|
||||
|
||||
process_document → chunking.service.run_strategies(...)
|
||||
→ ProcessResponse(completed[], failed[], corpus_*, boundary_*)
|
||||
```
|
||||
|
||||
### 7.3 Delete
|
||||
|
||||
Deletes SQLite row (cascade queries/experiments) and Qdrant points for that document across known collections (implementation in `documents/service.py` + `qdrant`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Chunking LLD
|
||||
|
||||
### 8.1 Interface
|
||||
|
||||
```python
|
||||
class ChunkingStrategy(ABC):
|
||||
name: StrategyName
|
||||
|
||||
@abstractmethod
|
||||
def chunk(*, doc_name: str, tree: DocumentTree, markdown: str) -> list[Chunk]:
|
||||
...
|
||||
```
|
||||
|
||||
Semantic Strategies accept extra kwargs from orchestrator (`sentence_embeddings` / `paragraph_embeddings`, `semantic_threshold`).
|
||||
|
||||
### 8.2 Orchestrator (`run_strategies`)
|
||||
|
||||
```
|
||||
corpus = resolve_corpus_model(corpus_model_id)
|
||||
boundary = resolve_boundary_model(...) if any(semantic*) else unused
|
||||
|
||||
for strategy in strategies:
|
||||
try:
|
||||
ensure_collection(strategy, corpus.id, corpus.dimension)
|
||||
chunks = _chunk_document(...) # inject Boundary embeds if needed
|
||||
texts = enriched_content or text
|
||||
embeddings = embed_texts(texts, model=corpus, purpose="document")
|
||||
upsert_chunks(chunks, embeddings, model_id=corpus.id)
|
||||
except → failed[] (others continue)
|
||||
|
||||
update_chunk_counts + update_process_embedding_provenance
|
||||
```
|
||||
|
||||
### 8.3 Strategy algorithms (summary)
|
||||
|
||||
| Strategy | Input units | Boundary | Output |
|
||||
|----------|-------------|----------|--------|
|
||||
| **fixed_size** | markdown tokens | none | sliding window `chunk_size` / `chunk_overlap` |
|
||||
| **recursive** | markdown | cascade `#` → `\n\n` → `\n` → punct → space | merge up to target size |
|
||||
| **semantic** | sentences (`split_sentences`, Farsi-aware) | adjacent cosine vs Boundary threshold; `semantic_min_chunk_size` | joined sentence groups |
|
||||
| **contextual_retrieval** | base chunks + LLM context prefix | none | `text` = original; `enriched_content` = prefix+text for embed |
|
||||
| **semantic_parent_child** | paragraphs | paragraph cosine vs Boundary threshold | parents + children with `parent_id` |
|
||||
|
||||
Semantic Strategies **fail hard** if Boundary embeddings missing/mismatched (ADR-0020) — no fixed-count fallback.
|
||||
|
||||
**Sentence split fallback:** punctuation → non-empty lines → blank-line paragraphs → single unit.
|
||||
|
||||
Deep dive: [strategy-technical-details.md](strategy-technical-details.md).
|
||||
|
||||
---
|
||||
|
||||
## 9. Query LLD
|
||||
|
||||
### 9.1 `run_query` pipeline
|
||||
|
||||
```
|
||||
1. Load document (filename for filter)
|
||||
2. Resolve Corpus model
|
||||
3. embed_single(question, purpose="query")
|
||||
4. qdr.search(strategy, vector, top_k, document_filter=filename, model_id)
|
||||
5. apply_neighbor_expansion(...) → retrieved_chunks, expansion_tree
|
||||
6. _build_context (parent scroll for semantic_parent_child)
|
||||
7. _generate_answer (settings.llm_model, temp 0)
|
||||
8. db.save_query(...)
|
||||
```
|
||||
|
||||
### 9.2 Neighbor Expansion (`apply_neighbor_expansion`)
|
||||
|
||||
**Applies only when** `strategy == fixed_size` and `(neighbor_prev > 0 or neighbor_next > 0)`.
|
||||
|
||||
1. Build Expansion Tree nodes for each top-k hit (score order)
|
||||
2. Collect needed `chunk_index` values: hit±prev/next (skip < 0)
|
||||
3. Fetch missing indices via Qdrant
|
||||
4. Fill `neighbors_prev` / `neighbors_next` per hit
|
||||
5. Flat list: hits + neighbors, **dedupe by chunk_id** (prefer hit), sort by `chunk_index`
|
||||
6. Neighbors have `role="neighbor"`, `score=None`
|
||||
|
||||
Knobs ignored for non-`fixed_size` Strategies (tree still returns hits with empty neighbor arrays).
|
||||
|
||||
### 9.3 Parent-child context
|
||||
|
||||
For `semantic_parent_child`, scroll Qdrant by `chunk_id == parent_id` and append parent text under each child in the prompt context.
|
||||
|
||||
---
|
||||
|
||||
## 10. Benchmark LLD
|
||||
|
||||
### 10.1 Questions format
|
||||
|
||||
```json
|
||||
{
|
||||
"questions": [
|
||||
{
|
||||
"id": "q1",
|
||||
"question": "...",
|
||||
"expected_answer": "...",
|
||||
"category": "...",
|
||||
"difficulty": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Load from `questions_file` path or inline `questions` list.
|
||||
|
||||
### 10.2 `run_benchmark`
|
||||
|
||||
```
|
||||
embedding_model = resolve_corpus_model(corpus_model_id)
|
||||
boundary_id = doc.last_boundary_embedding_model_id # if semantic in strategies
|
||||
|
||||
for question in questions:
|
||||
for strategy in strategies:
|
||||
result = run_query(..., neighbor_prev, neighbor_next, embedding_model=...)
|
||||
scores = evaluate_single(question, retrieved_chunks, expected, generated)
|
||||
append per_question row
|
||||
|
||||
aggregate_metrics[strategy] = averages + hallucination_rate
|
||||
best_strategy = argmax (e.g. answer_similarity / composite — see service)
|
||||
save experiment with provenance + benchmark_config
|
||||
```
|
||||
|
||||
### 10.3 Evaluation (`evaluate_single`)
|
||||
|
||||
LLM returns JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"context_relevance": 1-10,
|
||||
"answer_similarity": 1-10,
|
||||
"faithfulness": 1-10,
|
||||
"hallucination": true|false,
|
||||
"reasoning": "..."
|
||||
}
|
||||
```
|
||||
|
||||
`response_format=json_object`, temperature 0. Metrics definitions: [evaluation-metrics.md](evaluation-metrics.md).
|
||||
|
||||
### 10.4 Reports
|
||||
|
||||
`report.py` renders managerial or technical HTML from an Experiment (rankings, KPIs, Expansion Tree samples, token usage).
|
||||
|
||||
### 10.5 Cost estimate
|
||||
|
||||
`estimate_cost(num_questions, num_strategies)` — heuristic USD; Local Corpus → embedding cost 0.
|
||||
|
||||
---
|
||||
|
||||
## 11. Admin LLD
|
||||
|
||||
| Endpoint | Behavior |
|
||||
|----------|----------|
|
||||
| `GET /admin/health` | App + Qdrant + SQLite ping |
|
||||
| `GET /admin/embedding-models` | Registry + Boundary/Corpus defaults + thresholds |
|
||||
| `PUT .../corpus` · `.../boundary` | Set role defaults |
|
||||
| `PUT .../{id}/semantic-threshold` | Persist override `(0, 1]` |
|
||||
| `GET/POST/DELETE /admin/qdrant/collections*` | Collection CRUD + wipe points |
|
||||
| `GET /admin/chunks/{doc_id}?strategy=` | Chunk Preview from Qdrant |
|
||||
| `GET/POST/DELETE /admin/questions*` | Manage `files/*.json` |
|
||||
| `POST /admin/cost-estimate` | Same heuristic as dry-run |
|
||||
|
||||
Admin does **not** duplicate document/query/benchmark domain endpoints (ADR-0006).
|
||||
|
||||
---
|
||||
|
||||
## 12. Dashboard LLD (behavioral)
|
||||
|
||||
| Concern | Design |
|
||||
|---------|--------|
|
||||
| Delivery | One `index.html`; React + ReactDOM + Babel from CDN |
|
||||
| Navigation | Top Tabs; components stay mounted (`useState` on `App`) |
|
||||
| Theme | Dark `#111113`… + amber accent (ADR-0008) |
|
||||
| Word vs PDF | Documents/Query/Benchmarks filter Word; PDF Tab = same sections + format filter |
|
||||
| Retrieval Inspect | Full-page mode in Benchmarks: question rail, Strategy picker, Generated \| Expected, Expansion Tree |
|
||||
| Decision Board | Client-side Candidate discovery from `/experiments`; Corpus filter; exclude bad Experiment ids; two-stage heat comparison |
|
||||
| Neighbor badge | `±P/N` on Experiment list/Compare; mismatch warning across Boundary/Corpus/Neighbor |
|
||||
|
||||
No client router or global store — props from root state.
|
||||
|
||||
---
|
||||
|
||||
## 13. API Surface (concise)
|
||||
|
||||
| Method | Path | Service |
|
||||
|--------|------|---------|
|
||||
| GET/POST | `/documents` | list / upload |
|
||||
| POST | `/documents/{id}/process` | Strategies |
|
||||
| DELETE | `/documents/{id}` | delete |
|
||||
| GET | `/strategies` | catalog |
|
||||
| POST | `/queries` | query |
|
||||
| GET | `/queries/{id}` | history |
|
||||
| POST | `/benchmarks` | Experiment or dry_run |
|
||||
| GET | `/benchmarks/{id}` | Experiment detail |
|
||||
| GET | `/benchmarks/{id}/report` | HTML |
|
||||
| GET | `/experiments` | list (+ filters used by Decision Board) |
|
||||
| * | `/admin/*` | ops |
|
||||
| GET | `/app/` | Dashboard |
|
||||
|
||||
Full schemas: [api-reference.md](api-reference.md), Pydantic models in `documents/models.py` and `benchmarking/models.py`.
|
||||
|
||||
**Request highlights**
|
||||
|
||||
```python
|
||||
# Process
|
||||
boundary_model_id: Optional[str]
|
||||
corpus_model_id: Optional[str]
|
||||
|
||||
# Query / Benchmark
|
||||
top_k: int = 5
|
||||
neighbor_prev / neighbor_next: int = 0..5
|
||||
corpus_model_id: Optional[str]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Error Model
|
||||
|
||||
| Exception | Typical cause | HTTP |
|
||||
|-----------|---------------|------|
|
||||
| `DocumentProcessingError` | Bad file, Text-layer Gate, missing doc | 400 |
|
||||
| `ChunkingError` | Unknown Strategy, semantic embed failure | 400 |
|
||||
| `EmbeddingError` | Provider/API failure | 400 |
|
||||
| `QdrantError` | Collection/upsert/search failure | 400 |
|
||||
| `QueryError` | Missing doc, LLM answer failure | 400 |
|
||||
| `BenchmarkError` | Bad questions file, eval failure | 400 |
|
||||
|
||||
Per-Strategy process failures are returned in `strategies_failed` without aborting the whole request.
|
||||
|
||||
---
|
||||
|
||||
## 15. Sequence Diagrams
|
||||
|
||||
### Process (one Strategy)
|
||||
|
||||
```
|
||||
Client → DocumentsAPI → DocumentService → ChunkingService
|
||||
ChunkingService → resolve Boundary/Corpus
|
||||
ChunkingService → SemanticStrategy.chunk (w/ Boundary embeds)
|
||||
ChunkingService → embed_texts (Corpus)
|
||||
ChunkingService → Qdrant.upsert
|
||||
ChunkingService → SQLite.update counts + provenance
|
||||
Client ← ProcessResponse
|
||||
```
|
||||
|
||||
### Query with Neighbor Expansion
|
||||
|
||||
```
|
||||
Client → QueryAPI → run_query
|
||||
→ embed_single (Corpus)
|
||||
→ Qdrant.search top-k
|
||||
→ apply_neighbor_expansion → get_chunks_by_indices
|
||||
→ _build_context → OpenAI chat
|
||||
→ SQLite.save_query
|
||||
Client ← QueryResponse (retrieved_chunks + expansion_tree)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 16. Extension Points
|
||||
|
||||
| Extension | Steps |
|
||||
|-----------|-------|
|
||||
| New Strategy | Subclass `ChunkingStrategy`; register in `_STRATEGIES`; add `StrategyName`; update Dashboard labels |
|
||||
| New Embedding Model | Add `EmbeddingModelSpec` to registry; ensure dimension matches Qdrant collections |
|
||||
| New eval metric | Extend judge prompt JSON + aggregate in `benchmark_service` + report templates |
|
||||
| New Admin op | Prefer `/admin` only when domain routers lack the capability |
|
||||
|
||||
---
|
||||
|
||||
## 17. Scripts
|
||||
|
||||
`scripts/run_neighbor_sweep.py` — Benchmark Sweep: fixed Strategy + Corpus, steps Neighbor Expansion `(0,0)…(3,3)` across documents (each level = separate Experiment).
|
||||
|
||||
---
|
||||
|
||||
## 18. Related Documents
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| [HLD.md](HLD.md) | System context and component view |
|
||||
| [data-flow.md](data-flow.md) | Narrative pipelines |
|
||||
| [configuration.md](configuration.md) | Env knobs |
|
||||
| [adr/](adr/) | Decision records |
|
||||
| [CONTEXT.md](../CONTEXT.md) | Language and Tab semantics |
|
||||
Reference in New Issue
Block a user