docs: record embedding, neighbor, and inspect decisions

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-10 14:13:07 +03:30
parent bdf0c36e20
commit 1401e46c9a
10 changed files with 254 additions and 4 deletions

View File

@@ -0,0 +1,22 @@
# ADR-0022: Per-Embedding-Model semantic_threshold
## Status
Approved
## Context
Semantic Boundary Detection cuts when adjacent-unit cosine similarity falls below a threshold. Cloud OpenAI and local Nomic produce different similarity distributions for the same Farsi text: the global `SEMANTIC_THRESHOLD` (config, historically ~0.3) rarely triggers cuts under Nomic, collapsing `semantic` into a single chunk. Operators need a higher Nomic default without changing OpenAI behavior, and a way to tune without editing `.env` and restarting.
## Decision
- Each Embedding Model Registry entry has a `default_semantic_threshold` (OpenAI `text-embedding-3-small`: **0.3**; Nomic `nomic-embed-text-v2-moe`: **0.6`).
- Admin may override the effective value per model id in SQLite (`semantic_threshold:{model_id}`).
- Process snapshots the Active Embedding Model and uses that model’s effective threshold for `semantic` and `semantic_parent_child`.
- Admin API: list includes `semantic_threshold` / `default_semantic_threshold`; `PUT /admin/embedding-models/{id}/semantic-threshold` persists overrides.
- Changing the threshold does not rewrite existing corpora — re-process to apply.
## Consequences
- Global `SEMANTIC_THRESHOLD` remains a fallback only when a strategy is called without an explicit threshold (tests/legacy); production process path always passes the model-resolved value.
- Operators must re-process after tuning; Experiments under different thresholds are not auto-invalidated.