docs: record embedding, neighbor, and inspect decisions

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-10 14:13:07 +03:30
parent bdf0c36e20
commit 1401e46c9a
10 changed files with 254 additions and 4 deletions

View File

@@ -0,0 +1,16 @@
# Semantic strategies require Semantic Boundary Detection
`semantic` and `semantic_parent_child` must cut chunks from meaning: the orchestrator embeds consecutive units (sentences / paragraphs) with the Active Embedding Model snapshot, passes those vectors into `chunk()`, then embeds finished chunks for Qdrant. Fixed-count fallbacks are removed — missing or length-mismatched boundary embeddings fail that Strategy. This completes ADR 0012’s sentence-level design in the process path (and the parent/child analogue) so benchmarks cannot silently measure “every N units” under a semantic name.
## Considered Options
- **Wire both strategies + fail hard** — chosen; same Active model for boundaries and storage; accept double embed cost; re-process to replace old fake-semantic vectors
- **Wire `semantic` only** — rejected; would leave parent/child on the same lie
- **Keep fallback with warnings** — rejected; that is how the bug stayed hidden
- **Always OpenAI for boundaries regardless of Active** — rejected; confounds model A/B experiments
## Consequences
- Process cost rises for these two Strategies (unit embeds + chunk embeds)
- Prior Experiments / collections produced under the fallback are not true semantic — operator must re-process and re-benchmark
- Strategy modules raise `ChunkingError` if boundary embeddings are absent or mismatched; orchestration is responsible for supplying them