docs: record embedding, neighbor, and inspect decisions
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
16
docs/adr/0020-semantic-boundary-detection-required.md
Normal file
16
docs/adr/0020-semantic-boundary-detection-required.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Semantic strategies require Semantic Boundary Detection
|
||||
|
||||
`semantic` and `semantic_parent_child` must cut chunks from meaning: the orchestrator embeds consecutive units (sentences / paragraphs) with the Active Embedding Model snapshot, passes those vectors into `chunk()`, then embeds finished chunks for Qdrant. Fixed-count fallbacks are removed — missing or length-mismatched boundary embeddings fail that Strategy. This completes ADR 0012’s sentence-level design in the process path (and the parent/child analogue) so benchmarks cannot silently measure “every N units” under a semantic name.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Wire both strategies + fail hard** — chosen; same Active model for boundaries and storage; accept double embed cost; re-process to replace old fake-semantic vectors
|
||||
- **Wire `semantic` only** — rejected; would leave parent/child on the same lie
|
||||
- **Keep fallback with warnings** — rejected; that is how the bug stayed hidden
|
||||
- **Always OpenAI for boundaries regardless of Active** — rejected; confounds model A/B experiments
|
||||
|
||||
## Consequences
|
||||
|
||||
- Process cost rises for these two Strategies (unit embeds + chunk embeds)
|
||||
- Prior Experiments / collections produced under the fallback are not true semantic — operator must re-process and re-benchmark
|
||||
- Strategy modules raise `ChunkingError` if boundary embeddings are absent or mismatched; orchestration is responsible for supplying them
|
||||
Reference in New Issue
Block a user