17 lines
1.4 KiB
Markdown
17 lines
1.4 KiB
Markdown
# Semantic strategies require Semantic Boundary Detection
|
||
|
||
`semantic` and `semantic_parent_child` must cut chunks from meaning: the orchestrator embeds consecutive units (sentences / paragraphs) with the Active Embedding Model snapshot, passes those vectors into `chunk()`, then embeds finished chunks for Qdrant. Fixed-count fallbacks are removed — missing or length-mismatched boundary embeddings fail that Strategy. This completes ADR 0012’s sentence-level design in the process path (and the parent/child analogue) so benchmarks cannot silently measure “every N units” under a semantic name.
|
||
|
||
## Considered Options
|
||
|
||
- **Wire both strategies + fail hard** — chosen; same Active model for boundaries and storage; accept double embed cost; re-process to replace old fake-semantic vectors
|
||
- **Wire `semantic` only** — rejected; would leave parent/child on the same lie
|
||
- **Keep fallback with warnings** — rejected; that is how the bug stayed hidden
|
||
- **Always OpenAI for boundaries regardless of Active** — rejected; confounds model A/B experiments
|
||
|
||
## Consequences
|
||
|
||
- Process cost rises for these two Strategies (unit embeds + chunk embeds)
|
||
- Prior Experiments / collections produced under the fallback are not true semantic — operator must re-process and re-benchmark
|
||
- Strategy modules raise `ChunkingError` if boundary embeddings are absent or mismatched; orchestration is responsible for supplying them
|