Mahdi Bazrafshan
640691b8ab
Why: - Close the benchmark with a written family decision backed by Experiment composites. Changes: - Add the decision memo and SVG charts generated from SQLite. Impact: - Documents fixed_size over semantic under text-embedding-3-large; ±N remains follow-up. Co-authored-by: Cursor <cursoragent@cursor.com>
7.4 KiB
Chunking Strategy — Final Decision
Date: 17 August 2026
Decision: Adopt fixed_size as the chunking Strategy family.
Not decided here: Neighbor Expansion level ±N (follow-up).
Corpus Embedding Model: text-embedding-3-large
Evaluation set: 10 Word documents (Decision Board / neighbor-sweep universe)
Questions: 201 per Candidate (nomic semantic: 195)
One-line close: fixed_size wins on official mean composite (9.040 vs 8.727). Pick ±N later, but not ±0.
Charts below are generated from data/chunking_benchmark.db with the same composite as the Decision Board:
(0.3 × context relevance + 0.4 × answer similarity + 0.3 × faithfulness)
× (1 − hallucination rate)
Regenerate figures after new Experiments:
.venv/bin/python docs/assets/decision/generate_charts.py
1. Why fixed_size is the winner
Decision Board ranks by mean composite, not by “how many documents won.” Stage 1 auto-picks fixed_size ±3 vs semantic @ text-embedding-3-large. Stage 2 mean composite is +0.313 for fixed_size.
| Candidate | Mean composite | Context | Similarity | Faithfulness | Hallucination | Docs |
|---|---|---|---|---|---|---|
| fixed_size ±3 | 9.040 | 9.40 | 8.95 | 9.47 | 2.2% | 10/10 |
| fixed_size ±2 | 8.987 | 9.43 | 8.93 | 9.44 | 2.7% | 10/10 |
| fixed_size ±1 | 8.926 | 9.38 | 8.89 | 9.42 | 3.0% | 10/10 |
| semantic @ text-embedding-3-large | 8.727 | 9.38 | 8.83 | 9.30 | 4.8% | 10/10 |
| fixed_size ±0 | 8.606 | 9.18 | 8.79 | 9.25 | 4.9% | 10/10 |
| semantic @ nomic-embed-text-v2-moe | 8.551 | 9.19 | 8.65 | 9.14 | 4.7% | 10/10 |
fixed_size ±1, ±2, and ±3 all beat the best semantic Candidate on mean. ±0 (no Neighbor Expansion) does not (8.606 vs 8.727). Shipping fixed_size without expansion would weaken this family decision.
2. Judge metrics (stage 2 winners)
fixed_size ±3 leads on every judge metric, including lower hallucination.
3. Per-document showdown
Mean ranking still favors fixed_size. Head-to-head at ±3: semantic 6 / fixed_size 4. Semantic’s six wins are mostly small; fixed_size’s four wins are larger, especially fire.
| Document | Questions | fixed_size ±3 | semantic @ large | Winner |
|---|---|---|---|---|
| fire.docx | 20 | 8.810 | 6.228 | fixed_size (+2.58) |
| moavenin.docx | 20 | 9.735 | 8.621 | fixed_size (+1.11) |
| general-havades-individuals.doc | 20 | 9.205 | 8.564 | fixed_size (+0.64) |
| havades.docx | 20 | 8.483 | 7.969 | fixed_size (+0.51) |
| bazresi.docx | 16 | 9.366 | 9.405 | semantic (−0.04) |
| life-time-individual.docx | 20 | 9.445 | 9.505 | semantic (−0.06) |
| Refah.docx | 20 | 9.680 | 9.815 | semantic (−0.14) |
| lifetime-compensation.docx | 15 | 8.761 | 9.129 | semantic (−0.37) |
| customer1.docx | 20 | 8.451 | 8.973 | semantic (−0.52) |
| website.docx | 30 | 8.466 | 9.058 | semantic (−0.59) |
Win-count would pick semantic. Official product ranking (mean composite) picks fixed_size. This memo follows the Decision Board rule (ADR-0026).
4. Full Candidate heatmap
Every cell is a newest single-strategy Experiment under text-embedding-3-large. Semantic @ large collapses on fire; fixed_size ±1…±3 stay high across the set.
Best fixed_size ±N by document (does not change the family call):
| ±N | Documents where it is the best fixed_size cell |
|---|---|
| ±3 | bazresi, fire, general-havades-individuals, havades, life-time-individual (5) |
| ±1 | customer1, moavenin, Refah (3) |
| ±2 | website (1) |
| ±0 | lifetime-compensation (1) |
5. Method
- Corpus Embedding Model locked to
text-embedding-3-large - Newest single-strategy Experiment per document × Candidate (Decision Board cells)
- Retrieval
top_k = 5 - Stage 1: best
fixed_size±N vs bestsemantic@Boundary - Stage 2: mean composite + per-document breakdown
- Experiments dated 9–10 August 2026
logs/neighbor_sweep.logrecorded mid-run connection errors on some units; SQLite now has a full 10×4fixed_sizegrid — treat the database as source of truth
6. Excluded strategies
No comparable 10-doc, single-strategy, text-embedding-3-large Experiments exist for the three Strategies below. They are not Decision Board Candidates (ADR-0026).
| Strategy | Why it is not the winner |
|---|---|
| recursive | Not on the Decision Board grid. Informal PDF / text-embedding-3-small multi-strategy runs (not comparable to this close-out) were mixed; recursive does not get Neighbor Expansion (expansion is fixed_size only). |
| contextual_retrieval | Extra LLM call per chunk at process time. Same informal PDF/small runs scored below recursive and fixed_size. |
| semantic_parent_child | Boundary-detection cost plus parent fetch at query. Informal PDF/small composites were much worse (~1.8–2.8). Fail-hard if Boundary embeds mismatch (ADR-0020). |
One leftover five-strategy Experiment on customer1 under large is invalid (four Strategies scored 0 / errors) and was ignored.
7. Binding decision
Use fixed_size for chunking under text-embedding-3-large.
Do not use semantic as the default Strategy on this evaluation universe.
Follow-up: choose Neighbor Expansion ±N among ±1, ±2, ±3. Stage 1 auto-pick is ±3. ±0 is not recommended if this family decision is to remain valid.
Production config change is out of scope for this record.
8. Evidence (stage 2 cells)
| Document | fixed_size ±3 experiment id |
semantic @ large experiment id |
|---|---|---|
| bazresi.docx | 6fe08750ee324130b3d46d8b3bc95280 |
6263c4f9e63347cea9eb19f4d36309c0 |
| customer1.docx | 8c4d888d367b4c0299b2d19dd9654f80 |
0c4572e80902408796f0dd11776e8e55 |
| fire.docx | 230a3cf57ae74b679ecde83cfeb9b6f5 |
414162efc314406db082ba8537c2423f |
| general-havades-individuals.doc | 7ad892cfcd6143109f77a7808c8dfbd9 |
14fa4f872373446c8b042fc8e84e6983 |
| havades.docx | 04bd45d836a64c3da2e68bb747c0c9b8 |
232238b4d3304243b82b02326ef64617 |
| life-time-individual.docx | d682706f54824469b235c099bf5a3d3c |
bc7d9f1fbe3540c097a5f8cc1a1c44bb |
| moavenin.docx | 63f81cf0a632418aab2f93884518ed71 |
4e01960b5db14e81975a4fbc86c4e10c |
| Refah.docx | 6304dba39f3f4c05935195c588a36a64 |
b5c630c45615451a93d97d80dee361df |
| website.docx | fb9d47750ca74761a3c82c57aa51030b |
53256897cfb74dd7a6508e750ff1ba63 |
| lifetime-compensation.docx | 99736ffbc6da455b8b3ca3a0fb8e5ef7 |
e51c2b57e7364dfa97e5429f615d6e4e |
HTML reports: GET /benchmarks/{id}/report. Dashboard: Decision Tab, Corpus = text-embedding-3-large.