Compare commits

..

2 Commits

Author SHA1 Message Date
dc76ca67b8 docs: lock fixed_size ±3 in decision record and lld
Why:
- Close the open ±N follow-up from the strategy finalization memo.

Changes:
- Add addendum with ±0…±3 evidence and binding ±3/3 default.
- Sync LLD config table with new neighbor expansion defaults.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-22 13:27:18 +03:30
d5ccec2f2c feat(config): apply fixed_size ±3 as stabilized defaults
Why:
- Final benchmark selected fixed_size with neighbor expansion ±3 as the winning Candidate.

Changes:
- Set neighbor_prev/neighbor_next defaults to 3 in Settings.
- Default Query tab to fixed_size ±3/3 and Benchmarks to single-strategy fixed_size.

Impact:
- API and Dashboard runs use ±3 unless explicitly overridden.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-22 13:27:15 +03:30
4 changed files with 37 additions and 15 deletions

View File

@@ -84,7 +84,7 @@ get_qdrant_client()
| `ollama_base_url` | host URL | Local embeds | | `ollama_base_url` | host URL | Local embeds |
| `qdrant_url` | `http://localhost:6333` | Vectors | | `qdrant_url` | `http://localhost:6333` | Vectors |
| `top_k` | `5` | Retrieval | | `top_k` | `5` | Retrieval |
| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults | | `neighbor_prev` / `neighbor_next` | `3` / `3` | Expansion defaults (fixed_size ±3) |
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets | | `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
| `semantic_threshold` | `0.3` | Fallback if model default missing | | `semantic_threshold` | `0.3` | Fallback if model default missing |
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk | | `semantic_min_chunk_size` | `3` | Min units per semantic chunk |

View File

@@ -1,13 +1,12 @@
# Chunking Strategy — Final Decision # Chunking Strategy — Final Decision
**Date:** 17 August 2026 **Date:** 17 August 2026 (family); **22 August 2026** (±N locked)
**Decision:** Adopt **`fixed_size`** as the chunking Strategy family. **Decision:** Adopt **`fixed_size`** as the chunking Strategy family with **Neighbor Expansion ±3/3**.
**Not decided here:** Neighbor Expansion level **±N** (follow-up).
**Corpus Embedding Model:** `text-embedding-3-large` **Corpus Embedding Model:** `text-embedding-3-large`
**Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe) **Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe)
**Questions:** 201 per Candidate (nomic semantic: 195) **Questions:** 201 per Candidate (nomic semantic: 195)
**One-line close:** **`fixed_size` wins** on official mean composite (**9.040** vs **8.727**). Pick **±N later**, but **not ±0**. **One-line close:** **`fixed_size ±3`** is the stabilized Strategy Candidate (**9.040** mean composite vs **8.727** for semantic @ large). **Do not use ±0** as the default — it loses to semantic on mean.
Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board: Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board:
@@ -123,15 +122,38 @@ One leftover five-strategy Experiment on `customer1` under large is **invalid**
**Use `fixed_size` for chunking** under **`text-embedding-3-large`.** **Use `fixed_size` for chunking** under **`text-embedding-3-large`.**
**Use Neighbor Expansion `neighbor_prev=3`, `neighbor_next=3`** (symmetric **±3/3**) for Query and Experiment defaults when running `fixed_size`.
**Do not use `semantic` as the default Strategy** on this evaluation universe. **Do not use `semantic` as the default Strategy** on this evaluation universe.
**Follow-up:** choose Neighbor Expansion **±N** among **±1, ±2, ±3**. Stage 1 auto-pick is **±3**. **±0 is not recommended** if this family decision is to remain valid. **Do not default to ±0.** Without expansion, semantic @ large beats `fixed_size` on mean composite (8.727 vs 8.606).
Production config change is **out of scope** for this record. Operational defaults are set in `src/core/config.py`, `.env.example`, and the Dashboard Query/Benchmarks forms. Override per request is still supported.
--- ---
## 8. Evidence (stage 2 cells) ## 8. Addendum — why ±3 (22 August 2026)
Stage 1 ranked all `fixed_size` Candidates on the 10-doc grid:
| ±N | Mean composite | vs semantic @ large |
|----|----------------|---------------------|
| ±0 | 8.606 | **loses** (−0.121) |
| ±1 | 8.926 | wins (+0.199) |
| ±2 | 8.987 | wins (+0.260) |
| **±3** | **9.040** | **wins (+0.313)** |
**±3** is the stage 1 auto-pick and the highest mean composite. **±1** and **±2** are close; **±0** is ruled out.
Per-document best ±N varies (±3 wins on 5 docs, ±1 on 3, ±2 on 1, ±0 on 1). The **global** default is still **±3** because Decision Board ranks by mean composite across the full set, not by win-count among ±N levels.
**Caveats kept from stage 2:** semantic still wins head-to-head on 6/10 docs vs ±3 (website, customer1, etc.), but mean composite and judge metrics favor **`fixed_size ±3`**. Retrieval Inspect on outlier docs remains optional follow-up.
**Production RAG outside this repo** is not changed by this addendum — only benchmarker defaults and this record.
---
## 9. Evidence (stage 2 cells)
| Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id | | Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id |
|----------|-------------------------------|----------------------------------| |----------|-------------------------------|----------------------------------|

View File

@@ -22,9 +22,9 @@ class Settings(BaseSettings):
# Retrieval # Retrieval
top_k: int = 5 top_k: int = 5
# Neighbor Expansion for fixed_size (ADR-0023); 0/0 = off # Neighbor Expansion for fixed_size (ADR-0023); 3/3 = decision default (see final-chunking-strategy-decision.md)
neighbor_prev: int = 0 neighbor_prev: int = 3
neighbor_next: int = 0 neighbor_next: int = 3
# LLM generation # LLM generation
temperature: float = 0.0 temperature: float = 0.0

View File

@@ -1287,8 +1287,8 @@ function RetrievalInspectView({ experiment, onBack }) {
function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) { function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
const [form, setForm] = useState({ const [form, setForm] = useState({
document_id: '', strategy: '', question: '', top_k: 5, document_id: '', strategy: 'fixed_size', question: '', top_k: 5,
neighbor_prev: 0, neighbor_next: 0, corpus_model_id: '', neighbor_prev: 3, neighbor_next: 3, corpus_model_id: '',
}); });
const [result, setResult] = useState(null); const [result, setResult] = useState(null);
const [loading, setLoading] = useState(false); const [loading, setLoading] = useState(false);
@@ -1761,9 +1761,9 @@ function ComparisonView({ experiments, addToast, onBack }) {
function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) { function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) {
const [experiments, setExperiments] = useState([]); const [experiments, setExperiments] = useState([]);
const [form, setForm] = useState({ const [form, setForm] = useState({
document_id: '', strategies: strategies.map(s => s.name), document_id: '', strategies: ['fixed_size'],
questions_file: questionsFile, top_k: 5, questions_file: questionsFile, top_k: 5,
neighbor_prev: 0, neighbor_next: 0, dry_run: false, corpus_model_id: '', neighbor_prev: 3, neighbor_next: 3, dry_run: false, corpus_model_id: '',
}); });
const [running, setRunning] = useState(false); const [running, setRunning] = useState(false);
const [embModels, setEmbModels] = useState([]); const [embModels, setEmbModels] = useState([]);