Compare commits
2 Commits
640691b8ab
...
dc76ca67b8
| Author | SHA1 | Date | |
|---|---|---|---|
| dc76ca67b8 | |||
| d5ccec2f2c |
@@ -84,7 +84,7 @@ get_qdrant_client()
|
|||||||
| `ollama_base_url` | host URL | Local embeds |
|
| `ollama_base_url` | host URL | Local embeds |
|
||||||
| `qdrant_url` | `http://localhost:6333` | Vectors |
|
| `qdrant_url` | `http://localhost:6333` | Vectors |
|
||||||
| `top_k` | `5` | Retrieval |
|
| `top_k` | `5` | Retrieval |
|
||||||
| `neighbor_prev` / `neighbor_next` | `0` / `0` | Expansion defaults |
|
| `neighbor_prev` / `neighbor_next` | `3` / `3` | Expansion defaults (fixed_size ±3) |
|
||||||
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
|
| `chunk_size` / `chunk_overlap` | `512` / `50` | fixed_size / recursive targets |
|
||||||
| `semantic_threshold` | `0.3` | Fallback if model default missing |
|
| `semantic_threshold` | `0.3` | Fallback if model default missing |
|
||||||
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
|
| `semantic_min_chunk_size` | `3` | Min units per semantic chunk |
|
||||||
|
|||||||
@@ -1,13 +1,12 @@
|
|||||||
# Chunking Strategy — Final Decision
|
# Chunking Strategy — Final Decision
|
||||||
|
|
||||||
**Date:** 17 August 2026
|
**Date:** 17 August 2026 (family); **22 August 2026** (±N locked)
|
||||||
**Decision:** Adopt **`fixed_size`** as the chunking Strategy family.
|
**Decision:** Adopt **`fixed_size`** as the chunking Strategy family with **Neighbor Expansion ±3/3**.
|
||||||
**Not decided here:** Neighbor Expansion level **±N** (follow-up).
|
|
||||||
**Corpus Embedding Model:** `text-embedding-3-large`
|
**Corpus Embedding Model:** `text-embedding-3-large`
|
||||||
**Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe)
|
**Evaluation set:** 10 Word documents (Decision Board / neighbor-sweep universe)
|
||||||
**Questions:** 201 per Candidate (nomic semantic: 195)
|
**Questions:** 201 per Candidate (nomic semantic: 195)
|
||||||
|
|
||||||
**One-line close:** **`fixed_size` wins** on official mean composite (**9.040** vs **8.727**). Pick **±N later**, but **not ±0**.
|
**One-line close:** **`fixed_size ±3`** is the stabilized Strategy Candidate (**9.040** mean composite vs **8.727** for semantic @ large). **Do not use ±0** as the default — it loses to semantic on mean.
|
||||||
|
|
||||||
Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board:
|
Charts below are generated from `data/chunking_benchmark.db` with the same composite as the Decision Board:
|
||||||
|
|
||||||
@@ -123,15 +122,38 @@ One leftover five-strategy Experiment on `customer1` under large is **invalid**
|
|||||||
|
|
||||||
**Use `fixed_size` for chunking** under **`text-embedding-3-large`.**
|
**Use `fixed_size` for chunking** under **`text-embedding-3-large`.**
|
||||||
|
|
||||||
|
**Use Neighbor Expansion `neighbor_prev=3`, `neighbor_next=3`** (symmetric **±3/3**) for Query and Experiment defaults when running `fixed_size`.
|
||||||
|
|
||||||
**Do not use `semantic` as the default Strategy** on this evaluation universe.
|
**Do not use `semantic` as the default Strategy** on this evaluation universe.
|
||||||
|
|
||||||
**Follow-up:** choose Neighbor Expansion **±N** among **±1, ±2, ±3**. Stage 1 auto-pick is **±3**. **±0 is not recommended** if this family decision is to remain valid.
|
**Do not default to ±0.** Without expansion, semantic @ large beats `fixed_size` on mean composite (8.727 vs 8.606).
|
||||||
|
|
||||||
Production config change is **out of scope** for this record.
|
Operational defaults are set in `src/core/config.py`, `.env.example`, and the Dashboard Query/Benchmarks forms. Override per request is still supported.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Evidence (stage 2 cells)
|
## 8. Addendum — why ±3 (22 August 2026)
|
||||||
|
|
||||||
|
Stage 1 ranked all `fixed_size` Candidates on the 10-doc grid:
|
||||||
|
|
||||||
|
| ±N | Mean composite | vs semantic @ large |
|
||||||
|
|----|----------------|---------------------|
|
||||||
|
| ±0 | 8.606 | **loses** (−0.121) |
|
||||||
|
| ±1 | 8.926 | wins (+0.199) |
|
||||||
|
| ±2 | 8.987 | wins (+0.260) |
|
||||||
|
| **±3** | **9.040** | **wins (+0.313)** |
|
||||||
|
|
||||||
|
**±3** is the stage 1 auto-pick and the highest mean composite. **±1** and **±2** are close; **±0** is ruled out.
|
||||||
|
|
||||||
|
Per-document best ±N varies (±3 wins on 5 docs, ±1 on 3, ±2 on 1, ±0 on 1). The **global** default is still **±3** because Decision Board ranks by mean composite across the full set, not by win-count among ±N levels.
|
||||||
|
|
||||||
|
**Caveats kept from stage 2:** semantic still wins head-to-head on 6/10 docs vs ±3 (website, customer1, etc.), but mean composite and judge metrics favor **`fixed_size ±3`**. Retrieval Inspect on outlier docs remains optional follow-up.
|
||||||
|
|
||||||
|
**Production RAG outside this repo** is not changed by this addendum — only benchmarker defaults and this record.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Evidence (stage 2 cells)
|
||||||
|
|
||||||
| Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id |
|
| Document | `fixed_size ±3` experiment id | `semantic @ large` experiment id |
|
||||||
|----------|-------------------------------|----------------------------------|
|
|----------|-------------------------------|----------------------------------|
|
||||||
|
|||||||
@@ -22,9 +22,9 @@ class Settings(BaseSettings):
|
|||||||
|
|
||||||
# Retrieval
|
# Retrieval
|
||||||
top_k: int = 5
|
top_k: int = 5
|
||||||
# Neighbor Expansion for fixed_size (ADR-0023); 0/0 = off
|
# Neighbor Expansion for fixed_size (ADR-0023); 3/3 = decision default (see final-chunking-strategy-decision.md)
|
||||||
neighbor_prev: int = 0
|
neighbor_prev: int = 3
|
||||||
neighbor_next: int = 0
|
neighbor_next: int = 3
|
||||||
|
|
||||||
# LLM generation
|
# LLM generation
|
||||||
temperature: float = 0.0
|
temperature: float = 0.0
|
||||||
|
|||||||
@@ -1287,8 +1287,8 @@ function RetrievalInspectView({ experiment, onBack }) {
|
|||||||
|
|
||||||
function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
|
function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
|
||||||
const [form, setForm] = useState({
|
const [form, setForm] = useState({
|
||||||
document_id: '', strategy: '', question: '', top_k: 5,
|
document_id: '', strategy: 'fixed_size', question: '', top_k: 5,
|
||||||
neighbor_prev: 0, neighbor_next: 0, corpus_model_id: '',
|
neighbor_prev: 3, neighbor_next: 3, corpus_model_id: '',
|
||||||
});
|
});
|
||||||
const [result, setResult] = useState(null);
|
const [result, setResult] = useState(null);
|
||||||
const [loading, setLoading] = useState(false);
|
const [loading, setLoading] = useState(false);
|
||||||
@@ -1761,9 +1761,9 @@ function ComparisonView({ experiments, addToast, onBack }) {
|
|||||||
function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) {
|
function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) {
|
||||||
const [experiments, setExperiments] = useState([]);
|
const [experiments, setExperiments] = useState([]);
|
||||||
const [form, setForm] = useState({
|
const [form, setForm] = useState({
|
||||||
document_id: '', strategies: strategies.map(s => s.name),
|
document_id: '', strategies: ['fixed_size'],
|
||||||
questions_file: questionsFile, top_k: 5,
|
questions_file: questionsFile, top_k: 5,
|
||||||
neighbor_prev: 0, neighbor_next: 0, dry_run: false, corpus_model_id: '',
|
neighbor_prev: 3, neighbor_next: 3, dry_run: false, corpus_model_id: '',
|
||||||
});
|
});
|
||||||
const [running, setRunning] = useState(false);
|
const [running, setRunning] = useState(false);
|
||||||
const [embModels, setEmbModels] = useState([]);
|
const [embModels, setEmbModels] = useState([]);
|
||||||
|
|||||||
Reference in New Issue
Block a user