# Human Evaluation Report — fixed_size ±3

**Date:** 22 August 2026  
**Strategy:** `fixed_size` with Neighbor Expansion **±3/3**  
**Corpus Embedding Model:** `text-embedding-3-large`  
**Sample:** **60 questions** (6 per document × **10 documents**)  
**Total benchmark universe:** 201 questions  
**Review effort:** ~6 hours

**Verdict:** Human review **supports** the stabilized **`fixed_size ±3`** default (see [final decision](final-chunking-strategy-decision.md)).

| KPI | Value |
|-----|-------|
| Mean human score (1–5) | **4.30** |
| Mean LLM answer similarity (1–10) | **8.88** |
| Human–LLM agreement | **57%** agree · 42% partial · 2% disagree |
| Questions reviewed | **60** / 201 (30%) |

Raw scores: [fixed_size_plus3_human_scores.csv](assets/human-eval/fixed_size_plus3_human_scores.csv)

---

## 1. Mean human score by document

![Mean human score by document](assets/human-eval/mean-human-by-document.svg)

## 2. Human vs LLM-as-Judge agreement

![Agreement breakdown](assets/human-eval/human-llm-agreement.svg)

Agreement = same quality tier when mapping LLM answer similarity (1–10) to human scale (1–5).

## 3. Human score vs LLM similarity by document

![Human vs LLM by document](assets/human-eval/human-vs-llm-by-document.svg)

## 4. Human score distribution

![Score distribution](assets/human-eval/human-score-distribution.svg)

## 5. Document summary

| Document | Reviewed | Mean human (1–5) | Mean LLM similarity | Agreement | |
|----------|----------|------------------|---------------------|-----------|---|
| bazresi | 6 | 4.33 | 8.50 | 50% | ⚠ |
| customer1 | 6 | 4.00 | 9.00 | 33% | ⚠ |
| fire | 6 | 4.50 | 8.33 | 67% |  |
| havades-ind | 6 | 3.83 | 8.50 | 50% | ⚠ |
| havades | 6 | 4.00 | 8.83 | 50% | ⚠ |
| lifetime-ind | 6 | 4.33 | 9.33 | 50% | ⚠ |
| moavenin | 6 | 4.83 | 9.50 | 83% |  |
| Refah | 6 | 4.67 | 9.33 | 83% |  |
| website | 6 | 4.17 | 9.00 | 17% | ⚠ |
| lifetime-comp | 6 | 4.33 | 8.50 | 83% |  |

## 6. Notable disagreements (human stricter than LLM)

| Document | Question (excerpt) | Human | LLM sim | Notes |
|----------|-------------------|-------|---------|-------|
| havades-ind | آیا خطرات جنگ، زلزله و ورزش‌های خطرناک قابل پوشش هستند؟ | 2 | 7 | Answer incomplete compared to expected reference. |

## 7. Method

- Stratified sample: **6 questions per document**, all **10** evaluation documents represented.
- Human rubric: **1–5** (1 = wrong/unhelpful, 3 = partial, 5 = matches expected answer).
- Each row: read question, expected answer, generated answer; score independently of automation.
- LLM-as-Judge scores retained in CSV for comparison (context relevance, similarity, faithfulness, hallucination).
- Source Experiments: newest single-strategy **`fixed_size ±3`** run per document under **`text-embedding-3-large`**.

## 8. Conclusions

1. **Overall quality is strong** — mean human score **4.30/5** on the reviewed sample.
2. **Automated evaluation is directionally reliable** — **57%** agreement on quality tier; disagreements cluster on nuanced or incomplete answers.
3. **website** and **customer1** show the largest human–LLM gaps; worth optional Retrieval Inspect follow-up, but do not overturn the global **`fixed_size ±3`** decision.
4. Human review **confirms** the family and ±3 configuration documented in [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md).

Regenerate charts after CSV updates:

```bash
.venv/bin/python docs/assets/human-eval/generate_human_eval_report.py
```