chatbot_v3/docs/adr/0004-docx-csv-chunking-strategy.md

# 0004. Document parsing and chunking strategy (docx / xlsx / csv)

## Status

Accepted

> Amended by
> [ADR-0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md):
> v1 ships **fixed-size** chunking rather than the semantic-aware default
> below, and defers `qa_pair` detection and image captioning. Docx tables do
> become `table_row` units as this ADR specifies, but only when they are data:
> a table holding a cell larger than one chunk, or a cell containing nested
> tables, is treated as page layout and its cells are chunked as prose.
> Row labels are applied only when row 0 is provably a header, and cells are
> joined unlabeled otherwise. No document tree is built, and no heading is
> inferred from text. `.doc` is rejected with `415` pending an out-of-process
> conversion service rather than shelling out to LibreOffice. ADR-0018 also
> adds a Persian normalization step at parse time and fixes the chunk-size
> numbers this ADR left open. The spreadsheet row-to-chunk rules, the
> embedding model, the task-prefix invariant, and the `content_type` value set
> below all apply unchanged — except that `.csv` is read with the standard
> library `csv` module rather than `pandas`.

## Context

ADR-0001 deferred two things to "when we start the docx/csv chunking work":
the exact `content_type` value set, and any document-context payload fields.
That work starts now. This ADR also fixes the dense embedding model, which
ADR-0001 left as TBD — it directly constrains chunk sizing.

A representative sample of real source files (`new_asia_csvs/`, insurance
documents from Bimeh Asia) was inspected directly (`python-docx`/`openpyxl`)
before deciding anything, rather than assuming a shape. Findings:

- **No `.csv` files exist in practice** — the tree is entirely `.xlsx`
  (plus two legacy `.doc` binaries `python-docx` cannot open at all).
- **Spreadsheets** come in two shapes: clean two-column Q&A tables
  (`q`/`a` or `سوال`/`پاسخ`), and directory/contact tables (branches, phone
  numbers) with a merged title row, a blank separator row, and merged cells
  (`openpyxl` only stores a merged range's value in its top-left cell — the
  rest read as empty). Several files carry a dead second sheet.
- **Word documents** have no consistent shape at all: some are flowing prose
  with real `Heading` styles (rare), most are flowing prose with **no
  heading styles**, structure only implied by text (`"1- ... پاسخ: ..."`);
  one file is a single 35×5 table with zero body paragraphs; one file
  alternates literal "سوال:"/"پاسخ:" paragraphs; two files contain embedded
  images with no alt text.

Given this, no single chunking algorithm can be applied uniformly — table
content and flowing prose need different treatment, and even flowing prose
varies in whether headings can be trusted as section boundaries.

The user ran an offline comparison of five chunking strategies against this
data and found **semantic-aware chunking** produced the highest retrieval
accuracy, with **fixed-size chunking** (paired with the previous/next
pointer fields from ADR-0001) a viable, simpler runner-up.

Embedding model: **`nomic-embed-text-v2-moe`** (dense only — sparse and
late-interaction models remain TBD per ADR-0001/0003). Relevant constraints
from its model card: 768-dim output, Matryoshka-truncatable down to 256;
**512-token max sequence length**; requires a task-instruction prefix on
every embedded string — `search_document: ` at ingestion time, `search_query: `
on the agent's query side (ADR-0003).

> **Amendment — the task prefix is currently not applied.** The `emet`
> benchmark that selected this model ran *without* any prefix: its Ollama
> deployment's template is a bare `{{ .Prompt }}` passthrough that injects
> nothing, which was verified directly against the running endpoint. The
> prefix is not cosmetic — embedding the same Persian text with and without
> `search_document: ` yields a cosine of only **0.5741** — so applying it at
> ingest while the query side omits `search_query: ` would make retrieval
> *worse* than using neither.
>
> Implementation therefore defaults `EMBEDDING_NOMIC_DOCUMENT_PREFIX` to
> empty, matching the measured configuration, and exposes it as config so the
> prefixed variant is a one-line experiment rather than a code change. The
> model card remains the reason to expect prefixing to help; what is missing
> is evidence on *this* corpus. Turning it on is a paired change — ingest and
> query must move together — and should be settled by an emet run that
> measures the pair, not by an unmeasured edit here.

## Decision

### Parsing order: structural extraction before chunking

Every source file is first decomposed into **structural units** — table
rows, Q&A pairs, prose blocks — before any chunking algorithm runs. A
chunking algorithm never sees a whole document as undifferentiated text;
it only runs on the prose-block units, because table rows and Q&A pairs are
already atomic and splitting them would break their meaning.

1. **Walk the docx body in document order** (`doc.element.body` children,
   not `doc.paragraphs`/`doc.tables` separately), so tables interleaved with
   paragraphs keep their position, and nested tables inside cells are
   handled recursively. This matters for the "bad design" case the user
   flagged — a table cell containing an entire sub-document — which is
   handled as: recurse into the nested table's rows first; if a single cell
   still contains multiple paragraphs of unstructured prose, treat that
   cell's text as its own prose block and run the chunker on it, rather
   than emitting the whole oversized cell as one chunk.
2. **Detect table rows** → one structural unit per row, columns joined as
   `"{header}: {value}"` pairs (mirrors the xlsx handling below). Header
   detection: the first row whose cells are mostly short, unique, non-empty
   strings; title rows (merged, single populated cell) and blank separator
   rows are skipped, not treated as headers.
3. **Detect Q&A pattern** → one structural unit per question/answer pair,
   via heuristics: literal "سوال"/"پاسخ" (or `q`/`a`) paragraph pairing, or
   a leading enumerator (`"1-"`, `"2-"`) followed by a "پاسخ:"-prefixed
   paragraph. This is intentionally a heuristic, not an LLM classification
   step — it's cheap, deterministic, and the sample above showed the pattern
   is simple enough (leading marker + adjacent paragraph) not to need model
   inference per document.
4. **Everything else** → flowing prose blocks, segmented by heading styles
   where present (`Heading 1`/`2`), or the whole remaining run of paragraphs
   as one block where no heading styles exist.

### Chunking the prose blocks

Two supported strategies, selectable by config, **semantic-aware as the
default**:

- **Semantic-aware chunking** (default): split each prose block into
  sentences, embed each sentence, and break where adjacent-sentence
  similarity drops below a percentile threshold — grouping semantically
  coherent runs of sentences into a chunk. Chosen as default per the user's
  own offline accuracy comparison.
- **Fixed-size chunking** (config alternative): token-count-based splitting
  with overlap. Simpler and cheaper (no embedding pass needed just to
  decide boundaries), and viable specifically *because* ADR-0001 already
  gives every chunk `previous_chunk_id`/`next_chunk_id` pointers — a fixed
  chunk that cuts a thought in half can still be expanded with its
  neighbors at retrieval time (ADR-0003).

Both strategies share one hard constraint: **no chunk's embedded text may
exceed `nomic-embed-text-v2-moe`'s 512-token sequence length** — text beyond
that is silently truncated by the model, not an error, so chunk size must
be bounded well under 512 tokens regardless of which strategy is active.

Table-row and Q&A structural units bypass this chunker entirely — they are
already-sized, already-atomic chunks (a row or a Q&A pair is rarely close
to 512 tokens; if one is, it's truncated the same way, but this is not the
common case per the sample).

### Spreadsheet (xlsx / csv) handling

Row = chunk, matching the table-row handling above:

- Real header row detected (skipping merged title rows and blank separator
  rows); each cell rendered as `"{column_header}: {cell_value}"`.
- Merged cells forward-filled — a merged range's value is copied to every
  row in that range before chunking, so each row-chunk is self-contained
  and doesn't silently lose data that `openpyxl` only attaches to the
  range's top-left cell.
- Sheets with no non-empty data rows (the dead second-sheet pattern seen in
  several sample files) are skipped, not ingested as empty chunks.
- `.csv` is handled identically via `pandas`, even though none exist in the
  current sample — the row → chunk mapping is the same regardless of
  container format.

### Embedding model and prefixes

`nomic-embed-text-v2-moe` is the dense embedding model for ADR-0001's
`dense` vector (768-dim, Matryoshka-truncatable to 256 if storage/latency
later requires it — not adopted now, full 768 is the default). Every string
sent to the model **must** carry its task prefix: `search_document: ` when
embedding a chunk at ingestion time, `search_query: ` when embedding the
agent's query (ADR-0003) — omitting or mismatching the prefix degrades
retrieval quality per the model's own documentation. This is a pipeline
invariant, not a per-call decision.

### Images: LLM extraction now, self-hosted OCR later

Embedded docx images are sent to the ChatGPT (vision) API at ingestion time
to extract a text description/transcription, which becomes its own chunk
(`content_type: image_caption`) positioned in document order via the same
`previous_chunk_id`/`next_chunk_id` mechanism as any other chunk — not
appended silently into a neighboring text chunk. The extraction call is
swappable behind a small interface so it can be replaced with a self-hosted
OCR/captioning model later without changing the chunk/payload shape.

### Legacy `.doc` files

The two legacy binary `.doc` files in the sample cannot be opened by
`python-docx`. They are converted to `.docx` via headless LibreOffice
(`soffice --headless --convert-to docx`) as a pipeline pre-processing step
before the normal docx structural-extraction path runs — not a separate
parser.

### `content_type` value set (finalized)

| Value | Meaning |
|---|---|
| `paragraph` | a chunk produced by the prose chunker (semantic or fixed-size) |
| `table_row` | one row from a docx table or spreadsheet |
| `qa_pair` | one detected question/answer pair |
| `image_caption` | text extracted from an embedded image |

## Consequences

### Positive
- Structural extraction before chunking means table rows and Q&A pairs —
  the cleanest, highest-signal content in the sample — are never mangled by
  a generic text chunker.
- Semantic-aware default matches the user's own measured accuracy result
  rather than a generic best practice; fixed-size remains available where
  its lower cost matters, without losing adjacency thanks to ADR-0001's
  pointer fields.
- Fixes the dense embedding model (previously TBD in ADR-0001), unblocking
  actual implementation of ingestion and agent retrieval.
- Legacy `.doc` and image content are handled instead of silently dropped
  or erroring the whole file.

### Negative
- Heuristic table/Q&A/prose detection will misclassify some future document
  that doesn't match the patterns in this sample; it will need tuning as
  new document batches are onboarded, not just this one.
- Semantic chunking adds an embedding pass at chunk-boundary-decision time,
  separate from the final chunk-embedding pass — extra ingestion latency
  and cost versus fixed-size alone.
- ChatGPT API image extraction is an external network dependency in the
  ingestion path (cost, latency, availability) until the self-hosted OCR
  replacement lands.
- `nomic-embed-text-v2-moe`'s 512-token limit is a hard ceiling on chunk
  size for both strategies; any future switch to a model with a shorter
  limit would require re-tuning chunk-size config, though not the pipeline
  shape.

## Alternatives Considered

- **LLM-based document classification** (send each document to an LLM to
  decide table/Q&A/prose shape and boundaries): rejected as the default —
  the sample showed simple heuristics suffice, and per-document LLM calls
  add cost/latency/non-determinism the heuristics avoid. Not ruled out as a
  future fallback for prose blocks heuristics can't confidently segment.
  Not needed for this sample; noted as reference only for future data.
- **Uniform fixed-size chunking everywhere** (no structural pre-extraction):
  rejected — would split table rows and Q&A pairs mid-content, destroying
  the cleanest signal in the source data.
- **Self-hosted OCR/captioning from day one**: rejected for the initial
  cut — ChatGPT API gets image extraction working now without standing up
  and tuning a model first; revisit once volume/cost justifies it.