initial commit

docs/adr/0004-docx-csv-chunking-strategy.md

@@ -0,0 +1,207 @@
+# 0004. Document parsing and chunking strategy (docx / xlsx / csv)
+
+## Status
+
+Accepted
+
+## Context
+
+ADR-0001 deferred two things to "when we start the docx/csv chunking work":
+the exact `content_type` value set, and any document-context payload fields.
+That work starts now. This ADR also fixes the dense embedding model, which
+ADR-0001 left as TBD — it directly constrains chunk sizing.
+
+A representative sample of real source files (`new_asia_csvs/`, insurance
+documents from Bimeh Asia) was inspected directly (`python-docx`/`openpyxl`)
+before deciding anything, rather than assuming a shape. Findings:
+
+- **No `.csv` files exist in practice** — the tree is entirely `.xlsx`
+  (plus two legacy `.doc` binaries `python-docx` cannot open at all).
+- **Spreadsheets** come in two shapes: clean two-column Q&A tables
+  (`q`/`a` or `سوال`/`پاسخ`), and directory/contact tables (branches, phone
+  numbers) with a merged title row, a blank separator row, and merged cells
+  (`openpyxl` only stores a merged range's value in its top-left cell — the
+  rest read as empty). Several files carry a dead second sheet.
+- **Word documents** have no consistent shape at all: some are flowing prose
+  with real `Heading` styles (rare), most are flowing prose with **no
+  heading styles**, structure only implied by text (`"1- ... پاسخ: ..."`);
+  one file is a single 35×5 table with zero body paragraphs; one file
+  alternates literal "سوال:"/"پاسخ:" paragraphs; two files contain embedded
+  images with no alt text.
+
+Given this, no single chunking algorithm can be applied uniformly — table
+content and flowing prose need different treatment, and even flowing prose
+varies in whether headings can be trusted as section boundaries.
+
+The user ran an offline comparison of five chunking strategies against this
+data and found **semantic-aware chunking** produced the highest retrieval
+accuracy, with **fixed-size chunking** (paired with the previous/next
+pointer fields from ADR-0001) a viable, simpler runner-up.
+
+Embedding model: **`nomic-embed-text-v2-moe`** (dense only — sparse and
+late-interaction models remain TBD per ADR-0001/0003). Relevant constraints
+from its model card: 768-dim output, Matryoshka-truncatable down to 256;
+**512-token max sequence length**; requires a task-instruction prefix on
+every embedded string — `search_document: ` at ingestion time, `search_query: `
+on the agent's query side (ADR-0003).
+
+## Decision
+
+### Parsing order: structural extraction before chunking
+
+Every source file is first decomposed into **structural units** — table
+rows, Q&A pairs, prose blocks — before any chunking algorithm runs. A
+chunking algorithm never sees a whole document as undifferentiated text;
+it only runs on the prose-block units, because table rows and Q&A pairs are
+already atomic and splitting them would break their meaning.
+
+1. **Walk the docx body in document order** (`doc.element.body` children,
+   not `doc.paragraphs`/`doc.tables` separately), so tables interleaved with
+   paragraphs keep their position, and nested tables inside cells are
+   handled recursively. This matters for the "bad design" case the user
+   flagged — a table cell containing an entire sub-document — which is
+   handled as: recurse into the nested table's rows first; if a single cell
+   still contains multiple paragraphs of unstructured prose, treat that
+   cell's text as its own prose block and run the chunker on it, rather
+   than emitting the whole oversized cell as one chunk.
+2. **Detect table rows** → one structural unit per row, columns joined as
+   `"{header}: {value}"` pairs (mirrors the xlsx handling below). Header
+   detection: the first row whose cells are mostly short, unique, non-empty
+   strings; title rows (merged, single populated cell) and blank separator
+   rows are skipped, not treated as headers.
+3. **Detect Q&A pattern** → one structural unit per question/answer pair,
+   via heuristics: literal "سوال"/"پاسخ" (or `q`/`a`) paragraph pairing, or
+   a leading enumerator (`"1-"`, `"2-"`) followed by a "پاسخ:"-prefixed
+   paragraph. This is intentionally a heuristic, not an LLM classification
+   step — it's cheap, deterministic, and the sample above showed the pattern
+   is simple enough (leading marker + adjacent paragraph) not to need model
+   inference per document.
+4. **Everything else** → flowing prose blocks, segmented by heading styles
+   where present (`Heading 1`/`2`), or the whole remaining run of paragraphs
+   as one block where no heading styles exist.
+
+### Chunking the prose blocks
+
+Two supported strategies, selectable by config, **semantic-aware as the
+default**:
+
+- **Semantic-aware chunking** (default): split each prose block into
+  sentences, embed each sentence, and break where adjacent-sentence
+  similarity drops below a percentile threshold — grouping semantically
+  coherent runs of sentences into a chunk. Chosen as default per the user's
+  own offline accuracy comparison.
+- **Fixed-size chunking** (config alternative): token-count-based splitting
+  with overlap. Simpler and cheaper (no embedding pass needed just to
+  decide boundaries), and viable specifically *because* ADR-0001 already
+  gives every chunk `previous_chunk_id`/`next_chunk_id` pointers — a fixed
+  chunk that cuts a thought in half can still be expanded with its
+  neighbors at retrieval time (ADR-0003).
+
+Both strategies share one hard constraint: **no chunk's embedded text may
+exceed `nomic-embed-text-v2-moe`'s 512-token sequence length** — text beyond
+that is silently truncated by the model, not an error, so chunk size must
+be bounded well under 512 tokens regardless of which strategy is active.
+
+Table-row and Q&A structural units bypass this chunker entirely — they are
+already-sized, already-atomic chunks (a row or a Q&A pair is rarely close
+to 512 tokens; if one is, it's truncated the same way, but this is not the
+common case per the sample).
+
+### Spreadsheet (xlsx / csv) handling
+
+Row = chunk, matching the table-row handling above:
+
+- Real header row detected (skipping merged title rows and blank separator
+  rows); each cell rendered as `"{column_header}: {cell_value}"`.
+- Merged cells forward-filled — a merged range's value is copied to every
+  row in that range before chunking, so each row-chunk is self-contained
+  and doesn't silently lose data that `openpyxl` only attaches to the
+  range's top-left cell.
+- Sheets with no non-empty data rows (the dead second-sheet pattern seen in
+  several sample files) are skipped, not ingested as empty chunks.
+- `.csv` is handled identically via `pandas`, even though none exist in the
+  current sample — the row → chunk mapping is the same regardless of
+  container format.
+
+### Embedding model and prefixes
+
+`nomic-embed-text-v2-moe` is the dense embedding model for ADR-0001's
+`dense` vector (768-dim, Matryoshka-truncatable to 256 if storage/latency
+later requires it — not adopted now, full 768 is the default). Every string
+sent to the model **must** carry its task prefix: `search_document: ` when
+embedding a chunk at ingestion time, `search_query: ` when embedding the
+agent's query (ADR-0003) — omitting or mismatching the prefix degrades
+retrieval quality per the model's own documentation. This is a pipeline
+invariant, not a per-call decision.
+
+### Images: LLM extraction now, self-hosted OCR later
+
+Embedded docx images are sent to the ChatGPT (vision) API at ingestion time
+to extract a text description/transcription, which becomes its own chunk
+(`content_type: image_caption`) positioned in document order via the same
+`previous_chunk_id`/`next_chunk_id` mechanism as any other chunk — not
+appended silently into a neighboring text chunk. The extraction call is
+swappable behind a small interface so it can be replaced with a self-hosted
+OCR/captioning model later without changing the chunk/payload shape.
+
+### Legacy `.doc` files
+
+The two legacy binary `.doc` files in the sample cannot be opened by
+`python-docx`. They are converted to `.docx` via headless LibreOffice
+(`soffice --headless --convert-to docx`) as a pipeline pre-processing step
+before the normal docx structural-extraction path runs — not a separate
+parser.
+
+### `content_type` value set (finalized)
+
+| Value | Meaning |
+|---|---|
+| `paragraph` | a chunk produced by the prose chunker (semantic or fixed-size) |
+| `table_row` | one row from a docx table or spreadsheet |
+| `qa_pair` | one detected question/answer pair |
+| `image_caption` | text extracted from an embedded image |
+
+## Consequences
+
+### Positive
+- Structural extraction before chunking means table rows and Q&A pairs —
+  the cleanest, highest-signal content in the sample — are never mangled by
+  a generic text chunker.
+- Semantic-aware default matches the user's own measured accuracy result
+  rather than a generic best practice; fixed-size remains available where
+  its lower cost matters, without losing adjacency thanks to ADR-0001's
+  pointer fields.
+- Fixes the dense embedding model (previously TBD in ADR-0001), unblocking
+  actual implementation of ingestion and agent retrieval.
+- Legacy `.doc` and image content are handled instead of silently dropped
+  or erroring the whole file.
+
+### Negative
+- Heuristic table/Q&A/prose detection will misclassify some future document
+  that doesn't match the patterns in this sample; it will need tuning as
+  new document batches are onboarded, not just this one.
+- Semantic chunking adds an embedding pass at chunk-boundary-decision time,
+  separate from the final chunk-embedding pass — extra ingestion latency
+  and cost versus fixed-size alone.
+- ChatGPT API image extraction is an external network dependency in the
+  ingestion path (cost, latency, availability) until the self-hosted OCR
+  replacement lands.
+- `nomic-embed-text-v2-moe`'s 512-token limit is a hard ceiling on chunk
+  size for both strategies; any future switch to a model with a shorter
+  limit would require re-tuning chunk-size config, though not the pipeline
+  shape.
+
+## Alternatives Considered
+
+- **LLM-based document classification** (send each document to an LLM to
+  decide table/Q&A/prose shape and boundaries): rejected as the default —
+  the sample showed simple heuristics suffice, and per-document LLM calls
+  add cost/latency/non-determinism the heuristics avoid. Not ruled out as a
+  future fallback for prose blocks heuristics can't confidently segment.
+  Not needed for this sample; noted as reference only for future data.
+- **Uniform fixed-size chunking everywhere** (no structural pre-extraction):
+  rejected — would split table rows and Q&A pairs mid-content, destroying
+  the cleanest signal in the source data.
+- **Self-hosted OCR/captioning from day one**: rejected for the initial
+  cut — ChatGPT API gets image extraction working now without standing up
+  and tuning a model first; revisit once volume/cost justifies it.