chatbot_v3/docs/adr/0004-docx-csv-chunking-strategy.md

0004. Document parsing and chunking strategy (docx / xlsx / csv)

Status

Accepted

Amended by ADR-0018: v1 ships fixed-size chunking rather than the semantic-aware default below, and defers qa_pair detection and image captioning. Docx tables do become table_row units as this ADR specifies, but only when they are data: a table holding a cell larger than one chunk, or a cell containing nested tables, is treated as page layout and its cells are chunked as prose. Row labels are applied only when row 0 is provably a header, and cells are joined unlabeled otherwise. No document tree is built, and no heading is inferred from text. .doc is rejected with 415 pending an out-of-process conversion service rather than shelling out to LibreOffice. ADR-0018 also adds a Persian normalization step at parse time and fixes the chunk-size numbers this ADR left open. The spreadsheet row-to-chunk rules, the embedding model, the task-prefix invariant, and the content_type value set below all apply unchanged — except that .csv is read with the standard library csv module rather than pandas.

Context

ADR-0001 deferred two things to "when we start the docx/csv chunking work": the exact content_type value set, and any document-context payload fields. That work starts now. This ADR also fixes the dense embedding model, which ADR-0001 left as TBD — it directly constrains chunk sizing.

A representative sample of real source files (new_asia_csvs/, insurance documents from Bimeh Asia) was inspected directly (python-docx/openpyxl) before deciding anything, rather than assuming a shape. Findings:

• No .csv files exist in practice — the tree is entirely .xlsx (plus two legacy .doc binaries python-docx cannot open at all).
• Spreadsheets come in two shapes: clean two-column Q&A tables (q/a or سوال/پاسخ), and directory/contact tables (branches, phone numbers) with a merged title row, a blank separator row, and merged cells (openpyxl only stores a merged range's value in its top-left cell — the rest read as empty). Several files carry a dead second sheet.
• Word documents have no consistent shape at all: some are flowing prose with real Heading styles (rare), most are flowing prose with no heading styles, structure only implied by text ("1- ... پاسخ: ..."); one file is a single 35×5 table with zero body paragraphs; one file alternates literal "سوال:"/"پاسخ:" paragraphs; two files contain embedded images with no alt text.

Given this, no single chunking algorithm can be applied uniformly — table content and flowing prose need different treatment, and even flowing prose varies in whether headings can be trusted as section boundaries.

The user ran an offline comparison of five chunking strategies against this data and found semantic-aware chunking produced the highest retrieval accuracy, with fixed-size chunking (paired with the previous/next pointer fields from ADR-0001) a viable, simpler runner-up.

Embedding model: nomic-embed-text-v2-moe (dense only — sparse and late-interaction models remain TBD per ADR-0001/0003). Relevant constraints from its model card: 768-dim output, Matryoshka-truncatable down to 256; 512-token max sequence length; requires a task-instruction prefix on every embedded string — search_document: at ingestion time, search_query: on the agent's query side (ADR-0003).

Amendment — the task prefix is currently not applied. The emet benchmark that selected this model ran without any prefix: its Ollama deployment's template is a bare {{ .Prompt }} passthrough that injects nothing, which was verified directly against the running endpoint. The prefix is not cosmetic — embedding the same Persian text with and without search_document: yields a cosine of only 0.5741 — so applying it at ingest while the query side omits search_query: would make retrieval worse than using neither.

Implementation therefore defaults EMBEDDING_NOMIC_DOCUMENT_PREFIX to empty, matching the measured configuration, and exposes it as config so the prefixed variant is a one-line experiment rather than a code change. The model card remains the reason to expect prefixing to help; what is missing is evidence on this corpus. Turning it on is a paired change — ingest and query must move together — and should be settled by an emet run that measures the pair, not by an unmeasured edit here.

Decision

Parsing order: structural extraction before chunking

Every source file is first decomposed into structural units — table rows, Q&A pairs, prose blocks — before any chunking algorithm runs. A chunking algorithm never sees a whole document as undifferentiated text; it only runs on the prose-block units, because table rows and Q&A pairs are already atomic and splitting them would break their meaning.

1. Walk the docx body in document order (doc.element.body children, not doc.paragraphs/doc.tables separately), so tables interleaved with paragraphs keep their position, and nested tables inside cells are handled recursively. This matters for the "bad design" case the user flagged — a table cell containing an entire sub-document — which is handled as: recurse into the nested table's rows first; if a single cell still contains multiple paragraphs of unstructured prose, treat that cell's text as its own prose block and run the chunker on it, rather than emitting the whole oversized cell as one chunk.
2. Detect table rows → one structural unit per row, columns joined as "{header}: {value}" pairs (mirrors the xlsx handling below). Header detection: the first row whose cells are mostly short, unique, non-empty strings; title rows (merged, single populated cell) and blank separator rows are skipped, not treated as headers.
3. Detect Q&A pattern → one structural unit per question/answer pair, via heuristics: literal "سوال"/"پاسخ" (or q/a) paragraph pairing, or a leading enumerator ("1-", "2-") followed by a "پاسخ:"-prefixed paragraph. This is intentionally a heuristic, not an LLM classification step — it's cheap, deterministic, and the sample above showed the pattern is simple enough (leading marker + adjacent paragraph) not to need model inference per document.
4. Everything else → flowing prose blocks, segmented by heading styles where present (Heading 1/2), or the whole remaining run of paragraphs as one block where no heading styles exist.

Chunking the prose blocks

Two supported strategies, selectable by config, semantic-aware as the default:

• Semantic-aware chunking (default): split each prose block into sentences, embed each sentence, and break where adjacent-sentence similarity drops below a percentile threshold — grouping semantically coherent runs of sentences into a chunk. Chosen as default per the user's own offline accuracy comparison.
• Fixed-size chunking (config alternative): token-count-based splitting with overlap. Simpler and cheaper (no embedding pass needed just to decide boundaries), and viable specifically because ADR-0001 already gives every chunk previous_chunk_id/next_chunk_id pointers — a fixed chunk that cuts a thought in half can still be expanded with its neighbors at retrieval time (ADR-0003).

Both strategies share one hard constraint: no chunk's embedded text may exceed nomic-embed-text-v2-moe's 512-token sequence length — text beyond that is silently truncated by the model, not an error, so chunk size must be bounded well under 512 tokens regardless of which strategy is active.

Table-row and Q&A structural units bypass this chunker entirely — they are already-sized, already-atomic chunks (a row or a Q&A pair is rarely close to 512 tokens; if one is, it's truncated the same way, but this is not the common case per the sample).

Spreadsheet (xlsx / csv) handling

Row = chunk, matching the table-row handling above:

• Real header row detected (skipping merged title rows and blank separator rows); each cell rendered as "{column_header}: {cell_value}".
• Merged cells forward-filled — a merged range's value is copied to every row in that range before chunking, so each row-chunk is self-contained and doesn't silently lose data that openpyxl only attaches to the range's top-left cell.
• Sheets with no non-empty data rows (the dead second-sheet pattern seen in several sample files) are skipped, not ingested as empty chunks.
• .csv is handled identically via pandas, even though none exist in the current sample — the row → chunk mapping is the same regardless of container format.

Embedding model and prefixes

nomic-embed-text-v2-moe is the dense embedding model for ADR-0001's dense vector (768-dim, Matryoshka-truncatable to 256 if storage/latency later requires it — not adopted now, full 768 is the default). Every string sent to the model must carry its task prefix: search_document: when embedding a chunk at ingestion time, search_query: when embedding the agent's query (ADR-0003) — omitting or mismatching the prefix degrades retrieval quality per the model's own documentation. This is a pipeline invariant, not a per-call decision.

Images: LLM extraction now, self-hosted OCR later

Embedded docx images are sent to the ChatGPT (vision) API at ingestion time to extract a text description/transcription, which becomes its own chunk (content_type: image_caption) positioned in document order via the same previous_chunk_id/next_chunk_id mechanism as any other chunk — not appended silently into a neighboring text chunk. The extraction call is swappable behind a small interface so it can be replaced with a self-hosted OCR/captioning model later without changing the chunk/payload shape.

Legacy .doc files

The two legacy binary .doc files in the sample cannot be opened by python-docx. They are converted to .docx via headless LibreOffice (soffice --headless --convert-to docx) as a pipeline pre-processing step before the normal docx structural-extraction path runs — not a separate parser.

content_type value set (finalized)

Value	Meaning
paragraph	a chunk produced by the prose chunker (semantic or fixed-size)
table_row	one row from a docx table or spreadsheet
qa_pair	one detected question/answer pair
image_caption	text extracted from an embedded image

Consequences

Positive

• Structural extraction before chunking means table rows and Q&A pairs — the cleanest, highest-signal content in the sample — are never mangled by a generic text chunker.
• Semantic-aware default matches the user's own measured accuracy result rather than a generic best practice; fixed-size remains available where its lower cost matters, without losing adjacency thanks to ADR-0001's pointer fields.
• Fixes the dense embedding model (previously TBD in ADR-0001), unblocking actual implementation of ingestion and agent retrieval.
• Legacy .doc and image content are handled instead of silently dropped or erroring the whole file.

Negative

• Heuristic table/Q&A/prose detection will misclassify some future document that doesn't match the patterns in this sample; it will need tuning as new document batches are onboarded, not just this one.
• Semantic chunking adds an embedding pass at chunk-boundary-decision time, separate from the final chunk-embedding pass — extra ingestion latency and cost versus fixed-size alone.
• ChatGPT API image extraction is an external network dependency in the ingestion path (cost, latency, availability) until the self-hosted OCR replacement lands.
• nomic-embed-text-v2-moe's 512-token limit is a hard ceiling on chunk size for both strategies; any future switch to a model with a shorter limit would require re-tuning chunk-size config, though not the pipeline shape.

Alternatives Considered

• LLM-based document classification (send each document to an LLM to decide table/Q&A/prose shape and boundaries): rejected as the default — the sample showed simple heuristics suffice, and per-document LLM calls add cost/latency/non-determinism the heuristics avoid. Not ruled out as a future fallback for prose blocks heuristics can't confidently segment. Not needed for this sample; noted as reference only for future data.
• Uniform fixed-size chunking everywhere (no structural pre-extraction): rejected — would split table rows and Q&A pairs mid-content, destroying the cleanest signal in the source data.
• Self-hosted OCR/captioning from day one: rejected for the initial cut — ChatGPT API gets image extraction working now without standing up and tuning a model first; revisit once volume/cost justifies it.