feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
This commit is contained in:
@@ -4,6 +4,23 @@
|
||||
|
||||
Accepted
|
||||
|
||||
> Amended by
|
||||
> [ADR-0018](0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md):
|
||||
> v1 ships **fixed-size** chunking rather than the semantic-aware default
|
||||
> below, and defers `qa_pair` detection and image captioning. Docx tables do
|
||||
> become `table_row` units as this ADR specifies, but only when they are data:
|
||||
> a table holding a cell larger than one chunk, or a cell containing nested
|
||||
> tables, is treated as page layout and its cells are chunked as prose.
|
||||
> Row labels are applied only when row 0 is provably a header, and cells are
|
||||
> joined unlabeled otherwise. No document tree is built, and no heading is
|
||||
> inferred from text. `.doc` is rejected with `415` pending an out-of-process
|
||||
> conversion service rather than shelling out to LibreOffice. ADR-0018 also
|
||||
> adds a Persian normalization step at parse time and fixes the chunk-size
|
||||
> numbers this ADR left open. The spreadsheet row-to-chunk rules, the
|
||||
> embedding model, the task-prefix invariant, and the `content_type` value set
|
||||
> below all apply unchanged — except that `.csv` is read with the standard
|
||||
> library `csv` module rather than `pandas`.
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0001 deferred two things to "when we start the docx/csv chunking work":
|
||||
|
||||
Reference in New Issue
Block a user