feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
This commit is contained in:
@@ -11,10 +11,13 @@ dependencies = [
|
||||
"fastapi[standard]==0.141.1",
|
||||
"langgraph>=1.2.10",
|
||||
"minio>=7.2.20",
|
||||
"openpyxl>=3.1.5",
|
||||
"pydantic-settings>=2.15.0",
|
||||
"python-docx>=1.2.0",
|
||||
"qdrant-client>=1.19.0",
|
||||
"sqlalchemy>=2.0.51",
|
||||
"structlog>=26.1.0",
|
||||
"tiktoken>=0.13.0",
|
||||
]
|
||||
|
||||
[dependency-groups]
|
||||
|
||||
Reference in New Issue
Block a user