feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
This commit is contained in:
14
.env.example
14
.env.example
@@ -44,3 +44,17 @@ INGESTION_EMBED_CONCURRENCY=4
|
||||
# Qdrant
|
||||
QDRANT_URL=http://127.0.0.1:6343
|
||||
QDRANT_API_KEY=
|
||||
|
||||
# Parsing and chunking (ADR-0018).
|
||||
# max_chunk_tokens is nomic-embed-text-v2-moe's sequence length; text past it
|
||||
# is silently truncated by the model, so the cap is enforced before embedding.
|
||||
# chunk_size sits under it to leave room for the `search_document: ` prefix.
|
||||
CHUNKING_STRATEGY=fixed_size
|
||||
CHUNKING_CHUNK_SIZE=400
|
||||
CHUNKING_CHUNK_OVERLAP=60
|
||||
CHUNKING_MAX_CHUNK_TOKENS=512
|
||||
CHUNKING_ENCODING_NAME=cl100k_base
|
||||
|
||||
# tiktoken downloads its vocabulary on first use; point this at a
|
||||
# pre-populated directory for offline/air-gapped deployments.
|
||||
# TIKTOKEN_CACHE_DIR=
|
||||
|
||||
Reference in New Issue
Block a user