feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
This commit is contained in:
41
src/application/ingestion/errors.py
Normal file
41
src/application/ingestion/errors.py
Normal file
@@ -0,0 +1,41 @@
|
||||
"""Errors raised by the parsing and chunking pipeline (ADR-0018).
|
||||
|
||||
These carry no HTTP knowledge — the API layer maps them to status codes
|
||||
(ADR-0015: `application/` contains no FastAPI request objects).
|
||||
"""
|
||||
|
||||
|
||||
class IngestionError(Exception):
|
||||
"""Base class for ingestion failures."""
|
||||
|
||||
|
||||
class DocumentParseError(IngestionError):
|
||||
"""A source file could not be decoded, opened, or yielded no text.
|
||||
|
||||
Maps to `400` per ADR-0017 ("unparseable file → 400").
|
||||
"""
|
||||
|
||||
|
||||
class UnsupportedSourceTypeError(IngestionError):
|
||||
"""A source file's type is not ingestible in this version.
|
||||
|
||||
Maps to `415`. `.doc` lands here until an out-of-process conversion
|
||||
service exists (ADR-0018).
|
||||
"""
|
||||
|
||||
|
||||
class ChunkLimitExceededError(IngestionError):
|
||||
"""A document produced more chunks than `INGESTION_MAX_CHUNKS_PER_FILE`.
|
||||
|
||||
Maps to `413` per ADR-0017.
|
||||
"""
|
||||
|
||||
|
||||
class ChunkTooLargeError(IngestionError):
|
||||
"""A chunk exceeded the embedding model's sequence length.
|
||||
|
||||
This is an internal invariant violation, not a user error: the splitter is
|
||||
supposed to make it impossible. It exists because the failure it guards
|
||||
against is silent — `nomic-embed-text-v2-moe` truncates over-long input
|
||||
without raising (ADR-0004).
|
||||
"""
|
||||
Reference in New Issue
Block a user