Ali Zarinkolah
5cdfb70085
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
42 lines
1.2 KiB
Python
42 lines
1.2 KiB
Python
"""Errors raised by the parsing and chunking pipeline (ADR-0018).
|
|
|
|
These carry no HTTP knowledge — the API layer maps them to status codes
|
|
(ADR-0015: `application/` contains no FastAPI request objects).
|
|
"""
|
|
|
|
|
|
class IngestionError(Exception):
|
|
"""Base class for ingestion failures."""
|
|
|
|
|
|
class DocumentParseError(IngestionError):
|
|
"""A source file could not be decoded, opened, or yielded no text.
|
|
|
|
Maps to `400` per ADR-0017 ("unparseable file → 400").
|
|
"""
|
|
|
|
|
|
class UnsupportedSourceTypeError(IngestionError):
|
|
"""A source file's type is not ingestible in this version.
|
|
|
|
Maps to `415`. `.doc` lands here until an out-of-process conversion
|
|
service exists (ADR-0018).
|
|
"""
|
|
|
|
|
|
class ChunkLimitExceededError(IngestionError):
|
|
"""A document produced more chunks than `INGESTION_MAX_CHUNKS_PER_FILE`.
|
|
|
|
Maps to `413` per ADR-0017.
|
|
"""
|
|
|
|
|
|
class ChunkTooLargeError(IngestionError):
|
|
"""A chunk exceeded the embedding model's sequence length.
|
|
|
|
This is an internal invariant violation, not a user error: the splitter is
|
|
supposed to make it impossible. It exists because the failure it guards
|
|
against is silent — `nomic-embed-text-v2-moe` truncates over-long input
|
|
without raising (ADR-0004).
|
|
"""
|