feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
This commit is contained in:
33
src/application/ingestion/tokenizer.py
Normal file
33
src/application/ingestion/tokenizer.py
Normal file
@@ -0,0 +1,33 @@
|
||||
"""Token counting for chunk sizing (ADR-0018).
|
||||
|
||||
`cl100k_base` is a deliberate proxy for the embedding models' own tokenizers.
|
||||
`text-embedding-3-large` has an 8191-token window and never binds;
|
||||
`nomic-embed-text-v2-moe`'s 512-token sequence length is the only real
|
||||
constraint. cl100k tokenizes Persian inefficiently while nomic's multilingual
|
||||
tokenizer does not, so a cl100k count reliably over-estimates the nomic count --
|
||||
safe in the conservative direction, without shipping a second tokenizer and its
|
||||
model download into the ingestion path.
|
||||
"""
|
||||
|
||||
from functools import lru_cache
|
||||
|
||||
import tiktoken
|
||||
|
||||
|
||||
@lru_cache(maxsize=4)
|
||||
def get_encoder(encoding_name: str) -> tiktoken.Encoding:
|
||||
"""Return a cached tiktoken encoder.
|
||||
|
||||
Deliberately not a module-level constant: `tiktoken` fetches the BPE
|
||||
vocabulary over the network the first time an encoding is used, and
|
||||
ADR-0012 forbids external resource setup as an import-time side effect.
|
||||
The lifespan warms this at startup so a process fails fast at boot rather
|
||||
than inside the first ingestion request. Set `TIKTOKEN_CACHE_DIR` to a
|
||||
pre-populated directory for offline deployments.
|
||||
"""
|
||||
return tiktoken.get_encoding(encoding_name)
|
||||
|
||||
|
||||
def count_tokens(text: str, encoding_name: str) -> int:
|
||||
"""Return the number of tokens `text` encodes to."""
|
||||
return len(get_encoder(encoding_name).encode(text))
|
||||
Reference in New Issue
Block a user