Ali Zarinkolah
5cdfb70085
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
34 lines
1.3 KiB
Python
34 lines
1.3 KiB
Python
"""Token counting for chunk sizing (ADR-0018).
|
|
|
|
`cl100k_base` is a deliberate proxy for the embedding models' own tokenizers.
|
|
`text-embedding-3-large` has an 8191-token window and never binds;
|
|
`nomic-embed-text-v2-moe`'s 512-token sequence length is the only real
|
|
constraint. cl100k tokenizes Persian inefficiently while nomic's multilingual
|
|
tokenizer does not, so a cl100k count reliably over-estimates the nomic count --
|
|
safe in the conservative direction, without shipping a second tokenizer and its
|
|
model download into the ingestion path.
|
|
"""
|
|
|
|
from functools import lru_cache
|
|
|
|
import tiktoken
|
|
|
|
|
|
@lru_cache(maxsize=4)
|
|
def get_encoder(encoding_name: str) -> tiktoken.Encoding:
|
|
"""Return a cached tiktoken encoder.
|
|
|
|
Deliberately not a module-level constant: `tiktoken` fetches the BPE
|
|
vocabulary over the network the first time an encoding is used, and
|
|
ADR-0012 forbids external resource setup as an import-time side effect.
|
|
The lifespan warms this at startup so a process fails fast at boot rather
|
|
than inside the first ingestion request. Set `TIKTOKEN_CACHE_DIR` to a
|
|
pre-populated directory for offline deployments.
|
|
"""
|
|
return tiktoken.get_encoding(encoding_name)
|
|
|
|
|
|
def count_tokens(text: str, encoding_name: str) -> int:
|
|
"""Return the number of tokens `text` encodes to."""
|
|
return len(get_encoder(encoding_name).encode(text))
|