Files

Ali Zarinkolah 5cdfb70085 feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)

Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.

2026-08-18 10:22:17 +03:30

5.3 KiB

Raw Blame History

Backlog

Ideas and open questions not yet ready to be an ADR decision or a plan phase. Each entry is short: what the idea is, which ADR/plan it would eventually touch, and what's still unresolved. When an entry is picked up, turn it into an ADR amendment (or a new ADR) and delete it from here — this file is not a permanent record, docs/adr/ is.

Legacy `.doc` conversion

Relates to: ADR-0018.

.doc is rejected with 415. ADR-0004 specified soffice --headless --convert-to docx, which ADR-0018 rejected as a multi-second subprocess inside an inline request. Gotenberg is the obvious candidate since it is already in use elsewhere — but verify before committing to it: Gotenberg's LibreOffice route is built for converting to PDF, and .doc → .docx output may not be supported on that endpoint. If it is not, the options are a dedicated LibreOffice sidecar or asking uploaders to re-save.

Structural units ADR-0004 specifies but v1 does not emit

Relates to: ADR-0004, ADR-0018.

qa_pair: one sample document alternates literal سوال:/پاسخ: paragraphs. v1 chunks it as prose, so a chunk boundary can fall between a question and its answer.
image_caption: two sample documents embed images with no alt text. ADR-0004 routes these through a vision API at ingest; v1 drops them silently.

Both need a decision on whether heuristic detection is worth the misfire risk — the header-detection work showed that guessing structure is expensive when wrong.

Tables whose header cannot be proven

Relates to: ADR-0018.

A table of short text over short text (branch,city with no numeric or long column) is genuinely ambiguous, so v1 emits unlabeled " | " rows rather than risk labeling every row from a data row. No file in the current corpus hits this, but a future one will.

The honest fix is not a better heuristic — it is to stop guessing: let the upload declare whether a sheet has a header, since the uploader knows. That is a POST /v1/files contract change, so it belongs with plan 001 Phase 3 rather than in the parser.

Recalibrate chunk size against nomic's tokenizer

Relates to: ADR-0018. Revisit after retrieval quality is measurable — deliberately deferred, not forgotten.

ADR-0018 counts tokens with tiktoken cl100k_base and caps chunks at 512. But 512 is nomic-embed-text-v2-moe's limit, measured in nomic's tokenizer, not OpenAI's. Those are different units, and on Persian they differ by a lot.

Measured against a real production document (bimeh_havades.docx, 5,911 chars of Farsi) via the Ollama server that already hosts the model:

Sample	cl100k tokens	nomic tokens	ratio
300 chars	217	80	2.71
600 chars	426	165	2.58
1,200 chars	846	303	2.79

So ~2.7 cl100k tokens per nomic token on Persian. The current chunk_size=400 is therefore about 148 nomic tokens — roughly 29% of the 512-token window. Chunks land near 570 characters where ~1,500 would fit.

Two things this measurement also established:

Silent truncation is real, and now demonstrated. Feeding 2,400 and 4,800 characters both returned prompt_eval_count of exactly 512, with no error and no warning. This is what ADR-0004 meant by "silently truncated by the model, not an error", confirmed on our own hardware.
Measuring nomic tokens needs no new dependency. Ollama's /api/embed returns prompt_eval_count, so the real count is obtainable from the embedding call we already have to make. Note the value saturates at 512, so it cannot measure anything longer than the window — calibration samples must stay under it.

When picking this up, decide between: raising chunk_size/max_chunk_tokens in cl100k terms using a calibration ratio (cheap, drifts if the corpus language mix changes); counting with nomic's own tokenizer offline via HuggingFace tokenizers and its tokenizer.json (exact, and lighter than ADR-0018 assumed — the tokenizer file only, not the 475M-param model weights); or keeping small chunks because neighbor expansion recovers the context anyway.

Do not change this on the ratio alone. The reason to keep 400/60/512 for now is that smaller chunks are not automatically worse for retrieval — measure retrieval quality first, then decide.

Also note nomic-embed-text:latest (v1.5) is on the same Ollama server with a 2,048-token context, but it is the English-focused model; v2-moe is the multilingual one and the reason ADR-0004 chose it for Farsi. Do not switch to v1.5 just to get a bigger window.

Get LLM usage/price from the OpenAI API

Relates to: ADR-0009's llm_calls/llm_pricing tables.

Get token usage and price from the OpenAI API's response metadata, instead of computing/tracking them ourselves. Need to check whether OpenAI actually returns price, or only token counts — if only counts, we still need llm_pricing for price and this only changes how llm_calls gets its usage numbers.

5.3 KiB Raw Blame History