feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)

Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.

docs/backlog.md

@@ -0,0 +1,109 @@
+# Backlog
+
+Ideas and open questions not yet ready to be an ADR decision or a plan phase.
+Each entry is short: what the idea is, which ADR/plan it would eventually
+touch, and what's still unresolved. When an entry is picked up, turn it into
+an ADR amendment (or a new ADR) and delete it from here — this file is not a
+permanent record, `docs/adr/` is.
+
+## Legacy `.doc` conversion
+
+Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
+
+`.doc` is rejected with `415`. ADR-0004 specified `soffice --headless
+--convert-to docx`, which ADR-0018 rejected as a multi-second subprocess inside
+an inline request. Gotenberg is the obvious candidate since it is already in
+use elsewhere — **but verify before committing to it**: Gotenberg's LibreOffice
+route is built for converting *to PDF*, and `.doc` → `.docx` output may not be
+supported on that endpoint. If it is not, the options are a dedicated
+LibreOffice sidecar or asking uploaders to re-save.
+
+## Structural units ADR-0004 specifies but v1 does not emit
+
+Relates to: [ADR-0004](adr/0004-docx-csv-chunking-strategy.md),
+[ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
+
+- **`qa_pair`**: one sample document alternates literal `سوال:`/`پاسخ:`
+  paragraphs. v1 chunks it as prose, so a chunk boundary can fall between a
+  question and its answer.
+- **`image_caption`**: two sample documents embed images with no alt text.
+  ADR-0004 routes these through a vision API at ingest; v1 drops them silently.
+
+Both need a decision on whether heuristic detection is worth the misfire risk —
+the header-detection work showed that guessing structure is expensive when wrong.
+
+## Tables whose header cannot be proven
+
+Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
+
+A table of short text over short text (`branch,city` with no numeric or long
+column) is genuinely ambiguous, so v1 emits unlabeled `" | "` rows rather than
+risk labeling every row from a data row. No file in the current corpus hits
+this, but a future one will.
+
+The honest fix is not a better heuristic — it is to stop guessing: let the
+upload declare whether a sheet has a header, since the uploader knows. That is
+a `POST /v1/files` contract change, so it belongs with plan 001 Phase 3 rather
+than in the parser.
+
+## Recalibrate chunk size against nomic's tokenizer
+
+Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
+**Revisit after retrieval quality is measurable** — deliberately deferred, not
+forgotten.
+
+ADR-0018 counts tokens with tiktoken `cl100k_base` and caps chunks at 512. But
+512 is `nomic-embed-text-v2-moe`'s limit, measured in *nomic's* tokenizer, not
+OpenAI's. Those are different units, and on Persian they differ by a lot.
+
+Measured against a real production document (`bimeh_havades.docx`, 5,911 chars
+of Farsi) via the Ollama server that already hosts the model:
+
+| Sample | cl100k tokens | nomic tokens | ratio |
+|---|---|---|---|
+| 300 chars | 217 | 80 | 2.71 |
+| 600 chars | 426 | 165 | 2.58 |
+| 1,200 chars | 846 | 303 | 2.79 |
+
+So **~2.7 cl100k tokens per nomic token** on Persian. The current
+`chunk_size=400` is therefore about **148 nomic tokens — roughly 29% of the
+512-token window**. Chunks land near 570 characters where ~1,500 would fit.
+
+Two things this measurement also established:
+
+- **Silent truncation is real, and now demonstrated.** Feeding 2,400 and 4,800
+  characters both returned `prompt_eval_count` of exactly 512, with no error
+  and no warning. This is what ADR-0004 meant by "silently truncated by the
+  model, not an error", confirmed on our own hardware.
+- **Measuring nomic tokens needs no new dependency.** Ollama's `/api/embed`
+  returns `prompt_eval_count`, so the real count is obtainable from the
+  embedding call we already have to make. Note the value saturates at 512, so
+  it cannot measure anything longer than the window — calibration samples must
+  stay under it.
+
+When picking this up, decide between: raising `chunk_size`/`max_chunk_tokens`
+in cl100k terms using a calibration ratio (cheap, drifts if the corpus language
+mix changes); counting with nomic's own tokenizer offline via HuggingFace
+`tokenizers` and its `tokenizer.json` (exact, and lighter than ADR-0018
+assumed — the tokenizer file only, not the 475M-param model weights); or
+keeping small chunks because neighbor expansion recovers the context anyway.
+
+Do not change this on the ratio alone. The reason to keep 400/60/512 for now is
+that smaller chunks are not automatically worse for retrieval — measure
+retrieval quality first, then decide.
+
+Also note `nomic-embed-text:latest` (v1.5) is on the same Ollama server with a
+2,048-token context, but it is the English-focused model; v2-moe is the
+multilingual one and the reason ADR-0004 chose it for Farsi. Do not switch to
+v1.5 just to get a bigger window.
+
+## Get LLM usage/price from the OpenAI API
+
+Relates to: [ADR-0009](adr/0009-postgres-sqlalchemy-alembic-schema.md)'s
+`llm_calls`/`llm_pricing` tables.
+
+Get token usage and price from the OpenAI API's response metadata, instead of
+computing/tracking them ourselves. Need to check whether OpenAI actually
+returns price, or only token counts — if only counts, we still need
+`llm_pricing` for price and this only changes how `llm_calls` gets its
+usage numbers.