Ali Zarinkolah
5cdfb70085
Adds src/application/ingestion/ -- Persian normalization, DOCX body walk with structural data/layout table classification, CSV/XLSX row rendering, and fixed-size token chunking (cl100k_base, 400/60/512) -- as pure functions per ADR-0015, tested against real production documents (asia_data_sample, kept out of the repo). ADR-0018 records where this diverges from ADR-0004 (fixed-size default, no invented headings/tree, structural table classification, header-provable labeling only). Plan 001's scope line is corrected from CSV-only to DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is updated to match current implementation state.
110 lines
5.3 KiB
Markdown
110 lines
5.3 KiB
Markdown
# Backlog
|
|
|
|
Ideas and open questions not yet ready to be an ADR decision or a plan phase.
|
|
Each entry is short: what the idea is, which ADR/plan it would eventually
|
|
touch, and what's still unresolved. When an entry is picked up, turn it into
|
|
an ADR amendment (or a new ADR) and delete it from here — this file is not a
|
|
permanent record, `docs/adr/` is.
|
|
|
|
## Legacy `.doc` conversion
|
|
|
|
Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
|
|
|
|
`.doc` is rejected with `415`. ADR-0004 specified `soffice --headless
|
|
--convert-to docx`, which ADR-0018 rejected as a multi-second subprocess inside
|
|
an inline request. Gotenberg is the obvious candidate since it is already in
|
|
use elsewhere — **but verify before committing to it**: Gotenberg's LibreOffice
|
|
route is built for converting *to PDF*, and `.doc` → `.docx` output may not be
|
|
supported on that endpoint. If it is not, the options are a dedicated
|
|
LibreOffice sidecar or asking uploaders to re-save.
|
|
|
|
## Structural units ADR-0004 specifies but v1 does not emit
|
|
|
|
Relates to: [ADR-0004](adr/0004-docx-csv-chunking-strategy.md),
|
|
[ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
|
|
|
|
- **`qa_pair`**: one sample document alternates literal `سوال:`/`پاسخ:`
|
|
paragraphs. v1 chunks it as prose, so a chunk boundary can fall between a
|
|
question and its answer.
|
|
- **`image_caption`**: two sample documents embed images with no alt text.
|
|
ADR-0004 routes these through a vision API at ingest; v1 drops them silently.
|
|
|
|
Both need a decision on whether heuristic detection is worth the misfire risk —
|
|
the header-detection work showed that guessing structure is expensive when wrong.
|
|
|
|
## Tables whose header cannot be proven
|
|
|
|
Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
|
|
|
|
A table of short text over short text (`branch,city` with no numeric or long
|
|
column) is genuinely ambiguous, so v1 emits unlabeled `" | "` rows rather than
|
|
risk labeling every row from a data row. No file in the current corpus hits
|
|
this, but a future one will.
|
|
|
|
The honest fix is not a better heuristic — it is to stop guessing: let the
|
|
upload declare whether a sheet has a header, since the uploader knows. That is
|
|
a `POST /v1/files` contract change, so it belongs with plan 001 Phase 3 rather
|
|
than in the parser.
|
|
|
|
## Recalibrate chunk size against nomic's tokenizer
|
|
|
|
Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
|
|
**Revisit after retrieval quality is measurable** — deliberately deferred, not
|
|
forgotten.
|
|
|
|
ADR-0018 counts tokens with tiktoken `cl100k_base` and caps chunks at 512. But
|
|
512 is `nomic-embed-text-v2-moe`'s limit, measured in *nomic's* tokenizer, not
|
|
OpenAI's. Those are different units, and on Persian they differ by a lot.
|
|
|
|
Measured against a real production document (`bimeh_havades.docx`, 5,911 chars
|
|
of Farsi) via the Ollama server that already hosts the model:
|
|
|
|
| Sample | cl100k tokens | nomic tokens | ratio |
|
|
|---|---|---|---|
|
|
| 300 chars | 217 | 80 | 2.71 |
|
|
| 600 chars | 426 | 165 | 2.58 |
|
|
| 1,200 chars | 846 | 303 | 2.79 |
|
|
|
|
So **~2.7 cl100k tokens per nomic token** on Persian. The current
|
|
`chunk_size=400` is therefore about **148 nomic tokens — roughly 29% of the
|
|
512-token window**. Chunks land near 570 characters where ~1,500 would fit.
|
|
|
|
Two things this measurement also established:
|
|
|
|
- **Silent truncation is real, and now demonstrated.** Feeding 2,400 and 4,800
|
|
characters both returned `prompt_eval_count` of exactly 512, with no error
|
|
and no warning. This is what ADR-0004 meant by "silently truncated by the
|
|
model, not an error", confirmed on our own hardware.
|
|
- **Measuring nomic tokens needs no new dependency.** Ollama's `/api/embed`
|
|
returns `prompt_eval_count`, so the real count is obtainable from the
|
|
embedding call we already have to make. Note the value saturates at 512, so
|
|
it cannot measure anything longer than the window — calibration samples must
|
|
stay under it.
|
|
|
|
When picking this up, decide between: raising `chunk_size`/`max_chunk_tokens`
|
|
in cl100k terms using a calibration ratio (cheap, drifts if the corpus language
|
|
mix changes); counting with nomic's own tokenizer offline via HuggingFace
|
|
`tokenizers` and its `tokenizer.json` (exact, and lighter than ADR-0018
|
|
assumed — the tokenizer file only, not the 475M-param model weights); or
|
|
keeping small chunks because neighbor expansion recovers the context anyway.
|
|
|
|
Do not change this on the ratio alone. The reason to keep 400/60/512 for now is
|
|
that smaller chunks are not automatically worse for retrieval — measure
|
|
retrieval quality first, then decide.
|
|
|
|
Also note `nomic-embed-text:latest` (v1.5) is on the same Ollama server with a
|
|
2,048-token context, but it is the English-focused model; v2-moe is the
|
|
multilingual one and the reason ADR-0004 chose it for Farsi. Do not switch to
|
|
v1.5 just to get a bigger window.
|
|
|
|
## Get LLM usage/price from the OpenAI API
|
|
|
|
Relates to: [ADR-0009](adr/0009-postgres-sqlalchemy-alembic-schema.md)'s
|
|
`llm_calls`/`llm_pricing` tables.
|
|
|
|
Get token usage and price from the OpenAI API's response metadata, instead of
|
|
computing/tracking them ourselves. Need to check whether OpenAI actually
|
|
returns price, or only token counts — if only counts, we still need
|
|
`llm_pricing` for price and this only changes how `llm_calls` gets its
|
|
usage numbers.
|