# Backlog

Ideas and open questions not yet ready to be an ADR decision or a plan phase.
Each entry is short: what the idea is, which ADR/plan it would eventually
touch, and what's still unresolved. When an entry is picked up, turn it into
an ADR amendment (or a new ADR) and delete it from here — this file is not a
permanent record, `docs/adr/` is.

## Legacy `.doc` conversion

Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).

`.doc` is rejected with `415`. ADR-0004 specified `soffice --headless
--convert-to docx`, which ADR-0018 rejected as a multi-second subprocess inside
an inline request. Gotenberg is the obvious candidate since it is already in
use elsewhere — **but verify before committing to it**: Gotenberg's LibreOffice
route is built for converting *to PDF*, and `.doc` → `.docx` output may not be
supported on that endpoint. If it is not, the options are a dedicated
LibreOffice sidecar or asking uploaders to re-save.

## Structural units ADR-0004 specifies but v1 does not emit

Relates to: [ADR-0004](adr/0004-docx-csv-chunking-strategy.md),
[ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).

- **`qa_pair`**: one sample document alternates literal `سوال:`/`پاسخ:`
  paragraphs. v1 chunks it as prose, so a chunk boundary can fall between a
  question and its answer.
- **`image_caption`**: two sample documents embed images with no alt text.
  ADR-0004 routes these through a vision API at ingest; v1 drops them silently.

Both need a decision on whether heuristic detection is worth the misfire risk —
the header-detection work showed that guessing structure is expensive when wrong.

## Tables whose header cannot be proven

Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).

A table of short text over short text (`branch,city` with no numeric or long
column) is genuinely ambiguous, so v1 emits unlabeled `" | "` rows rather than
risk labeling every row from a data row. No file in the current corpus hits
this, but a future one will.

The honest fix is not a better heuristic — it is to stop guessing: let the
upload declare whether a sheet has a header, since the uploader knows. That is
a `POST /v1/files` contract change, so it belongs with plan 001 Phase 3 rather
than in the parser.

## Recalibrate chunk size against nomic's tokenizer

Relates to: [ADR-0018](adr/0018-docx-and-spreadsheet-parsing-with-fixed-size-chunking.md).
**Revisit after retrieval quality is measurable** — deliberately deferred, not
forgotten.

ADR-0018 counts tokens with tiktoken `cl100k_base` and caps chunks at 512. But
512 is `nomic-embed-text-v2-moe`'s limit, measured in *nomic's* tokenizer, not
OpenAI's. Those are different units, and on Persian they differ by a lot.

Measured against a real production document (`bimeh_havades.docx`, 5,911 chars
of Farsi) via the Ollama server that already hosts the model:

| Sample | cl100k tokens | nomic tokens | ratio |
|---|---|---|---|
| 300 chars | 217 | 80 | 2.71 |
| 600 chars | 426 | 165 | 2.58 |
| 1,200 chars | 846 | 303 | 2.79 |

So **~2.7 cl100k tokens per nomic token** on Persian. The current
`chunk_size=400` is therefore about **148 nomic tokens — roughly 29% of the
512-token window**. Chunks land near 570 characters where ~1,500 would fit.

Two things this measurement also established:

- **Silent truncation is real, and now demonstrated.** Feeding 2,400 and 4,800
  characters both returned `prompt_eval_count` of exactly 512, with no error
  and no warning. This is what ADR-0004 meant by "silently truncated by the
  model, not an error", confirmed on our own hardware.
- **Measuring nomic tokens needs no new dependency.** Ollama's `/api/embed`
  returns `prompt_eval_count`, so the real count is obtainable from the
  embedding call we already have to make. Note the value saturates at 512, so
  it cannot measure anything longer than the window — calibration samples must
  stay under it.

When picking this up, decide between: raising `chunk_size`/`max_chunk_tokens`
in cl100k terms using a calibration ratio (cheap, drifts if the corpus language
mix changes); counting with nomic's own tokenizer offline via HuggingFace
`tokenizers` and its `tokenizer.json` (exact, and lighter than ADR-0018
assumed — the tokenizer file only, not the 475M-param model weights); or
keeping small chunks because neighbor expansion recovers the context anyway.

Do not change this on the ratio alone. The reason to keep 400/60/512 for now is
that smaller chunks are not automatically worse for retrieval — measure
retrieval quality first, then decide.

Also note `nomic-embed-text:latest` (v1.5) is on the same Ollama server with a
2,048-token context, but it is the English-focused model; v2-moe is the
multilingual one and the reason ADR-0004 chose it for Farsi. Do not switch to
v1.5 just to get a bigger window.

## Get LLM usage/price from the OpenAI API

Relates to: [ADR-0009](adr/0009-postgres-sqlalchemy-alembic-schema.md)'s
`llm_calls`/`llm_pricing` tables.

Get token usage and price from the OpenAI API's response metadata, instead of
computing/tracking them ourselves. Need to check whether OpenAI actually
returns price, or only token counts — if only counts, we still need
`llm_pricing` for price and this only changes how `llm_calls` gets its
usage numbers.