docs: capture pdf domain language and deferred ingestion backlog

Co-authored-by: Cursor <cursoragent@cursor.com>

backlog/README.md

@@ -0,0 +1,12 @@
+# Backlog — PDF & ingestion follow-ups
+
+Items deferred from PDF ingestion v1 (Text PDF + PyMuPDF + Heading Reconstruction).
+
+| Item | When to pick up |
+|------|-----------------|
+| [Scanned PDF OCR](./scanned-pdf-ocr.md) | After Text PDF Experiments are stable; chatbot needs image PDFs |
+| [Persian / cloud OCR](./persian-cloud-ocr.md) | Local OCR quality on Farsi sample set is too low |
+| [pdfplumber tables](./pdfplumber-tables.md) | PyMuPDF table flattening hurts Experiment scores |
+| [LibreOffice PDF→DOCX fallback](./libreoffice-pdf-to-docx-fallback.md) | Heading Reconstruction false-positives dominate; want second opinion path |
+| [Multi-column layout](./multi-column-layout.md) | Reading order is wrong on multi-column policies |
+| [Page metadata in chunks](./page-metadata-in-chunks.md) | Users need “page N” citations in answers / Chunk Preview |

backlog/libreoffice-pdf-to-docx-fallback.md

@@ -0,0 +1,20 @@
+# LibreOffice PDF→DOCX fallback
+
+## Why deferred
+
+We already convert `.doc` → `.docx` via LibreOffice. PDF→DOCX is tempting for “reuse the DOCX parser,” but styles/fonts often come out wrong — fighting the reason we chose PyMuPDF.
+
+## Trigger
+
+Heading Reconstruction on PyMuPDF has an unacceptable false-positive/false-negative rate on a representative sample, and bookmarks/fonts are absent.
+
+## Approach to try
+
+1. Optional fallback: `soffice --headless --convert-to docx` then existing `parse_docx`
+2. Compare markdown side-by-side (PyMuPDF vs converted DOCX) in Chunk Preview before making it default
+3. Keep as opt-in or auto-fallback only when PDF outline + font variance is below a threshold
+
+## Success criteria
+
+- Converted DOCX headings beat PyMuPDF heuristics on the failing sample set
+- Conversion time acceptable for upload UX (or async later — currently sync REST)

backlog/multi-column-layout.md

@@ -0,0 +1,20 @@
+# Multi-column layout reading order
+
+## Why deferred
+
+Many Text PDFs are single-column policies. Multi-column (or sidebar) layouts can make naive top-to-bottom extraction interleave columns.
+
+## Trigger
+
+Chunk Preview shows sentences from column A mixed with column B; Experiment Faithfulness drops on those docs only.
+
+## Approach to try
+
+1. Use PyMuPDF block/bbox clustering to detect columns
+2. Read column-by-column (right-to-left for Farsi multi-column if applicable)
+3. Add a fixture PDF with known two-column layout to regression tests
+
+## Success criteria
+
+- Reading order matches human reading on the fixture
+- No regression on single-column Farsi docs

backlog/page-metadata-in-chunks.md

@@ -0,0 +1,20 @@
+# Page metadata in chunks
+
+## Why deferred
+
+v1 Strategies and Qdrant payloads don’t need page numbers. Page-as-section markdown was rejected (arbitrary cuts). Page info is citation UX, not chunking semantics.
+
+## Trigger
+
+Users (or the chatbot) need “see page N” citations, or Chunk Preview should show page ranges per chunk.
+
+## Approach to try
+
+1. During PDF parse, map character/block offsets → page numbers
+2. Attach optional `page_start` / `page_end` on Chunk / ChunkMetadata (nullable for DOCX)
+3. Do **not** inject `## Page N` into markdown unless a Strategy specifically needs it
+
+## Success criteria
+
+- DOCX path unchanged (null pages)
+- Query/answer path can cite pages without changing Strategy ranking logic

backlog/pdfplumber-tables.md

@@ -0,0 +1,20 @@
+# pdfplumber for tables
+
+## Why deferred
+
+v1 uses PyMuPDF only. Dual extractors add complexity before we know tables are the failure mode.
+
+## Trigger
+
+Chunk Preview / Experiments show table cells concatenated in wrong order, or key cells missing, on table-heavy insurance PDFs — after Heading Reconstruction is otherwise fine.
+
+## Approach to try
+
+1. Keep PyMuPDF for body text + Heading Reconstruction
+2. Detect table regions (PyMuPDF or pdfplumber) and extract those via pdfplumber
+3. Flatten tables the same way DOCX does (sequential text blocks), not HTML tables in markdown (unless Strategies learn to use them)
+
+## Success criteria
+
+- Questions that answer from table cells improve without regressing prose headings
+- No second full-document parse path unless necessary

backlog/persian-cloud-ocr.md

@@ -0,0 +1,22 @@
+# Persian / cloud OCR
+
+## Why deferred
+
+Local OCR (e.g. Tesseract + `fas`/`fa`) may be good enough — or may destroy RTL, digits, and legal phrasing. Don’t commit to a vendor until measured.
+
+## Trigger
+
+Local OCR on a Farsi Scanned PDF sample set produces systematically bad characters, broken line order, or unusable tables.
+
+## Options to A/B
+
+| Option | Pros | Cons |
+|--------|------|------|
+| Tesseract + Persian models | Free, offline, no data leaving the box | Quality varies; tuning burden |
+| Cloud OCR (Google / Azure / etc.) | Often better on Persian print | Cost, latency, compliance |
+| Commercial Persian-focused OCR | Domain fit | Vendor lock-in |
+
+## Success criteria
+
+- Character error rate low enough that answer Faithfulness in Experiments doesn’t collapse vs Text PDF baseline
+- RTL paragraphs stay coherent in Chunk Preview

backlog/scanned-pdf-ocr.md

@@ -0,0 +1,25 @@
+# Scanned PDF OCR (v1.1+)
+
+## Why deferred
+
+v1 only accepts Text PDFs and applies a Text-layer Gate: Scanned PDFs are hard-rejected at upload (not stored, not processed). OCR is a second product surface: noisy text poisons every Strategy equally and makes Experiment comparisons misleading.
+
+## Trigger
+
+Pick this up when:
+
+1. Text PDF path is trusted on real docs
+2. You have ~10–20 labeled Scanned PDFs + question JSONs matching chatbot traffic
+3. Production coverage of those docs matters more than clean benchmarks alone
+
+## Approach to try
+
+1. Detect empty / near-empty text layer at upload (same gate as v1 rejection)
+2. Run OCR as an **explicit pre-step** that still emits `ParseResult` (markdown + DocumentTree)
+3. Tag source as `ocr` so Experiments can filter Text PDF vs Scanned PDF runs
+4. Reuse Heading Reconstruction heuristics on OCR text (expect more false positives)
+
+## Success criteria
+
+- OCR’d markdown is readable enough that recursive Strategy headings aren’t nonsense
+- Side-by-side Experiment: Text PDF original vs OCR of a scan of the same doc — score delta understood, not mysterious