docs: capture pdf domain language and deferred ingestion backlog

Co-authored-by: Cursor <cursoragent@cursor.com>

CONTEXT.md

@@ -19,9 +19,33 @@ The ability to visualize what a specific strategy produces for a given document
 _Avoid_: Chunk inspection, chunk view
 
 **Tab**:
-A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives.
+A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
 _Avoid_: Page, route, view
 
+**PDF Workspace**:
+The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
+_Avoid_: PDF page, PDF mode, PDF dashboard
+
+**Text PDF**:
+A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1.
+_Avoid_: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk)
+
+**Scanned PDF**:
+A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1.
+_Avoid_: Image PDF, photo PDF, OCR PDF
+
+**Heading Reconstruction**:
+Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown `#` / `##` for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present.
+_Avoid_: Heading detection, structure recovery, outline parsing (too narrow)
+
+**Text-layer Gate**:
+Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus.
+_Avoid_: OCR check, PDF validation, empty-page filter
+
+**Table Flattening**:
+Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
+_Avoid_: Table extraction, table parsing, structured tables
+
 ## Architecture Decisions
 
 | # | Decision | Status |
@@ -40,3 +64,5 @@ _Avoid_: Page, route, view
  ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved |
  ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved |
  ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
+ ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
+ ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |

backlog/README.md

@@ -0,0 +1,12 @@
+# Backlog — PDF & ingestion follow-ups
+
+Items deferred from PDF ingestion v1 (Text PDF + PyMuPDF + Heading Reconstruction).
+
+| Item | When to pick up |
+|------|-----------------|
+| [Scanned PDF OCR](./scanned-pdf-ocr.md) | After Text PDF Experiments are stable; chatbot needs image PDFs |
+| [Persian / cloud OCR](./persian-cloud-ocr.md) | Local OCR quality on Farsi sample set is too low |
+| [pdfplumber tables](./pdfplumber-tables.md) | PyMuPDF table flattening hurts Experiment scores |
+| [LibreOffice PDF→DOCX fallback](./libreoffice-pdf-to-docx-fallback.md) | Heading Reconstruction false-positives dominate; want second opinion path |
+| [Multi-column layout](./multi-column-layout.md) | Reading order is wrong on multi-column policies |
+| [Page metadata in chunks](./page-metadata-in-chunks.md) | Users need “page N” citations in answers / Chunk Preview |

backlog/libreoffice-pdf-to-docx-fallback.md

@@ -0,0 +1,20 @@
+# LibreOffice PDF→DOCX fallback
+
+## Why deferred
+
+We already convert `.doc` → `.docx` via LibreOffice. PDF→DOCX is tempting for “reuse the DOCX parser,” but styles/fonts often come out wrong — fighting the reason we chose PyMuPDF.
+
+## Trigger
+
+Heading Reconstruction on PyMuPDF has an unacceptable false-positive/false-negative rate on a representative sample, and bookmarks/fonts are absent.
+
+## Approach to try
+
+1. Optional fallback: `soffice --headless --convert-to docx` then existing `parse_docx`
+2. Compare markdown side-by-side (PyMuPDF vs converted DOCX) in Chunk Preview before making it default
+3. Keep as opt-in or auto-fallback only when PDF outline + font variance is below a threshold
+
+## Success criteria
+
+- Converted DOCX headings beat PyMuPDF heuristics on the failing sample set
+- Conversion time acceptable for upload UX (or async later — currently sync REST)

backlog/multi-column-layout.md

@@ -0,0 +1,20 @@
+# Multi-column layout reading order
+
+## Why deferred
+
+Many Text PDFs are single-column policies. Multi-column (or sidebar) layouts can make naive top-to-bottom extraction interleave columns.
+
+## Trigger
+
+Chunk Preview shows sentences from column A mixed with column B; Experiment Faithfulness drops on those docs only.
+
+## Approach to try
+
+1. Use PyMuPDF block/bbox clustering to detect columns
+2. Read column-by-column (right-to-left for Farsi multi-column if applicable)
+3. Add a fixture PDF with known two-column layout to regression tests
+
+## Success criteria
+
+- Reading order matches human reading on the fixture
+- No regression on single-column Farsi docs

backlog/page-metadata-in-chunks.md

@@ -0,0 +1,20 @@
+# Page metadata in chunks
+
+## Why deferred
+
+v1 Strategies and Qdrant payloads don’t need page numbers. Page-as-section markdown was rejected (arbitrary cuts). Page info is citation UX, not chunking semantics.
+
+## Trigger
+
+Users (or the chatbot) need “see page N” citations, or Chunk Preview should show page ranges per chunk.
+
+## Approach to try
+
+1. During PDF parse, map character/block offsets → page numbers
+2. Attach optional `page_start` / `page_end` on Chunk / ChunkMetadata (nullable for DOCX)
+3. Do **not** inject `## Page N` into markdown unless a Strategy specifically needs it
+
+## Success criteria
+
+- DOCX path unchanged (null pages)
+- Query/answer path can cite pages without changing Strategy ranking logic

backlog/pdfplumber-tables.md

@@ -0,0 +1,20 @@
+# pdfplumber for tables
+
+## Why deferred
+
+v1 uses PyMuPDF only. Dual extractors add complexity before we know tables are the failure mode.
+
+## Trigger
+
+Chunk Preview / Experiments show table cells concatenated in wrong order, or key cells missing, on table-heavy insurance PDFs — after Heading Reconstruction is otherwise fine.
+
+## Approach to try
+
+1. Keep PyMuPDF for body text + Heading Reconstruction
+2. Detect table regions (PyMuPDF or pdfplumber) and extract those via pdfplumber
+3. Flatten tables the same way DOCX does (sequential text blocks), not HTML tables in markdown (unless Strategies learn to use them)
+
+## Success criteria
+
+- Questions that answer from table cells improve without regressing prose headings
+- No second full-document parse path unless necessary

backlog/persian-cloud-ocr.md

@@ -0,0 +1,22 @@
+# Persian / cloud OCR
+
+## Why deferred
+
+Local OCR (e.g. Tesseract + `fas`/`fa`) may be good enough — or may destroy RTL, digits, and legal phrasing. Don’t commit to a vendor until measured.
+
+## Trigger
+
+Local OCR on a Farsi Scanned PDF sample set produces systematically bad characters, broken line order, or unusable tables.
+
+## Options to A/B
+
+| Option | Pros | Cons |
+|--------|------|------|
+| Tesseract + Persian models | Free, offline, no data leaving the box | Quality varies; tuning burden |
+| Cloud OCR (Google / Azure / etc.) | Often better on Persian print | Cost, latency, compliance |
+| Commercial Persian-focused OCR | Domain fit | Vendor lock-in |
+
+## Success criteria
+
+- Character error rate low enough that answer Faithfulness in Experiments doesn’t collapse vs Text PDF baseline
+- RTL paragraphs stay coherent in Chunk Preview

backlog/scanned-pdf-ocr.md

@@ -0,0 +1,25 @@
+# Scanned PDF OCR (v1.1+)
+
+## Why deferred
+
+v1 only accepts Text PDFs and applies a Text-layer Gate: Scanned PDFs are hard-rejected at upload (not stored, not processed). OCR is a second product surface: noisy text poisons every Strategy equally and makes Experiment comparisons misleading.
+
+## Trigger
+
+Pick this up when:
+
+1. Text PDF path is trusted on real docs
+2. You have ~10–20 labeled Scanned PDFs + question JSONs matching chatbot traffic
+3. Production coverage of those docs matters more than clean benchmarks alone
+
+## Approach to try
+
+1. Detect empty / near-empty text layer at upload (same gate as v1 rejection)
+2. Run OCR as an **explicit pre-step** that still emits `ParseResult` (markdown + DocumentTree)
+3. Tag source as `ocr` so Experiments can filter Text PDF vs Scanned PDF runs
+4. Reuse Heading Reconstruction heuristics on OCR text (expect more false positives)
+
+## Success criteria
+
+- OCR’d markdown is readable enough that recursive Strategy headings aren’t nonsense
+- Side-by-side Experiment: Text PDF original vs OCR of a scan of the same doc — score delta understood, not mysterious