From 118255acdc9eb568ee17baaa22243ea89efd1da2 Mon Sep 17 00:00:00 2001 From: Mahdi Bazrafshan Date: Sun, 2 Aug 2026 16:59:42 +0330 Subject: [PATCH] docs: capture pdf domain language and deferred ingestion backlog Co-authored-by: Cursor --- CONTEXT.md | 28 ++++++++++++++++++++- backlog/README.md | 12 +++++++++ backlog/libreoffice-pdf-to-docx-fallback.md | 20 +++++++++++++++ backlog/multi-column-layout.md | 20 +++++++++++++++ backlog/page-metadata-in-chunks.md | 20 +++++++++++++++ backlog/pdfplumber-tables.md | 20 +++++++++++++++ backlog/persian-cloud-ocr.md | 22 ++++++++++++++++ backlog/scanned-pdf-ocr.md | 25 ++++++++++++++++++ 8 files changed, 166 insertions(+), 1 deletion(-) create mode 100644 backlog/README.md create mode 100644 backlog/libreoffice-pdf-to-docx-fallback.md create mode 100644 backlog/multi-column-layout.md create mode 100644 backlog/page-metadata-in-chunks.md create mode 100644 backlog/pdfplumber-tables.md create mode 100644 backlog/persian-cloud-ocr.md create mode 100644 backlog/scanned-pdf-ocr.md diff --git a/CONTEXT.md b/CONTEXT.md index b2db2e9..601eda3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -19,9 +19,33 @@ The ability to visualize what a specific strategy produces for a given document _Avoid_: Chunk inspection, chunk view **Tab**: -A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives. +A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives. _Avoid_: Page, route, view +**PDF Workspace**: +The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI. +_Avoid_: PDF page, PDF mode, PDF dashboard + +**Text PDF**: +A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1. +_Avoid_: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk) + +**Scanned PDF**: +A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1. +_Avoid_: Image PDF, photo PDF, OCR PDF + +**Heading Reconstruction**: +Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown `#` / `##` for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present. +_Avoid_: Heading detection, structure recovery, outline parsing (too narrow) + +**Text-layer Gate**: +Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus. +_Avoid_: OCR check, PDF validation, empty-page filter + +**Table Flattening**: +Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1. +_Avoid_: Table extraction, table parsing, structured tables + ## Architecture Decisions | # | Decision | Status | @@ -40,3 +64,5 @@ _Avoid_: Page, route, view ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved | ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved | ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved | + ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved | + ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved | diff --git a/backlog/README.md b/backlog/README.md new file mode 100644 index 0000000..6496723 --- /dev/null +++ b/backlog/README.md @@ -0,0 +1,12 @@ +# Backlog — PDF & ingestion follow-ups + +Items deferred from PDF ingestion v1 (Text PDF + PyMuPDF + Heading Reconstruction). + +| Item | When to pick up | +|------|-----------------| +| [Scanned PDF OCR](./scanned-pdf-ocr.md) | After Text PDF Experiments are stable; chatbot needs image PDFs | +| [Persian / cloud OCR](./persian-cloud-ocr.md) | Local OCR quality on Farsi sample set is too low | +| [pdfplumber tables](./pdfplumber-tables.md) | PyMuPDF table flattening hurts Experiment scores | +| [LibreOffice PDF→DOCX fallback](./libreoffice-pdf-to-docx-fallback.md) | Heading Reconstruction false-positives dominate; want second opinion path | +| [Multi-column layout](./multi-column-layout.md) | Reading order is wrong on multi-column policies | +| [Page metadata in chunks](./page-metadata-in-chunks.md) | Users need “page N” citations in answers / Chunk Preview | diff --git a/backlog/libreoffice-pdf-to-docx-fallback.md b/backlog/libreoffice-pdf-to-docx-fallback.md new file mode 100644 index 0000000..57a5daf --- /dev/null +++ b/backlog/libreoffice-pdf-to-docx-fallback.md @@ -0,0 +1,20 @@ +# LibreOffice PDF→DOCX fallback + +## Why deferred + +We already convert `.doc` → `.docx` via LibreOffice. PDF→DOCX is tempting for “reuse the DOCX parser,” but styles/fonts often come out wrong — fighting the reason we chose PyMuPDF. + +## Trigger + +Heading Reconstruction on PyMuPDF has an unacceptable false-positive/false-negative rate on a representative sample, and bookmarks/fonts are absent. + +## Approach to try + +1. Optional fallback: `soffice --headless --convert-to docx` then existing `parse_docx` +2. Compare markdown side-by-side (PyMuPDF vs converted DOCX) in Chunk Preview before making it default +3. Keep as opt-in or auto-fallback only when PDF outline + font variance is below a threshold + +## Success criteria + +- Converted DOCX headings beat PyMuPDF heuristics on the failing sample set +- Conversion time acceptable for upload UX (or async later — currently sync REST) diff --git a/backlog/multi-column-layout.md b/backlog/multi-column-layout.md new file mode 100644 index 0000000..43b69e8 --- /dev/null +++ b/backlog/multi-column-layout.md @@ -0,0 +1,20 @@ +# Multi-column layout reading order + +## Why deferred + +Many Text PDFs are single-column policies. Multi-column (or sidebar) layouts can make naive top-to-bottom extraction interleave columns. + +## Trigger + +Chunk Preview shows sentences from column A mixed with column B; Experiment Faithfulness drops on those docs only. + +## Approach to try + +1. Use PyMuPDF block/bbox clustering to detect columns +2. Read column-by-column (right-to-left for Farsi multi-column if applicable) +3. Add a fixture PDF with known two-column layout to regression tests + +## Success criteria + +- Reading order matches human reading on the fixture +- No regression on single-column Farsi docs diff --git a/backlog/page-metadata-in-chunks.md b/backlog/page-metadata-in-chunks.md new file mode 100644 index 0000000..5d16325 --- /dev/null +++ b/backlog/page-metadata-in-chunks.md @@ -0,0 +1,20 @@ +# Page metadata in chunks + +## Why deferred + +v1 Strategies and Qdrant payloads don’t need page numbers. Page-as-section markdown was rejected (arbitrary cuts). Page info is citation UX, not chunking semantics. + +## Trigger + +Users (or the chatbot) need “see page N” citations, or Chunk Preview should show page ranges per chunk. + +## Approach to try + +1. During PDF parse, map character/block offsets → page numbers +2. Attach optional `page_start` / `page_end` on Chunk / ChunkMetadata (nullable for DOCX) +3. Do **not** inject `## Page N` into markdown unless a Strategy specifically needs it + +## Success criteria + +- DOCX path unchanged (null pages) +- Query/answer path can cite pages without changing Strategy ranking logic diff --git a/backlog/pdfplumber-tables.md b/backlog/pdfplumber-tables.md new file mode 100644 index 0000000..e5931b0 --- /dev/null +++ b/backlog/pdfplumber-tables.md @@ -0,0 +1,20 @@ +# pdfplumber for tables + +## Why deferred + +v1 uses PyMuPDF only. Dual extractors add complexity before we know tables are the failure mode. + +## Trigger + +Chunk Preview / Experiments show table cells concatenated in wrong order, or key cells missing, on table-heavy insurance PDFs — after Heading Reconstruction is otherwise fine. + +## Approach to try + +1. Keep PyMuPDF for body text + Heading Reconstruction +2. Detect table regions (PyMuPDF or pdfplumber) and extract those via pdfplumber +3. Flatten tables the same way DOCX does (sequential text blocks), not HTML tables in markdown (unless Strategies learn to use them) + +## Success criteria + +- Questions that answer from table cells improve without regressing prose headings +- No second full-document parse path unless necessary diff --git a/backlog/persian-cloud-ocr.md b/backlog/persian-cloud-ocr.md new file mode 100644 index 0000000..5b8bd84 --- /dev/null +++ b/backlog/persian-cloud-ocr.md @@ -0,0 +1,22 @@ +# Persian / cloud OCR + +## Why deferred + +Local OCR (e.g. Tesseract + `fas`/`fa`) may be good enough — or may destroy RTL, digits, and legal phrasing. Don’t commit to a vendor until measured. + +## Trigger + +Local OCR on a Farsi Scanned PDF sample set produces systematically bad characters, broken line order, or unusable tables. + +## Options to A/B + +| Option | Pros | Cons | +|--------|------|------| +| Tesseract + Persian models | Free, offline, no data leaving the box | Quality varies; tuning burden | +| Cloud OCR (Google / Azure / etc.) | Often better on Persian print | Cost, latency, compliance | +| Commercial Persian-focused OCR | Domain fit | Vendor lock-in | + +## Success criteria + +- Character error rate low enough that answer Faithfulness in Experiments doesn’t collapse vs Text PDF baseline +- RTL paragraphs stay coherent in Chunk Preview diff --git a/backlog/scanned-pdf-ocr.md b/backlog/scanned-pdf-ocr.md new file mode 100644 index 0000000..0709f1c --- /dev/null +++ b/backlog/scanned-pdf-ocr.md @@ -0,0 +1,25 @@ +# Scanned PDF OCR (v1.1+) + +## Why deferred + +v1 only accepts Text PDFs and applies a Text-layer Gate: Scanned PDFs are hard-rejected at upload (not stored, not processed). OCR is a second product surface: noisy text poisons every Strategy equally and makes Experiment comparisons misleading. + +## Trigger + +Pick this up when: + +1. Text PDF path is trusted on real docs +2. You have ~10–20 labeled Scanned PDFs + question JSONs matching chatbot traffic +3. Production coverage of those docs matters more than clean benchmarks alone + +## Approach to try + +1. Detect empty / near-empty text layer at upload (same gate as v1 rejection) +2. Run OCR as an **explicit pre-step** that still emits `ParseResult` (markdown + DocumentTree) +3. Tag source as `ocr` so Experiments can filter Text PDF vs Scanned PDF runs +4. Reuse Heading Reconstruction heuristics on OCR text (expect more false positives) + +## Success criteria + +- OCR’d markdown is readable enough that recursive Strategy headings aren’t nonsense +- Side-by-side Experiment: Text PDF original vs OCR of a scan of the same doc — score delta understood, not mysterious