mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: capture pdf domain language and deferred ingestion backlog

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Mahdi Bazrafshan
2026-08-02 16:59:42 +03:30
parent 987b0493ac
commit 118255acdc
8 changed files with 166 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
CONTEXT.md
View File

@@ -19,9 +19,33 @@ The ability to visualize what a specific strategy produces for a given document
_Avoid_: Chunk inspection, chunk view
**Tab**:
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives.
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
_Avoid_: Page, route, view
**PDF Workspace**:
The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
_Avoid_: PDF page, PDF mode, PDF dashboard
**Text PDF**:
A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1.
_Avoid_: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk)
**Scanned PDF**:
A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1.
_Avoid_: Image PDF, photo PDF, OCR PDF
**Heading Reconstruction**:
Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown `#` / `##` for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present.
_Avoid_: Heading detection, structure recovery, outline parsing (too narrow)
**Text-layer Gate**:
Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus.
_Avoid_: OCR check, PDF validation, empty-page filter
**Table Flattening**:
Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
_Avoid_: Table extraction, table parsing, structured tables
## Architecture Decisions
| # | Decision | Status |
@@ -40,3 +64,5 @@ _Avoid_: Page, route, view
ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved |
ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved |
ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |