Mahdi Bazrafshan
55907a8dec
Why: - Support Text PDFs in the same DocumentTree/markdown contract as DOCX. Changes: - PyMuPDF parser, text-layer gate, shared heading heuristics, upload dispatch for .pdf. Impact: - Scanned/image PDFs are rejected at upload; requires pymupdf installed. Co-authored-by: Cursor <cursoragent@cursor.com>
1.1 KiB
1.1 KiB
PyMuPDF for Text PDF extraction
Text PDF ingestion uses PyMuPDF (fitz) so Heading Reconstruction can read font size/weight, text blocks, and outline bookmarks — the same structural signals python-docx gives us for DOCX. We rejected flat string extractors (pypdf), table-first libraries as the primary path (pdfplumber), and LibreOffice PDF→DOCX conversion (lossy styles, slow, fights ADR 0006’s “prefer native structure”).
Considered Options
- PyMuPDF — chosen; best fit for Heading Reconstruction; AGPL acceptable while this stays an internal benchmarker
- pdfplumber — strong tables, weaker hierarchy; deferred to backlog if tables are a measured failure
- pypdf — too little layout/font signal
- LibreOffice PDF→DOCX → existing parser — reuses DOCX path but conversion quality is unreliable
Consequences
- Add
pymupdfdependency; keep a single PDF code path in the documents parser seam - If we later ship the parser as a distributed service, revisit AGPL vs a permissive alternative
- Table-heavy and Scanned PDF work stays out of this ADR (see
backlog/)