mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(documents): add text pdf ingestion with heading reconstruction

Browse Source 
Why:
- Support Text PDFs in the same DocumentTree/markdown contract as DOCX.

Changes:
- PyMuPDF parser, text-layer gate, shared heading heuristics, upload dispatch for .pdf.

Impact:
- Scanned/image PDFs are rejected at upload; requires pymupdf installed.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Mahdi Bazrafshan
2026-08-02 16:59:42 +03:30
parent afc20200f7
commit 55907a8dec
8 changed files with 418 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.env.example
View File

@@ -25,4 +25,7 @@ SEMANTIC_THRESHOLD=0.3
SEMANTIC_MIN_CHUNK_SIZE=3
# SQLite database path
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
# Text PDF gate (reject scanned/image PDFs)
PDF_MIN_TOTAL_CHARS=100
PDF_MIN_MEDIAN_CHARS_PER_PAGE=40