mba/chunking_strategies_evaluation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: capture pdf domain language and deferred ingestion backlog

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Mahdi Bazrafshan
2026-08-02 16:59:42 +03:30
parent 987b0493ac
commit 118255acdc
8 changed files with 166 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
backlog/page-metadata-in-chunks.md Normal file
View File

@@ -0,0 +1,20 @@
# Page metadata in chunks
## Why deferred
v1 Strategies and Qdrant payloads don’t need page numbers. Page-as-section markdown was rejected (arbitrary cuts). Page info is citation UX, not chunking semantics.
## Trigger
Users (or the chatbot) need “see page N” citations, or Chunk Preview should show page ranges per chunk.
## Approach to try
1. During PDF parse, map character/block offsets → page numbers
2. Attach optional `page_start` / `page_end` on Chunk / ChunkMetadata (nullable for DOCX)
3. Do **not** inject `## Page N` into markdown unless a Strategy specifically needs it
## Success criteria
- DOCX path unchanged (null pages)
- Query/answer path can cite pages without changing Strategy ranking logic