Compare commits

..

5 Commits

Author SHA1 Message Date
118255acdc docs: capture pdf domain language and deferred ingestion backlog
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 16:59:42 +03:30
987b0493ac feat(reports): add answer compare modal and source format badge
Why:
- Make per-question strategy answers inspectable and show DOCX vs Text PDF source.

Changes:
- Side-by-side compare modal with HTML escaping; header source badge from filename.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 16:59:42 +03:30
9ce2072c23 feat(dashboard): add pdf workspace via shared format filter
Why:
- Avoid a forked PDF UI while scoping Word vs PDF workflows.

Changes:
- formatFilter on shared tabs, PDF Workspace accordion tab, word-only defaults elsewhere.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 16:59:42 +03:30
55907a8dec feat(documents): add text pdf ingestion with heading reconstruction
Why:
- Support Text PDFs in the same DocumentTree/markdown contract as DOCX.

Changes:
- PyMuPDF parser, text-layer gate, shared heading heuristics, upload dispatch for .pdf.

Impact:
- Scanned/image PDFs are rejected at upload; requires pymupdf installed.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 16:59:42 +03:30
afc20200f7 build(deps): add pymupdf for text pdf extraction
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 16:59:42 +03:30
20 changed files with 1064 additions and 325 deletions

View File

@@ -26,3 +26,6 @@ SEMANTIC_MIN_CHUNK_SIZE=3
# SQLite database path
DATABASE_URL=sqlite:///./data/chunking_benchmark.db
# Text PDF gate (reject scanned/image PDFs)
PDF_MIN_TOTAL_CHARS=100
PDF_MIN_MEDIAN_CHARS_PER_PAGE=40

View File

@@ -19,9 +19,33 @@ The ability to visualize what a specific strategy produces for a given document
_Avoid_: Chunk inspection, chunk view
**Tab**:
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin). Tabs stay mounted when switching — state survives.
A persistent top-level navigation section of the Dashboard (Home, Documents, Query, Benchmarks, Admin, PDF). Tabs stay mounted when switching — state survives.
_Avoid_: Page, route, view
**PDF Workspace**:
The PDF Tab’s end-to-end surface for Text PDF work: upload, process Strategies, Chunk Preview, query, run Experiments, and open reports — scoped to PDF documents only. Documents / Query / Benchmarks Tabs list Word documents only (`.doc`/`.docx`). Inside the Tab, features are collapsible accordion sections (Upload & Process open by default); Admin-only ops (health, Qdrant CRUD) stay on Admin. Built from shared Dashboard section components with a format filter — not a forked UI.
_Avoid_: PDF page, PDF mode, PDF dashboard
**Text PDF**:
A PDF with a real, selectable text layer that can be extracted without OCR. In scope for PDF ingestion v1.
_Avoid_: Digital PDF, native PDF, searchable PDF (ambiguous in ops talk)
**Scanned PDF**:
A PDF whose pages are images (or have a useless/empty text layer) and need OCR before chunking. Out of scope for PDF ingestion v1.
_Avoid_: Image PDF, photo PDF, OCR PDF
**Heading Reconstruction**:
Inferring section/article boundaries from PDF signals in priority order — outline bookmarks, then font size/weight, then Farsi/English text heuristics — and emitting markdown `#` / `##` for Strategies. Later signals fill gaps; they do not override outline titles when an outline is present.
_Avoid_: Heading detection, structure recovery, outline parsing (too narrow)
**Text-layer Gate**:
Upload-time check that a PDF has enough extractable text to count as a Text PDF; failure rejects the upload. Scanned PDFs never enter the benchmark corpus.
_Avoid_: OCR check, PDF validation, empty-page filter
**Table Flattening**:
Turning table cells into sequential plain-text blocks in reading order (same contract for DOCX and Text PDF). Strategies never receive grid/markdown-table structure in v1.
_Avoid_: Table extraction, table parsing, structured tables
## Architecture Decisions
| # | Decision | Status |
@@ -40,3 +64,5 @@ _Avoid_: Page, route, view
ADR-0012 | Chunk Preview: table with expandable rows. Select document + strategy, click Load. Rows show index + 80-char text preview + token/char counts. Click to expand full text. Parent column hidden by default. | Approved |
ADR-0013 | Questions Management: file list table + upload button + expandable row detail (id, question, category, difficulty, expected answer) + delete + "Use This File" shortcut to set Benchmarks tab path and switch tabs. Auto-refresh after mutations. | Approved |
ADR-0014 | Cost Estimator: two number inputs (questions, strategies), Estimate button, result card with total cost in amber, token estimate, 3 breakdown cards (embedding/queries/evaluation). Simple numbers, no tables. | Approved |
ADR-0016 | Text PDF extraction via PyMuPDF + Heading Reconstruction; Scanned PDFs hard-rejected (Text-layer Gate). See docs/adr/0016-*.md | Approved |
ADR-0017 | PDF Workspace via shared Dashboard sections + format filter (not a forked UI). See docs/adr/0017-*.md | Approved |

12
backlog/README.md Normal file
View File

@@ -0,0 +1,12 @@
# Backlog — PDF & ingestion follow-ups
Items deferred from PDF ingestion v1 (Text PDF + PyMuPDF + Heading Reconstruction).
| Item | When to pick up |
|------|-----------------|
| [Scanned PDF OCR](./scanned-pdf-ocr.md) | After Text PDF Experiments are stable; chatbot needs image PDFs |
| [Persian / cloud OCR](./persian-cloud-ocr.md) | Local OCR quality on Farsi sample set is too low |
| [pdfplumber tables](./pdfplumber-tables.md) | PyMuPDF table flattening hurts Experiment scores |
| [LibreOffice PDF→DOCX fallback](./libreoffice-pdf-to-docx-fallback.md) | Heading Reconstruction false-positives dominate; want second opinion path |
| [Multi-column layout](./multi-column-layout.md) | Reading order is wrong on multi-column policies |
| [Page metadata in chunks](./page-metadata-in-chunks.md) | Users need “page N” citations in answers / Chunk Preview |

View File

@@ -0,0 +1,20 @@
# LibreOffice PDF→DOCX fallback
## Why deferred
We already convert `.doc` → `.docx` via LibreOffice. PDF→DOCX is tempting for “reuse the DOCX parser,” but styles/fonts often come out wrong — fighting the reason we chose PyMuPDF.
## Trigger
Heading Reconstruction on PyMuPDF has an unacceptable false-positive/false-negative rate on a representative sample, and bookmarks/fonts are absent.
## Approach to try
1. Optional fallback: `soffice --headless --convert-to docx` then existing `parse_docx`
2. Compare markdown side-by-side (PyMuPDF vs converted DOCX) in Chunk Preview before making it default
3. Keep as opt-in or auto-fallback only when PDF outline + font variance is below a threshold
## Success criteria
- Converted DOCX headings beat PyMuPDF heuristics on the failing sample set
- Conversion time acceptable for upload UX (or async later — currently sync REST)

View File

@@ -0,0 +1,20 @@
# Multi-column layout reading order
## Why deferred
Many Text PDFs are single-column policies. Multi-column (or sidebar) layouts can make naive top-to-bottom extraction interleave columns.
## Trigger
Chunk Preview shows sentences from column A mixed with column B; Experiment Faithfulness drops on those docs only.
## Approach to try
1. Use PyMuPDF block/bbox clustering to detect columns
2. Read column-by-column (right-to-left for Farsi multi-column if applicable)
3. Add a fixture PDF with known two-column layout to regression tests
## Success criteria
- Reading order matches human reading on the fixture
- No regression on single-column Farsi docs

View File

@@ -0,0 +1,20 @@
# Page metadata in chunks
## Why deferred
v1 Strategies and Qdrant payloads don’t need page numbers. Page-as-section markdown was rejected (arbitrary cuts). Page info is citation UX, not chunking semantics.
## Trigger
Users (or the chatbot) need “see page N” citations, or Chunk Preview should show page ranges per chunk.
## Approach to try
1. During PDF parse, map character/block offsets → page numbers
2. Attach optional `page_start` / `page_end` on Chunk / ChunkMetadata (nullable for DOCX)
3. Do **not** inject `## Page N` into markdown unless a Strategy specifically needs it
## Success criteria
- DOCX path unchanged (null pages)
- Query/answer path can cite pages without changing Strategy ranking logic

View File

@@ -0,0 +1,20 @@
# pdfplumber for tables
## Why deferred
v1 uses PyMuPDF only. Dual extractors add complexity before we know tables are the failure mode.
## Trigger
Chunk Preview / Experiments show table cells concatenated in wrong order, or key cells missing, on table-heavy insurance PDFs — after Heading Reconstruction is otherwise fine.
## Approach to try
1. Keep PyMuPDF for body text + Heading Reconstruction
2. Detect table regions (PyMuPDF or pdfplumber) and extract those via pdfplumber
3. Flatten tables the same way DOCX does (sequential text blocks), not HTML tables in markdown (unless Strategies learn to use them)
## Success criteria
- Questions that answer from table cells improve without regressing prose headings
- No second full-document parse path unless necessary

View File

@@ -0,0 +1,22 @@
# Persian / cloud OCR
## Why deferred
Local OCR (e.g. Tesseract + `fas`/`fa`) may be good enough — or may destroy RTL, digits, and legal phrasing. Don’t commit to a vendor until measured.
## Trigger
Local OCR on a Farsi Scanned PDF sample set produces systematically bad characters, broken line order, or unusable tables.
## Options to A/B
| Option | Pros | Cons |
|--------|------|------|
| Tesseract + Persian models | Free, offline, no data leaving the box | Quality varies; tuning burden |
| Cloud OCR (Google / Azure / etc.) | Often better on Persian print | Cost, latency, compliance |
| Commercial Persian-focused OCR | Domain fit | Vendor lock-in |
## Success criteria
- Character error rate low enough that answer Faithfulness in Experiments doesn’t collapse vs Text PDF baseline
- RTL paragraphs stay coherent in Chunk Preview

View File

@@ -0,0 +1,25 @@
# Scanned PDF OCR (v1.1+)
## Why deferred
v1 only accepts Text PDFs and applies a Text-layer Gate: Scanned PDFs are hard-rejected at upload (not stored, not processed). OCR is a second product surface: noisy text poisons every Strategy equally and makes Experiment comparisons misleading.
## Trigger
Pick this up when:
1. Text PDF path is trusted on real docs
2. You have ~10–20 labeled Scanned PDFs + question JSONs matching chatbot traffic
3. Production coverage of those docs matters more than clean benchmarks alone
## Approach to try
1. Detect empty / near-empty text layer at upload (same gate as v1 rejection)
2. Run OCR as an **explicit pre-step** that still emits `ParseResult` (markdown + DocumentTree)
3. Tag source as `ocr` so Experiments can filter Text PDF vs Scanned PDF runs
4. Reuse Heading Reconstruction heuristics on OCR text (expect more false positives)
## Success criteria
- OCR’d markdown is readable enough that recursive Strategy headings aren’t nonsense
- Side-by-side Experiment: Text PDF original vs OCR of a scan of the same doc — score delta understood, not mysterious

View File

@@ -0,0 +1,16 @@
# PyMuPDF for Text PDF extraction
Text PDF ingestion uses PyMuPDF (`fitz`) so Heading Reconstruction can read font size/weight, text blocks, and outline bookmarks — the same structural signals `python-docx` gives us for DOCX. We rejected flat string extractors (`pypdf`), table-first libraries as the primary path (`pdfplumber`), and LibreOffice PDF→DOCX conversion (lossy styles, slow, fights ADR 0006’s “prefer native structure”).
## Considered Options
- **PyMuPDF** — chosen; best fit for Heading Reconstruction; AGPL acceptable while this stays an internal benchmarker
- **pdfplumber** — strong tables, weaker hierarchy; deferred to backlog if tables are a measured failure
- **pypdf** — too little layout/font signal
- **LibreOffice PDF→DOCX → existing parser** — reuses DOCX path but conversion quality is unreliable
## Consequences
- Add `pymupdf` dependency; keep a single PDF code path in the documents parser seam
- If we later ship the parser as a distributed service, revisit AGPL vs a permissive alternative
- Table-heavy and Scanned PDF work stays out of this ADR (see `backlog/`)

View File

@@ -0,0 +1,11 @@
# Shared Dashboard sections with format filter
The PDF Workspace is not a second UI. Documents, Query, Benchmarks, and Chunk Preview are composed from shared section components in `index.html`, parameterized by a format filter (`word` | `pdf`). That keeps Word Tabs and the PDF Tab behaviorally identical while listing different documents, and avoids cloning the single-file Dashboard (ADR-0004).
Rejected: copy-paste PDF Tab (diverges immediately) and “filter props only with zero extraction” when it would still duplicate large JSX blocks.
## Consequences
- Refactor existing Tab bodies to call shared panels before adding the PDF Tab
- Filtering is client-side by filename suffix; APIs stay format-agnostic
- Report generator stays shared; PDF Experiments only get a source-format badge

View File

@@ -13,6 +13,7 @@ dependencies = [
"pydantic>=2.5.0",
"pydantic-settings>=2.1.0",
"python-docx>=1.0.0",
"pymupdf>=1.24.0",
"openai>=1.6.0",
"qdrant-client>=1.7.0",
"tiktoken>=0.5.0",

View File

@@ -7,6 +7,7 @@ Design: Dark mode, amber/teal accents, Inter + JetBrains Mono.
from __future__ import annotations
import html
import json
from typing import Any
@@ -34,8 +35,104 @@ FONT_DISPLAY = "'Inter', -apple-system, sans-serif"
FONT_DATA = "'JetBrains Mono', 'Fira Code', monospace"
# ── Compare Answers Modal (JS) ────────────────────────────────────
_MODAL_JS = """
// ── Compare Answers Modal ─────────────────────────────────
function escapeHtml(str) {
if (str == null) return '';
return String(str)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;');
}
function scoreChip(label, val) {
val = Number(val) || 0;
const cls = val >= 8 ? 'high' : (val >= 6 ? 'mid' : 'low');
return '<span class="score-chip ' + cls + '">' + label + ' ' + val.toFixed(1) + '</span>';
}
function showCompare(idx) {
const q = perQuestion[idx];
if (!q) return;
document.getElementById('modalQId').textContent = 'Question ' + (q.question_id || (idx + 1));
document.getElementById('modalQuestion').textContent = q.question || '';
const expected = q.expected_answer || '';
document.getElementById('modalExpected').innerHTML = expected
? '<div class="expected-block"><strong>Expected Answer (Ground Truth)</strong><p>' + escapeHtml(expected) + '</p></div>'
: '';
const grid = document.getElementById('modalAnswers');
grid.innerHTML = strategies.map(s => {
const strat = (q.strategies || {})[s] || {};
const scores = strat.scores || {};
const hasError = !!strat.error;
const hasAnswer = !!(strat.answer && String(strat.answer).trim());
const answer = hasAnswer
? strat.answer
: (hasError ? ('Error: ' + strat.error) : 'No answer recorded.');
const chips = [];
if (scores.context_relevance != null) chips.push(scoreChip('Context', scores.context_relevance));
if (scores.answer_similarity != null) chips.push(scoreChip('Sim', scores.answer_similarity));
if (scores.faithfulness != null) chips.push(scoreChip('Faith', scores.faithfulness));
if (scores.hallucination != null) {
chips.push(scores.hallucination
? '<span class="score-chip low">Hallucinated</span>'
: '<span class="score-chip high">No Hallucination</span>');
}
if (strat.latency && strat.latency.query_seconds != null) {
chips.push('<span class="score-chip">' + Number(strat.latency.query_seconds).toFixed(2) + 's</span>');
}
const usage = strat.token_usage || {};
if (usage.total_tokens != null) {
chips.push('<span class="score-chip">' + Number(usage.total_tokens).toLocaleString() + ' tokens</span>');
}
return '<div class="answer-card">'
+ '<div class="strategy-name">' + escapeHtml(s)
+ '<span class="badge pill ' + (hasError ? 'low' : 'mid') + '">' + (hasError ? 'FAILED' : 'ANSWER') + '</span></div>'
+ '<div class="score-row">' + chips.join('') + '</div>'
+ '<div class="answer-text' + (hasAnswer ? '' : ' empty') + '">' + escapeHtml(answer) + '</div>'
+ '</div>';
}).join('');
document.getElementById('compareModal').classList.add('open');
document.body.style.overflow = 'hidden';
}
function closeCompare() {
document.getElementById('compareModal').classList.remove('open');
document.body.style.overflow = '';
}
document.getElementById('compareModal').addEventListener('click', function (e) {
if (e.target === this) closeCompare();
});
document.addEventListener('keydown', function (e) {
if (e.key === 'Escape') closeCompare();
});
"""
# ── Main Entry Points ─────────────────────────────────────────────
def _source_format_label(experiment: dict) -> str:
"""Human-readable source format badge from document filename."""
filename = (experiment.get("document_filename") or "").lower()
if filename.endswith(".pdf"):
return "Text PDF"
if filename.endswith(".docx") or filename.endswith(".doc"):
return "DOCX"
return ""
def generate_report(experiment: dict, view: str = "managerial") -> str:
"""Generate report for specified view.
@@ -67,6 +164,8 @@ def generate_managerial_report(experiment: dict) -> str:
total_completion_tokens += usage.get("completion_tokens", 0)
estimated_cost = (total_prompt_tokens * 0.15 + total_completion_tokens * 0.60) / 1_000_000
source_label = _source_format_label(experiment)
source_meta = f" &middot; Source: {source_label}" if source_label else ""
html = _base_html(experiment, "Managerial View", f"""
<!-- Header -->
@@ -76,7 +175,7 @@ def generate_managerial_report(experiment: dict) -> str:
<div class="meta">
{config.get('num_questions', 0)} questions &middot;
{len(strategies)} strategies &middot;
${estimated_cost:.4f} cost
${estimated_cost:.4f} cost{source_meta}
</div>
</header>
@@ -196,6 +295,8 @@ def generate_technical_report(experiment: dict) -> str:
# Estimate latency (rough: ~0.5s per query + ~1s per evaluation)
estimated_latency = total_queries * 1.5
source_label = _source_format_label(experiment)
source_meta = f" &middot; Source: {source_label}" if source_label else ""
html = _base_html(experiment, "Technical View", f"""
<!-- Header -->
@@ -205,7 +306,7 @@ def generate_technical_report(experiment: dict) -> str:
<div class="meta">
Experiment: {experiment.get('id', 'N/A')[:16]}... &middot;
{total_questions} questions &middot;
{len(strategies)} strategies
{len(strategies)} strategies{source_meta}
</div>
</header>
@@ -279,6 +380,7 @@ def generate_technical_report(experiment: dict) -> str:
<h2>Per-Question Results</h2>
<div class="line"></div>
</div>
<div class="hint">&#128269; Click any question to compare full answers from all strategies side by side.</div>
<div class="table-wrapper">
<table>
<thead>
@@ -489,6 +591,34 @@ def _base_html(experiment: dict, title: str, content: str) -> str:
.link-text {{ flex: 1; font-weight: 500; }}
.link-arrow {{ color: var(--accent); font-size: 18px; }}
.question-link {{ color: var(--accent); cursor: pointer; text-decoration: none; border-bottom: 1px dashed var(--accent); }}
.question-link:hover {{ color: var(--accent-dim); border-bottom-style: solid; }}
.hint {{ font-size: 12px; color: var(--text-muted); font-family: {FONT_DATA}; margin-bottom: 16px; }}
.modal-overlay {{ position: fixed; inset: 0; background: rgba(4, 8, 12, 0.72); display: none; align-items: center; justify-content: center; z-index: 1000; padding: 24px; backdrop-filter: blur(4px); }}
.modal-overlay.open {{ display: flex; }}
.modal {{ background: var(--surface); border: 1px solid var(--border); border-radius: 16px; max-width: 1100px; width: 100%; max-height: 85vh; display: flex; flex-direction: column; box-shadow: 0 24px 48px rgba(0, 0, 0, 0.5); }}
.modal-header {{ padding: 24px 28px; border-bottom: 1px solid var(--border); display: flex; align-items: flex-start; gap: 16px; }}
.modal-header .eyebrow {{ margin-bottom: 8px; }}
.modal-header h3 {{ font-size: 18px; font-weight: 600; line-height: 1.4; }}
.modal-close {{ margin-left: auto; background: var(--surface-hover); border: 1px solid var(--border); color: var(--text-muted); width: 36px; height: 36px; border-radius: 8px; font-size: 20px; line-height: 1; cursor: pointer; flex-shrink: 0; transition: all 0.2s ease; }}
.modal-close:hover {{ color: var(--text); border-color: var(--accent); }}
.modal-body {{ padding: 24px 28px; overflow-y: auto; }}
.expected-block {{ background: rgba(139, 92, 246, 0.08); border-left: 3px solid var(--violet); border-radius: 8px; padding: 16px 20px; margin-bottom: 20px; }}
.expected-block strong {{ font-size: 12px; color: var(--violet); text-transform: uppercase; letter-spacing: 1px; display: block; margin-bottom: 6px; }}
.expected-block p {{ font-size: 13px; color: var(--text); margin: 0; }}
.answer-grid {{ display: grid; grid-template-columns: repeat(auto-fit, minmax(320px, 1fr)); gap: 16px; }}
.answer-card {{ background: var(--surface-hover); border: 1px solid var(--border); border-radius: 12px; padding: 20px; display: flex; flex-direction: column; gap: 12px; }}
.answer-card .strategy-name {{ font-size: 13px; font-weight: 600; color: var(--accent); display: flex; align-items: center; justify-content: space-between; gap: 8px; }}
.answer-card .answer-text {{ font-size: 13px; line-height: 1.7; color: var(--text); white-space: pre-wrap; word-break: break-word; background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 16px; max-height: 320px; overflow-y: auto; }}
.answer-card .answer-text.empty {{ color: var(--text-muted); font-style: italic; }}
.score-row {{ display: flex; flex-wrap: wrap; gap: 8px; }}
.score-chip {{ font-family: {FONT_DATA}; font-size: 11px; padding: 3px 8px; border-radius: 6px; background: var(--surface); border: 1px solid var(--border); }}
.score-chip.high {{ color: var(--emerald); border-color: rgba(16, 185, 129, 0.4); }}
.score-chip.mid {{ color: var(--accent); border-color: rgba(245, 158, 11, 0.4); }}
.score-chip.low {{ color: var(--rose); border-color: rgba(244, 63, 94, 0.4); }}
.footer {{ padding-top: 32px; border-top: 1px solid var(--border); font-size: 12px; color: var(--text-muted); text-align: center; }}
@media (max-width: 768px) {{
@@ -508,6 +638,27 @@ def _base_html(experiment: dict, title: str, content: str) -> str:
<div class="page">
{content}
<!-- Compare Answers Modal -->
<div class="modal-overlay" id="compareModal">
<div class="modal">
<div class="modal-header">
<div>
<div class="eyebrow" id="modalQId">Question</div>
<h3 id="modalQuestion"></h3>
</div>
<button class="modal-close" onclick="closeCompare()" aria-label="Close">&times;</button>
</div>
<div class="modal-body">
<div id="modalExpected"></div>
<div class="section-header">
<h2>Answers by Strategy</h2>
<div class="line"></div>
</div>
<div class="answer-grid" id="modalAnswers"></div>
</div>
</div>
</div>
<footer class="footer">
Generated by RAG Chunking Benchmarker &middot; {experiment.get('created_at', 'N/A')}
</footer>
@@ -517,9 +668,14 @@ def _base_html(experiment: dict, title: str, content: str) -> str:
const colors = {json.dumps(CHART_COLORS)};
const strategies = {json.dumps(strategies)};
const aggregate = {json.dumps(aggregate)};
const perQuestion = {json.dumps(experiment.get("per_question", []), ensure_ascii=False).replace("</", "<\\/")};
{_MODAL_JS}
// Radar
new Chart(document.getElementById('radarChart'), {{
const radarEl = document.getElementById('radarChart');
if (radarEl) {{
new Chart(radarEl, {{
type: 'radar',
data: {{
labels: ['Context', 'Similarity', 'Faithfulness', 'Consistency'],
@@ -553,9 +709,12 @@ def _base_html(experiment: dict, title: str, content: str) -> str:
}}
}}
}});
}}
// Bar
new Chart(document.getElementById('barChart'), {{
const barEl = document.getElementById('barChart');
if (barEl) {{
new Chart(barEl, {{
type: 'bar',
data: {{
labels: strategies.map(s => s.length > 12 ? s.substring(0, 12) + '...' : s),
@@ -575,6 +734,7 @@ def _base_html(experiment: dict, title: str, content: str) -> str:
}}
}}
}});
}}
</script>
</body>
</html>"""
@@ -739,13 +899,16 @@ def _build_strategy_headers(strategies: list) -> str:
def _build_per_question_rows(per_question: list, strategies: list) -> str:
"""Build per-question rows."""
"""Build per-question rows with clickable questions opening the compare modal."""
rows = ""
for qr in per_question:
q_id = qr.get("question_id", "")
q_text = qr.get("question", "")[:40]
category = qr.get("category", "")
difficulty = qr.get("difficulty", "")
for idx, qr in enumerate(per_question):
q_id = html.escape(str(qr.get("question_id", "")))
full_q = str(qr.get("question", "") or "")
q_preview = full_q[:60] + ("..." if len(full_q) > 60 else "")
q_text = html.escape(q_preview)
category = html.escape(str(qr.get("category", "") or ""))
difficulty = str(qr.get("difficulty", "") or "")
difficulty_safe = html.escape(difficulty)
cells = ""
for strategy in strategies:
@@ -755,11 +918,11 @@ def _build_per_question_rows(per_question: list, strategies: list) -> str:
cells += f"<td>{_pill(sim)}</td>"
rows += f"""
<tr>
<tr style="cursor: pointer;" onclick="showCompare({idx})" title="Click to compare all strategy answers">
<td style="font-family: {FONT_DATA}; color: var(--text-muted);">{q_id}</td>
<td>{q_text}...</td>
<td><span class="question-link">{q_text}</span></td>
<td style="color: var(--text-muted);">{category}</td>
<td><span class="pill {'high' if difficulty == 'easy' else 'mid' if difficulty == 'medium' else 'low'}">{difficulty}</span></td>
<td><span class="pill {'high' if difficulty == 'easy' else 'mid' if difficulty == 'medium' else 'low'}">{difficulty_safe}</span></td>
{cells}
</tr>"""
@@ -770,21 +933,24 @@ def _build_detailed_rows(per_question: list, strategies: list) -> str:
"""Build detailed score rows."""
rows = ""
for qr in per_question:
q_id = qr.get("question_id", "")
q_id = html.escape(str(qr.get("question_id", "")))
for strategy in strategies:
strat = qr.get("strategies", {}).get(strategy, {})
scores = strat.get("scores", {})
answer = strat.get("answer", "")[:60]
full_answer = str(strat.get("answer", "") or "")
answer_preview = full_answer[:60] + ("..." if len(full_answer) > 60 else "")
answer = html.escape(answer_preview)
strategy_safe = html.escape(str(strategy))
rows += f"""
<tr>
<td style="font-family: {FONT_DATA}; font-size: 12px;">{q_id}</td>
<td>{strategy}</td>
<td>{strategy_safe}</td>
<td>{_pill(scores.get('context_relevance', 0))}</td>
<td>{_pill(scores.get('answer_similarity', 0))}</td>
<td>{_pill(scores.get('faithfulness', 0))}</td>
<td>{'&#10003;' if not scores.get('hallucination', False) else '&#10007;'}</td>
<td style="font-size: 13px; color: var(--text-muted);">{answer}...</td>
<td style="font-size: 13px; color: var(--text-muted);">{answer}</td>
</tr>"""
return rows

View File

@@ -35,5 +35,9 @@ class Settings(BaseSettings):
# Database
database_url: str = "sqlite:///./data/chunking_benchmark.db"
# Text PDF gate (reject Scanned PDFs with near-empty text layer)
pdf_min_total_chars: int = 100
pdf_min_median_chars_per_page: int = 40
settings = Settings()

View File

@@ -0,0 +1,79 @@
"""Shared heading detection heuristics for DOCX and Text PDF parsers.
Used when native styles/fonts/outlines are missing or incomplete
(common in table-heavy Farsi documents).
"""
from __future__ import annotations
import re
from typing import Protocol
class StyledBlock(Protocol):
"""Minimal block interface: mutable style_name + text."""
style_name: str
text: str
_HEADING_PATTERNS: list[tuple[re.Pattern[str], int]] = [
(re.compile(r"^بخش\s+"), 1),
(re.compile(r"^\d+[\-\.]\d+"), 2),
(re.compile(r"^\*\s*بخش\s+"), 1),
(re.compile(r"^\*\s*\S"), 2),
]
def heading_level_from_style(style_name: str) -> int | None:
"""Return heading level from a style name like 'Heading 1' or 'Heading1'."""
m = re.match(r"^heading\s*(\d+)$", style_name.strip(), re.IGNORECASE)
if m:
return int(m.group(1))
return None
def is_likely_heading(text: str) -> int | None:
"""Detect heading-like patterns in body text.
Returns heading level (1 or 2) if detected, else None.
"""
text = text.strip()
if len(text) > 150:
return None
for pattern, level in _HEADING_PATTERNS:
if pattern.match(text):
return level
return None
def apply_heading_heuristics(
blocks: list[StyledBlock],
*,
only_when_no_headings: bool = True,
) -> bool:
"""Upgrade Normal blocks to HeadingN via text patterns.
Args:
blocks: Mutable text blocks with style_name / text.
only_when_no_headings: If True (DOCX default), skip when any
Heading styles already exist. If False, only upgrade remaining
Normal blocks (PDF gap-fill after outline/font).
Returns:
True if any blocks were upgraded.
"""
if only_when_no_headings:
if any(heading_level_from_style(b.style_name) is not None for b in blocks):
return False
upgraded = False
for block in blocks:
if block.style_name != "Normal":
continue
level = is_likely_heading(block.text)
if level is not None:
block.style_name = f"Heading{level}"
upgraded = True
return upgraded

View File

@@ -1,18 +1,13 @@
"""DOCX parser: extracts document tree + flat markdown.
"""Document parser: DOCX/DOC + dispatcher for all supported formats.
Uses python-docx to read paragraph styles and build a hierarchical
DocumentTree (Document > Section > Article > Paragraph). Also produces
a markdown representation consumed by chunking strategies.
Produces a hierarchical DocumentTree (Document > Section > Article > Paragraph)
and markdown consumed by chunking strategies.
Supports both .docx and .doc formats. .doc files are converted to
.docx via LibreOffice headless mode before parsing.
Handles documents where content is in tables (not just paragraphs).
Supports .docx, .doc (via LibreOffice), and .pdf (via pdf_parser).
"""
from __future__ import annotations
import re
import subprocess
import tempfile
from pathlib import Path
@@ -27,6 +22,10 @@ from src.core.models import (
NodeType,
)
from src.core.exceptions import DocumentProcessingError
from src.documents.heading_heuristics import (
apply_heading_heuristics,
heading_level_from_style,
)
# ── Heading level → node type mapping ──────────────────────────────
@@ -34,89 +33,41 @@ from src.core.exceptions import DocumentProcessingError
_HEADING_MAP: dict[int, NodeType] = {
1: NodeType.SECTION,
2: NodeType.ARTICLE,
# 3+ also ARTICLE (nesting depth determines hierarchy)
}
def _heading_level(style_name: str) -> int | None:
"""Return the heading level from a style name, or None if not a heading.
Handles both "Heading 1" (display name) and "Heading1" (style ID from XML).
"""
m = re.match(r"^heading\s*(\d+)$", style_name.strip(), re.IGNORECASE)
if m:
return int(m.group(1))
return None
def _node_type_for_level(level: int) -> NodeType:
return _HEADING_MAP.get(level, NodeType.ARTICLE)
# ── Heuristic heading detection for table-heavy documents ──────────
# ── Text block ─────────────────────────────────────────────────────
# Patterns that look like section/article headings in Farsi/English docs
_HEADING_PATTERNS: list[tuple[str, int]] = [
# Farsi section markers: "بخش اول", "بخش دوم", etc.
(re.compile(r"^بخش\s+"), 1),
# Numbered sections: "1-1", "2-1", "10-1", "1-1-8"
(re.compile(r"^\d+[\-\.]\d+"), 2),
# Starred sections: "*بخش اول", "*تعریف"
(re.compile(r"^\*\s*بخش\s+"), 1),
(re.compile(r"^\*\s*\S"), 2),
]
class TextBlock:
"""A unit of text extracted from a document, preserving reading order."""
__slots__ = ("style_name", "text", "font_size", "bold", "page", "locked")
def _is_likely_heading(text: str) -> int | None:
"""Detect heading-like patterns in table-extracted text.
Returns the heading level (1 or 2) if detected, else None.
Used when the document has no real heading styles (table-only content).
"""
text = text.strip()
if len(text) > 150: # headings are short
return None
for pattern, level in _HEADING_PATTERNS:
if pattern.match(text):
return level
return None
def _detect_heading_blocks(blocks: list[_TextBlock]) -> bool:
"""Upgrade paragraph blocks to heading blocks based on content patterns.
Returns True if any blocks were upgraded.
Only activates when no real headings exist in the document.
"""
# Check if there are already real headings
has_headings = any(_heading_level(b.style_name) is not None for b in blocks)
if has_headings:
return False
upgraded = False
for block in blocks:
if block.style_name != "Normal":
continue
level = _is_likely_heading(block.text)
if level is not None:
block.style_name = f"Heading{level}"
upgraded = True
return upgraded
def __init__(
self,
style_name: str,
text: str,
*,
font_size: float = 0.0,
bold: bool = False,
page: int = 0,
locked: bool = False,
) -> None:
self.style_name = style_name
self.text = text
self.font_size = font_size
self.bold = bold
self.page = page
self.locked = locked
# ── Text block extraction (paragraphs + tables) ───────────────────
class _TextBlock:
"""A unit of text extracted from the document, preserving reading order."""
__slots__ = ("style_name", "text")
def __init__(self, style_name: str, text: str) -> None:
self.style_name = style_name
self.text = text
def _extract_text_blocks(doc: DocxDocumentType) -> list[_TextBlock]:
def _extract_text_blocks(doc: DocxDocumentType) -> list[TextBlock]:
"""Walk the document body in reading order, extracting paragraphs and tables.
This handles documents where content lives inside table cells
@@ -124,20 +75,17 @@ def _extract_text_blocks(doc: DocxDocumentType) -> list[_TextBlock]:
"""
from docx.oxml.ns import qn
blocks: list[_TextBlock] = []
blocks: list[TextBlock] = []
for element in doc.element.body:
tag = element.tag.split("}")[-1] if "}" in element.tag else element.tag
if tag == "p":
# Paragraph — extract text and style
text = element.text or ""
# Also check for runs (text split across formatting)
if not text.strip():
runs = element.findall(qn("w:r"))
text = "".join(r.text or "" for r in runs)
# Get style name
ppr = element.find(qn("w:pPr"))
style_name = "Normal"
if ppr is not None:
@@ -146,19 +94,15 @@ def _extract_text_blocks(doc: DocxDocumentType) -> list[_TextBlock]:
style_name = pstyle.get(qn("w:val"), "Normal")
if text.strip():
blocks.append(_TextBlock(style_name, text.strip()))
blocks.append(TextBlock(style_name, text.strip()))
elif tag == "tbl":
# Table — extract all cell text as paragraph blocks
for row in element.findall(qn("w:tr")):
for cell in row.findall(qn("w:tc")):
for para in cell.findall(qn("w:p")):
# Get paragraph text
text = ""
runs = para.findall(qn("w:r"))
text = "".join(r.text or "" for r in runs)
# Get style
ppr = para.find(qn("w:pPr"))
style_name = "Normal"
if ppr is not None:
@@ -167,39 +111,36 @@ def _extract_text_blocks(doc: DocxDocumentType) -> list[_TextBlock]:
style_name = pstyle.get(qn("w:val"), "Normal")
if text.strip():
blocks.append(_TextBlock(style_name, text.strip()))
blocks.append(TextBlock(style_name, text.strip()))
return blocks
# ── Tree builder ───────────────────────────────────────────────────
def build_document_tree(paragraphs: list[Paragraph] | list[_TextBlock]) -> DocumentTreeNode:
def build_document_tree(paragraphs: list[Paragraph] | list[TextBlock]) -> DocumentTreeNode:
"""Build a DocumentTreeNode tree from a list of text blocks.
Accepts either python-docx Paragraph objects or _TextBlock objects.
Accepts either python-docx Paragraph objects or TextBlock objects.
"""
root = DocumentTreeNode(node_type=NodeType.DOCUMENT, text="", heading=None, heading_level=None)
# Stack of (level, node) for current nesting. level 0 = root.
stack: list[tuple[int, DocumentTreeNode]] = [(0, root)]
for block in paragraphs:
# Get style name and text from either type
if isinstance(block, _TextBlock):
if isinstance(block, TextBlock):
style_name = block.style_name
text = block.text
else:
style_name = block.style.name
text = block.text.strip()
level = _heading_level(style_name)
level = heading_level_from_style(style_name)
if not text:
continue # skip blank paragraphs
continue
if level is not None:
# Pop back to parent level
while len(stack) > 1 and stack[-1][0] >= level:
stack.pop()
@@ -212,7 +153,6 @@ def build_document_tree(paragraphs: list[Paragraph] | list[_TextBlock]) -> Docum
stack[-1][1].children.append(node)
stack.append((level, node))
else:
# Body text — add as paragraph child of current heading
node = DocumentTreeNode(
node_type=NodeType.PARAGRAPH,
text=text,
@@ -244,7 +184,7 @@ def tree_to_markdown(node: DocumentTreeNode, depth: int = 0) -> str:
# ── Public API ─────────────────────────────────────────────────────
class ParseResult:
"""Output of parse_docx(): tree + markdown + raw text."""
"""Output of parse_document() / parse_docx() / parse_pdf()."""
__slots__ = ("tree", "markdown", "plain_text", "paragraph_count")
@@ -261,6 +201,31 @@ class ParseResult:
self.paragraph_count = paragraph_count
SUPPORTED_SUFFIXES = (".docx", ".doc", ".pdf")
def parse_document(file_path: str | Path) -> ParseResult:
"""Parse a supported document into DocumentTree + markdown.
Dispatches by suffix: .pdf → pdf_parser; .docx/.doc → parse_docx.
"""
path = Path(file_path)
if not path.exists():
raise DocumentProcessingError(f"File not found: {path}")
suffix = path.suffix.lower()
if suffix == ".pdf":
from src.documents.pdf_parser import parse_pdf
return parse_pdf(path)
if suffix in (".docx", ".doc"):
return parse_docx(path)
raise DocumentProcessingError(
f"Not a supported file format: {suffix} (expected {', '.join(SUPPORTED_SUFFIXES)})"
)
# ── .doc → .docx conversion ───────────────────────────────────────
def _convert_doc_to_docx(doc_path: Path) -> Path:
@@ -268,9 +233,6 @@ def _convert_doc_to_docx(doc_path: Path) -> Path:
Returns the path to the converted .docx file (in a temp directory).
The caller is responsible for cleanup.
Raises:
DocumentProcessingError: If conversion fails.
"""
out_dir = Path(tempfile.mkdtemp(prefix="docconv_"))
try:
@@ -291,7 +253,6 @@ def _convert_doc_to_docx(doc_path: Path) -> Path:
f"LibreOffice conversion failed: {result.stderr}"
)
# Find the converted file
converted = out_dir / doc_path.with_suffix(".docx").name
if not converted.exists():
raise DocumentProcessingError(
@@ -312,15 +273,6 @@ def parse_docx(file_path: str | Path) -> ParseResult:
"""Parse a .docx or .doc file into a DocumentTree + markdown.
.doc files are automatically converted to .docx via LibreOffice.
Args:
file_path: Path to the .docx or .doc file.
Returns:
ParseResult with tree, markdown, plain_text, and paragraph_count.
Raises:
DocumentProcessingError: If the file cannot be parsed.
"""
path = Path(file_path)
if not path.exists():
@@ -328,9 +280,10 @@ def parse_docx(file_path: str | Path) -> ParseResult:
suffix = path.suffix.lower()
if suffix not in (".docx", ".doc"):
raise DocumentProcessingError(f"Not a supported file format: {suffix} (expected .docx or .doc)")
raise DocumentProcessingError(
f"Not a supported file format: {suffix} (expected .docx or .doc)"
)
# Convert .doc to .docx if needed
if suffix == ".doc":
path = _convert_doc_to_docx(path)
@@ -339,14 +292,12 @@ def parse_docx(file_path: str | Path) -> ParseResult:
except Exception as exc:
raise DocumentProcessingError(f"Failed to open DOCX: {exc}") from exc
# Extract text blocks from paragraphs + tables (preserves reading order)
blocks = _extract_text_blocks(doc)
if not blocks:
raise DocumentProcessingError("Document contains no text content")
# Detect heading patterns in table-only documents
_detect_heading_blocks(blocks)
apply_heading_heuristics(blocks, only_when_no_headings=True)
root = build_document_tree(blocks)
tree = DocumentTree(root=root)

228
src/documents/pdf_parser.py Normal file
View File

@@ -0,0 +1,228 @@
"""Text PDF parser via PyMuPDF with Heading Reconstruction.
Pipeline:
1. Text-layer Gate — reject Scanned PDFs (near-empty text)
2. Extract reading-order blocks (Table Flattening via block order)
3. Heading Reconstruction: outline → font size → shared heuristics
4. Build DocumentTree + markdown (same contract as DOCX)
"""
from __future__ import annotations
import statistics
from pathlib import Path
import fitz # PyMuPDF
from src.core.config import settings
from src.core.exceptions import DocumentProcessingError
from src.core.models import DocumentTree
from src.documents.heading_heuristics import apply_heading_heuristics
from src.documents.parser import (
ParseResult,
TextBlock,
build_document_tree,
tree_to_markdown,
)
def _median_chars_per_page(page_char_counts: list[int]) -> float:
if not page_char_counts:
return 0.0
return float(statistics.median(page_char_counts))
def _enforce_text_layer_gate(doc: fitz.Document) -> None:
"""Hard-reject PDFs without enough extractable text (Scanned PDFs)."""
page_counts: list[int] = []
for page in doc:
page_counts.append(len(page.get_text("text").strip()))
total = sum(page_counts)
median = _median_chars_per_page(page_counts)
if total < settings.pdf_min_total_chars or median < settings.pdf_min_median_chars_per_page:
raise DocumentProcessingError(
"PDF has no usable text layer (likely a scanned/image PDF). "
"OCR is not enabled for this benchmarker — only Text PDFs are supported. "
f"(total_chars={total}, median_chars_per_page={median:.0f}; "
f"need total>={settings.pdf_min_total_chars} and "
f"median>={settings.pdf_min_median_chars_per_page})"
)
def _line_font_meta(line: dict) -> tuple[str, float, bool]:
"""Return (text, max_font_size, any_bold) for a dict-line."""
spans = line.get("spans") or []
parts: list[str] = []
max_size = 0.0
any_bold = False
for span in spans:
text = (span.get("text") or "").strip()
if text:
parts.append(span.get("text") or "")
size = float(span.get("size") or 0)
if size > max_size:
max_size = size
# flags bit 4 (16) = bold in PyMuPDF
if int(span.get("flags") or 0) & 2**4:
any_bold = True
return ("".join(parts).strip(), max_size, any_bold)
def _extract_blocks(doc: fitz.Document) -> list[TextBlock]:
"""Extract reading-order text blocks with font metadata (tables flattened)."""
blocks: list[TextBlock] = []
for page_index, page in enumerate(doc):
page_dict = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)
for block in page_dict.get("blocks") or []:
if block.get("type") != 0: # 0 = text
continue
for line in block.get("lines") or []:
text, font_size, bold = _line_font_meta(line)
if not text:
continue
blocks.append(
TextBlock(
style_name="Normal",
text=text,
font_size=font_size,
bold=bold,
page=page_index + 1,
locked=False,
)
)
return blocks
def _normalize(s: str) -> str:
return " ".join(s.split()).casefold()
def _apply_outline_headings(doc: fitz.Document, blocks: list[TextBlock]) -> int:
"""Mark blocks that match PDF outline/bookmark titles. Returns match count."""
toc = doc.get_toc(simple=True)
if not toc:
return 0
matched = 0
used: set[int] = set()
for level, title, page in toc:
title_norm = _normalize(title)
if not title_norm:
continue
level = max(1, min(int(level), 6))
# Prefer same page; fall back to any unused block with matching title
candidates = [
(i, b)
for i, b in enumerate(blocks)
if i not in used and not b.locked
]
same_page = [(i, b) for i, b in candidates if b.page == page]
search_order = same_page + [(i, b) for i, b in candidates if b.page != page]
for i, block in search_order:
block_norm = _normalize(block.text)
if block_norm == title_norm or title_norm in block_norm or block_norm in title_norm:
block.style_name = f"Heading{level}"
block.locked = True
used.add(i)
matched += 1
break
return matched
def _apply_font_headings(blocks: list[TextBlock]) -> int:
"""Mark unlocked blocks as headings from font size clusters vs body size."""
sizes = [b.font_size for b in blocks if b.font_size > 0]
if len(sizes) < 3:
return 0
body_size = statistics.median(sizes)
if body_size <= 0:
return 0
# Distinct sizes clearly above body text
heading_sizes = sorted(
{round(s, 1) for s in sizes if s >= body_size * 1.2},
reverse=True,
)
if not heading_sizes:
# Bold + short lines at/near body size as weak H2
upgraded = 0
for block in blocks:
if block.locked or block.style_name != "Normal":
continue
if block.bold and len(block.text) <= 150:
block.style_name = "Heading2"
upgraded += 1
return upgraded
size_to_level: dict[float, int] = {}
for idx, size in enumerate(heading_sizes):
size_to_level[size] = 1 if idx == 0 else 2
upgraded = 0
for block in blocks:
if block.locked or block.style_name != "Normal":
continue
if len(block.text) > 150:
continue
rounded = round(block.font_size, 1)
level = size_to_level.get(rounded)
if level is None:
continue
block.style_name = f"Heading{level}"
upgraded += 1
return upgraded
def parse_pdf(file_path: str | Path) -> ParseResult:
"""Parse a Text PDF into DocumentTree + markdown.
Raises DocumentProcessingError for missing files, Scanned PDFs, or empty text.
"""
path = Path(file_path)
if not path.exists():
raise DocumentProcessingError(f"File not found: {path}")
if path.suffix.lower() != ".pdf":
raise DocumentProcessingError(f"Not a PDF: {path.suffix}")
try:
doc = fitz.open(path)
except Exception as exc:
raise DocumentProcessingError(f"Failed to open PDF: {exc}") from exc
try:
if doc.page_count < 1:
raise DocumentProcessingError("PDF has no pages")
_enforce_text_layer_gate(doc)
blocks = _extract_blocks(doc)
if not blocks:
raise DocumentProcessingError("PDF contains no extractable text blocks")
_apply_outline_headings(doc, blocks)
_apply_font_headings(blocks)
# Gap-fill remaining Normal blocks (outline titles stay locked).
# When outline/font found nothing, this is a full heuristic pass.
apply_heading_heuristics(blocks, only_when_no_headings=False)
root = build_document_tree(blocks)
tree = DocumentTree(root=root)
markdown = tree_to_markdown(root)
plain_text = "\n".join(b.text for b in blocks if b.text)
return ParseResult(
tree=tree,
markdown=markdown,
plain_text=plain_text,
paragraph_count=len(blocks),
)
finally:
doc.close()

View File

@@ -1,8 +1,8 @@
"""Document and strategy API routes.
Endpoints:
POST /documents Upload a .docx file
POST /documents/{id}/process Run chunking strategies (stub until Phase 2)
POST /documents Upload a .docx / .doc / .pdf file
POST /documents/{id}/process Run chunking strategies
DELETE /documents/{id} Remove document + vectors
GET /strategies List available chunking strategies
"""
@@ -10,7 +10,6 @@ Endpoints:
from fastapi import APIRouter, File, UploadFile
from src.core.exceptions import DocumentProcessingError
from src.core.models import PaginatedResponse, StrategyName
from src.documents.models import (
DeleteResponse,
DocumentResponse,
@@ -20,6 +19,7 @@ from src.documents.models import (
StrategiesResponse,
StrategyInfo,
)
from src.documents.parser import SUPPORTED_SUFFIXES
from src.documents import service
router = APIRouter()
@@ -39,11 +39,15 @@ async def list_documents(offset: int = 0, limit: int = 50):
@router.post("/documents", response_model=DocumentResponse, status_code=201)
async def upload_document(file: UploadFile = File(...)):
"""Upload a .docx file. Parses it, stores the document tree in SQLite."""
"""Upload a document. Parses it, stores the document tree in SQLite."""
if not file.filename:
raise DocumentProcessingError("No filename provided")
if not file.filename.lower().endswith((".docx", ".doc")):
raise DocumentProcessingError("Only .docx and .doc files are supported")
lower = file.filename.lower()
if not any(lower.endswith(s) for s in SUPPORTED_SUFFIXES):
raise DocumentProcessingError(
f"Only {', '.join(SUPPORTED_SUFFIXES)} files are supported"
)
content = await file.read()
if not content:
@@ -61,10 +65,7 @@ async def upload_document(file: UploadFile = File(...)):
@router.post("/documents/{doc_id}/process", response_model=ProcessResponse)
async def process_document(doc_id: str, request: ProcessRequest):
"""Run selected chunking strategies on an uploaded document.
Currently returns stub results until Phase 2 implements the strategies.
"""
"""Run selected chunking strategies on an uploaded document."""
return service.process_document(doc_id, request)

View File

@@ -16,7 +16,7 @@ from src.documents.models import (
ProcessResponse,
StrategyResult,
)
from src.documents.parser import parse_docx
from src.documents.parser import parse_document, SUPPORTED_SUFFIXES
from src.storage import sqlite as db
from src.storage import qdrant as qdr
@@ -49,17 +49,22 @@ STRATEGY_DESCRIPTIONS: dict[StrategyName, str] = {
# ── Upload ─────────────────────────────────────────────────────────
def upload_document(filename: str, file_bytes: bytes) -> dict[str, Any]:
"""Parse a .docx upload, store in SQLite, return document record."""
"""Parse an uploaded document, store in SQLite, return document record."""
import tempfile
from pathlib import Path
suffix = Path(filename).suffix or ".docx"
suffix = Path(filename).suffix.lower()
if suffix not in SUPPORTED_SUFFIXES:
raise DocumentProcessingError(
f"Only {', '.join(SUPPORTED_SUFFIXES)} files are supported"
)
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
tmp.write(file_bytes)
tmp_path = tmp.name
try:
result = parse_docx(tmp_path)
result = parse_document(tmp_path)
finally:
Path(tmp_path).unlink(missing_ok=True)

View File

@@ -276,6 +276,35 @@ async function api(path, opts = {}) {
return res.json();
}
// -- Format filter helpers (ADR-0017) -----------------------
function isPdfFilename(name) {
return (name || '').toLowerCase().endsWith('.pdf');
}
function isWordFilename(name) {
const n = (name || '').toLowerCase();
return n.endsWith('.doc') || n.endsWith('.docx');
}
function filterDocuments(docs, format) {
if (format === 'pdf') return (docs || []).filter(d => isPdfFilename(d.filename));
if (format === 'word') return (docs || []).filter(d => isWordFilename(d.filename));
return docs || [];
}
function filterExperiments(exps, format) {
if (format === 'pdf') return (exps || []).filter(e => isPdfFilename(e.document_filename || ''));
if (format === 'word') return (exps || []).filter(e => isWordFilename(e.document_filename || ''));
return exps || [];
}
function formatAccept(format) {
if (format === 'pdf') return '.pdf';
if (format === 'word') return '.doc,.docx';
return '.doc,.docx,.pdf';
}
function formatDropLabel(format) {
if (format === 'pdf') return '.pdf';
if (format === 'word') return '.doc or .docx';
return '.doc, .docx, or .pdf';
}
// -- Toast system -------------------------------------------
let _toastId = 0;
function ToastContainer({ toasts, removeToast }) {
@@ -329,10 +358,10 @@ function HomeTab() {
React.createElement('div', { className: 'card mt-4' },
React.createElement('div', { className: 'card-title' }, '⚡ Quick Start'),
React.createElement('ol', { style: { paddingLeft: '20px', lineHeight: '2' } },
React.createElement('li', null, 'Upload a document in the ', React.createElement('b', null, 'Documents'), ' tab'),
React.createElement('li', null, 'Upload a Word document in ', React.createElement('b', null, 'Documents'), ' or a Text PDF in ', React.createElement('b', null, 'PDF')),
React.createElement('li', null, 'Process it with chunking strategies'),
React.createElement('li', null, 'Ask a question in the ', React.createElement('b', null, 'Query'), ' tab'),
React.createElement('li', null, 'Run a full benchmark in the ', React.createElement('b', null, 'Benchmarks'), ' tab'),
React.createElement('li', null, 'Ask a question in ', React.createElement('b', null, 'Query'), ' (or inside PDF Workspace)'),
React.createElement('li', null, 'Run a full benchmark in ', React.createElement('b', null, 'Benchmarks'), ' / PDF Workspace'),
React.createElement('li', null, 'Check system health in the ', React.createElement('b', null, 'Admin'), ' tab')
)
)
@@ -340,12 +369,13 @@ function HomeTab() {
}
// -- Tab: Documents -----------------------------------------
function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
function DocumentsTab({ documents, setDocuments, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
const [uploading, setUploading] = useState(false);
const [processing, setProcessing] = useState({});
const [selectedStrategies, setSelectedStrategies] = useState({});
const [dragOver, setDragOver] = useState(false);
const fileRef = useRef(null);
const visibleDocs = filterDocuments(documents, formatFilter);
const fetchDocs = useCallback(() => {
api('/documents').then(d => setDocuments(d.items || [])).catch(() => {});
@@ -355,6 +385,15 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
const handleUpload = async (file) => {
if (!file) return;
const name = file.name || '';
if (formatFilter === 'pdf' && !isPdfFilename(name)) {
addToast('Only .pdf files are allowed here', 'error');
return;
}
if (formatFilter === 'word' && !isWordFilename(name)) {
addToast('Only .doc / .docx files are allowed here', 'error');
return;
}
setUploading(true);
try {
const form = new FormData();
@@ -425,7 +464,7 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
};
return React.createElement('div', null,
React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Documents'),
!hideTitle && React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Documents'),
// Upload zone
React.createElement('div', {
@@ -439,9 +478,9 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
React.createElement('div', { className: 'text' },
uploading
? 'Uploading...'
: React.createElement('span', null, 'Drag & drop a ', React.createElement('strong', null, '.doc or .docx'), ' file, or click to browse')),
: React.createElement('span', null, 'Drag & drop a ', React.createElement('strong', null, formatDropLabel(formatFilter)), ' file, or click to browse')),
React.createElement('input', {
ref: fileRef, type: 'file', accept: '.doc,.docx', style: { display: 'none' },
ref: fileRef, type: 'file', accept: formatAccept(formatFilter), style: { display: 'none' },
onChange: (e) => handleUpload(e.target.files[0])
})
),
@@ -449,13 +488,13 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
// Documents table
React.createElement('div', { className: 'card mt-4' },
React.createElement('div', { className: 'flex-between mb-2' },
React.createElement('div', { className: 'card-title mb-0' }, `Uploaded Documents (${documents.length})`),
React.createElement('div', { className: 'card-title mb-0' }, `Uploaded Documents (${visibleDocs.length})`),
React.createElement('button', { className: 'btn btn-secondary btn-sm', onClick: fetchDocs }, '↻ Refresh')
),
documents.length === 0
visibleDocs.length === 0
? React.createElement('div', { className: 'empty-state' },
React.createElement('div', { className: 'icon' }, '📭'),
React.createElement('div', null, 'No documents uploaded yet'))
React.createElement('div', null, formatFilter === 'pdf' ? 'No PDF documents uploaded yet' : 'No documents uploaded yet'))
: React.createElement('table', null,
React.createElement('thead', null,
React.createElement('tr', null,
@@ -468,7 +507,7 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
)
),
React.createElement('tbody', null,
documents.map(doc =>
visibleDocs.map(doc =>
React.createElement('tr', { key: doc.id },
React.createElement('td', null,
React.createElement('span', { className: 'truncate', style: { display: 'inline-block' } }, doc.filename)),
@@ -534,10 +573,11 @@ function DocumentsTab({ documents, setDocuments, strategies, addToast }) {
}
// -- Tab: Query ---------------------------------------------
function QueryTab({ documents, strategies, addToast }) {
function QueryTab({ documents, strategies, addToast, formatFilter = 'word', hideTitle = false }) {
const [form, setForm] = useState({ document_id: '', strategy: '', question: '', top_k: 5 });
const [result, setResult] = useState(null);
const [loading, setLoading] = useState(false);
const visibleDocs = filterDocuments(documents, formatFilter);
const handleQuery = async () => {
if (!form.document_id || !form.strategy || !form.question.trim()) {
@@ -557,7 +597,7 @@ function QueryTab({ documents, strategies, addToast }) {
};
return React.createElement('div', null,
React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Query'),
!hideTitle && React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Query'),
React.createElement('div', { className: 'card' },
React.createElement('div', { className: 'row' },
@@ -568,7 +608,7 @@ function QueryTab({ documents, strategies, addToast }) {
onChange: e => setForm({ ...form, document_id: e.target.value })
},
React.createElement('option', { value: '' }, 'Select document...'),
documents.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
visibleDocs.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
)
),
React.createElement('div', { className: 'col' },
@@ -868,13 +908,15 @@ function ComparisonView({ experiments, addToast, onBack }) {
}
// -- Tab: Benchmarks ----------------------------------------
function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
function BenchmarksTab({ documents, strategies, addToast, questionsFile, formatFilter = 'word', hideTitle = false }) {
const [experiments, setExperiments] = useState([]);
const [form, setForm] = useState({
document_id: '', strategies: strategies.map(s => s.name),
questions_file: questionsFile, top_k: 5, dry_run: false
});
const [running, setRunning] = useState(false);
const visibleDocs = filterDocuments(documents, formatFilter);
const visibleExps = filterExperiments(experiments, formatFilter);
// Comparison state
const [selectedExps, setSelectedExps] = useState(new Set());
@@ -914,10 +956,10 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
};
const selectAllExps = () => {
if (selectedExps.size === experiments.length) {
if (selectedExps.size === visibleExps.length) {
setSelectedExps(new Set());
} else {
setSelectedExps(new Set(experiments.map(e => e.id)));
setSelectedExps(new Set(visibleExps.map(e => e.id)));
}
};
@@ -975,7 +1017,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
}
setRunning(true);
try {
const result = await api('/benchmarks', { method: 'POST', body: form });
await api('/benchmarks', { method: 'POST', body: form });
addToast(`Benchmark ${form.dry_run ? 'estimate' : 'complete'}`, 'success');
if (!form.dry_run) fetchExperiments();
} catch (e) {
@@ -995,7 +1037,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
}
return React.createElement('div', null,
React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Benchmarks'),
!hideTitle && React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '16px', fontSize: '22px' } }, 'Benchmarks'),
// Run benchmark form
React.createElement('div', { className: 'card' },
@@ -1008,7 +1050,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
onChange: e => setForm({ ...form, document_id: e.target.value })
},
React.createElement('option', { value: '' }, 'Select document...'),
documents.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
visibleDocs.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
)
),
React.createElement('div', { className: 'col', style: { maxWidth: '120px' } },
@@ -1070,7 +1112,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
// Experiments list
React.createElement('div', { className: 'card mt-4' },
React.createElement('div', { className: 'flex-between mb-2' },
React.createElement('div', { className: 'card-title mb-0' }, `Experiments (${experiments.length})`),
React.createElement('div', { className: 'card-title mb-0' }, `Experiments (${visibleExps.length})`),
React.createElement('div', { style: { display: 'flex', gap: '6px', alignItems: 'center' } },
selectedExps.size >= 2 ? React.createElement('button', {
className: 'btn btn-primary btn-sm',
@@ -1080,17 +1122,17 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
React.createElement('button', { className: 'btn btn-secondary btn-sm', onClick: fetchExperiments }, '↻ Refresh')
)
),
experiments.length === 0
visibleExps.length === 0
? React.createElement('div', { className: 'empty-state' },
React.createElement('div', { className: 'icon' }, '📊'),
React.createElement('div', null, 'No experiments yet'))
React.createElement('div', null, formatFilter === 'pdf' ? 'No PDF experiments yet' : 'No experiments yet'))
: React.createElement('table', null,
React.createElement('thead', null,
React.createElement('tr', null,
React.createElement('th', { style: { width: '36px' } },
React.createElement('input', {
type: 'checkbox',
checked: experiments.length > 0 && selectedExps.size === experiments.length,
checked: visibleExps.length > 0 && selectedExps.size === visibleExps.length,
onChange: selectAllExps,
style: { cursor: 'pointer' }
})
@@ -1104,7 +1146,7 @@ function BenchmarksTab({ documents, strategies, addToast, questionsFile }) {
)
),
React.createElement('tbody', null,
experiments.map(exp => React.createElement(React.Fragment, { key: exp.id },
visibleExps.map(exp => React.createElement(React.Fragment, { key: exp.id },
React.createElement('tr', {
className: selectedExps.has(exp.id) ? 'cmp-highlight' : '',
style: { cursor: 'pointer' },
@@ -1228,6 +1270,165 @@ function CollapseSection({ title, icon, open, onToggle, badge, children }) {
);
}
// -- Shared: Chunk Preview Panel ----------------------------
function ChunkPreviewPanel({ documents, strategies, addToast, formatFilter = 'all' }) {
const visibleDocs = filterDocuments(documents, formatFilter);
const [chunkDocId, setChunkDocId] = useState('');
const [chunkStrategy, setChunkStrategy] = useState('');
const [chunkData, setChunkData] = useState(null);
const [chunkLoading, setChunkLoading] = useState(false);
const [expandedChunks, setExpandedChunks] = useState({});
const loadChunks = async () => {
if (!chunkDocId) { addToast('Select a document', 'error'); return; }
setChunkLoading(true);
setChunkData(null);
setExpandedChunks({});
try {
let url = `/admin/chunks/${chunkDocId}`;
if (chunkStrategy) url += `?strategy=${chunkStrategy}`;
const data = await api(url);
setChunkData(data);
} catch (e) { addToast(`Chunk load failed: ${e.message}`, 'error'); }
finally { setChunkLoading(false); }
};
return React.createElement(React.Fragment, null,
React.createElement('div', { className: 'row mb-4' },
React.createElement('div', { className: 'col' },
React.createElement('label', null, 'Document'),
React.createElement('select', {
value: chunkDocId,
onChange: e => { setChunkDocId(e.target.value); setChunkData(null); }
},
React.createElement('option', { value: '' }, 'Select document...'),
visibleDocs.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
)
),
React.createElement('div', { className: 'col' },
React.createElement('label', null, 'Strategy (optional)'),
React.createElement('select', {
value: chunkStrategy,
onChange: e => { setChunkStrategy(e.target.value); setChunkData(null); }
},
React.createElement('option', { value: '' }, 'All strategies'),
strategies.map(s => React.createElement('option', { key: s.name, value: s.name }, s.name))
)
),
React.createElement('div', { style: { flex: 'none' } },
React.createElement('button', {
className: 'btn btn-primary', onClick: loadChunks, disabled: chunkLoading || !chunkDocId,
}, chunkLoading ? React.createElement('span', { className: 'spinner' }) : 'Load Chunks')
)
),
chunkData && Object.entries(chunkData.strategies || {}).map(([stratName, stratData]) =>
React.createElement('div', { key: stratName, style: { marginBottom: '16px' } },
React.createElement('div', { className: 'flex-between mb-2' },
React.createElement('h3', { style: { fontSize: '14px', fontWeight: 600, color: 'var(--text-heading)' } },
`${stratName} `,
React.createElement('span', { className: 'badge badge-accent' }, `${stratData.count} chunks`)),
stratData.error ? React.createElement('span', { className: 'badge badge-danger' }, stratData.error) : null
),
stratData.count === 0
? React.createElement('div', { className: 'text-sm text-muted', style: { padding: '8px 0' } }, 'No chunks for this strategy')
: React.createElement('table', null,
React.createElement('thead', null,
React.createElement('tr', null,
React.createElement('th', { style: { width: '40px' } }, '#'),
React.createElement('th', null, 'Text'),
React.createElement('th', { style: { width: '60px' } }, 'Tokens'),
React.createElement('th', { style: { width: '60px' } }, 'Chars'),
)
),
React.createElement('tbody', null,
(stratData.chunks || []).map((chunk, i) => {
const key = `${stratName}-${chunk.chunk_index ?? i}`;
const isExpanded = expandedChunks[key];
return React.createElement(React.Fragment, { key },
React.createElement('tr', {
className: 'chunk-row',
onClick: () => setExpandedChunks(prev => ({ ...prev, [key]: !prev[key] })),
},
React.createElement('td', { className: 'text-muted text-sm' }, chunk.chunk_index ?? i + 1),
React.createElement('td', { className: 'text-sm' },
React.createElement('span', { style: { display: '-webkit-box', WebkitLineClamp: isExpanded ? 'unset' : 1, WebkitBoxOrient: 'vertical', overflow: isExpanded ? 'visible' : 'hidden' } },
chunk.text || '')),
React.createElement('td', { className: 'text-sm text-muted' }, chunk.token_count),
React.createElement('td', { className: 'text-sm text-muted' }, chunk.character_count),
),
isExpanded ? React.createElement('tr', null,
React.createElement('td', { colSpan: 4 },
React.createElement('div', { className: 'chunk-detail' },
React.createElement('div', null, chunk.text || '(empty)'),
React.createElement('div', { className: 'meta' },
`chunk_id: ${chunk.chunk_id} | parent: ${chunk.parent_id || 'none'}`)
)
)
) : null
);
})
)
)
)
)
);
}
// -- Tab: PDF Workspace -------------------------------------
function PdfWorkspaceTab({ documents, setDocuments, strategies, addToast, questionsFile }) {
const [openSections, setOpenSections] = useState({
upload: true, chunks: false, query: false, benchmarks: false,
});
const toggle = (key) => setOpenSections(prev => ({ ...prev, [key]: !prev[key] }));
const pdfCount = filterDocuments(documents, 'pdf').length;
return React.createElement('div', null,
React.createElement('h1', { style: { color: 'var(--text-heading)', marginBottom: '8px', fontSize: '22px' } }, 'PDF Workspace'),
React.createElement('p', { className: 'text-muted mb-4', style: { maxWidth: '640px' } },
'Text PDF only — upload, process, preview chunks, query, and benchmark. Scanned/image PDFs are rejected at upload.'),
React.createElement(CollapseSection, {
title: 'Upload & Process', icon: '📄', open: openSections.upload,
onToggle: () => toggle('upload'),
badge: pdfCount ? `${pdfCount} PDFs` : '0',
},
React.createElement(DocumentsTab, {
documents, setDocuments, strategies, addToast,
formatFilter: 'pdf', hideTitle: true,
})
),
React.createElement(CollapseSection, {
title: 'Chunk Preview', icon: '📦', open: openSections.chunks,
onToggle: () => toggle('chunks'),
},
React.createElement(ChunkPreviewPanel, {
documents, strategies, addToast, formatFilter: 'pdf',
})
),
React.createElement(CollapseSection, {
title: 'Query', icon: '⚡', open: openSections.query,
onToggle: () => toggle('query'),
},
React.createElement(QueryTab, {
documents, strategies, addToast,
formatFilter: 'pdf', hideTitle: true,
})
),
React.createElement(CollapseSection, {
title: 'Benchmarks', icon: '📊', open: openSections.benchmarks,
onToggle: () => toggle('benchmarks'),
},
React.createElement(BenchmarksTab, {
documents, strategies, addToast, questionsFile,
formatFilter: 'pdf', hideTitle: true,
})
)
);
}
// -- Tab: Admin ---------------------------------------------
function AdminTab({ addToast, documents, strategies, setActiveTab, setQuestionsFile }) {
// Collapse state for each section
@@ -1253,27 +1454,6 @@ function AdminTab({ addToast, documents, strategies, setActiveTab, setQuestionsF
// -- Qdrant state --
const [collectionsData, setCollectionsData] = useState([]);
// -- Chunk Preview state --
const [chunkDocId, setChunkDocId] = useState('');
const [chunkStrategy, setChunkStrategy] = useState('');
const [chunkData, setChunkData] = useState(null);
const [chunkLoading, setChunkLoading] = useState(false);
const [expandedChunks, setExpandedChunks] = useState({});
const loadChunks = async () => {
if (!chunkDocId) { addToast('Select a document', 'error'); return; }
setChunkLoading(true);
setChunkData(null);
setExpandedChunks({});
try {
let url = `/admin/chunks/${chunkDocId}`;
if (chunkStrategy) url += `?strategy=${chunkStrategy}`;
const data = await api(url);
setChunkData(data);
} catch (e) { addToast(`Chunk load failed: ${e.message}`, 'error'); }
finally { setChunkLoading(false); }
};
// -- Questions state --
const [qFiles, setQFiles] = useState([]);
const [qLoading, setQLoading] = useState(false);
@@ -1425,85 +1605,10 @@ function AdminTab({ addToast, documents, strategies, setActiveTab, setQuestionsF
React.createElement(CollapseSection, {
title: '📦 Chunk Preview', icon: '', open: openSections.chunks,
onToggle: () => toggle('chunks'),
badge: chunkData ? `${chunkData.strategies ? Object.values(chunkData.strategies).reduce((a, s) => a + (s.count || 0), 0) : 0} chunks` : null,
},
React.createElement('div', { className: 'row mb-4' },
React.createElement('div', { className: 'col' },
React.createElement('label', null, 'Document'),
React.createElement('select', {
value: chunkDocId,
onChange: e => { setChunkDocId(e.target.value); setChunkData(null); }
},
React.createElement('option', { value: '' }, 'Select document...'),
documents.map(d => React.createElement('option', { key: d.id, value: d.id }, d.filename))
)
),
React.createElement('div', { className: 'col' },
React.createElement('label', null, 'Strategy (optional)'),
React.createElement('select', {
value: chunkStrategy,
onChange: e => { setChunkStrategy(e.target.value); setChunkData(null); }
},
React.createElement('option', { value: '' }, 'All strategies'),
strategies.map(s => React.createElement('option', { key: s.name, value: s.name }, s.name))
)
),
React.createElement('div', { style: { flex: 'none' } },
React.createElement('button', {
className: 'btn btn-primary', onClick: loadChunks, disabled: chunkLoading || !chunkDocId,
}, chunkLoading ? React.createElement('span', { className: 'spinner' }) : 'Load Chunks')
)
),
chunkData && Object.entries(chunkData.strategies || {}).map(([stratName, stratData]) =>
React.createElement('div', { key: stratName, style: { marginBottom: '16px' } },
React.createElement('div', { className: 'flex-between mb-2' },
React.createElement('h3', { style: { fontSize: '14px', fontWeight: 600, color: 'var(--text-heading)' } },
`${stratName} `,
React.createElement('span', { className: 'badge badge-accent' }, `${stratData.count} chunks`)),
stratData.error ? React.createElement('span', { className: 'badge badge-danger' }, stratData.error) : null
),
stratData.count === 0
? React.createElement('div', { className: 'text-sm text-muted', style: { padding: '8px 0' } }, 'No chunks for this strategy')
: React.createElement('table', null,
React.createElement('thead', null,
React.createElement('tr', null,
React.createElement('th', { style: { width: '40px' } }, '#'),
React.createElement('th', null, 'Text'),
React.createElement('th', { style: { width: '60px' } }, 'Tokens'),
React.createElement('th', { style: { width: '60px' } }, 'Chars'),
)
),
React.createElement('tbody', null,
(stratData.chunks || []).map((chunk, i) => {
const key = `${stratName}-${chunk.chunk_index ?? i}`;
const isExpanded = expandedChunks[key];
return React.createElement(React.Fragment, { key },
React.createElement('tr', {
className: 'chunk-row',
onClick: () => setExpandedChunks(prev => ({ ...prev, [key]: !prev[key] })),
},
React.createElement('td', { className: 'text-muted text-sm' }, chunk.chunk_index ?? i + 1),
React.createElement('td', { className: 'text-sm' },
React.createElement('span', { style: { display: '-webkit-box', WebkitLineClamp: isExpanded ? 'unset' : 1, WebkitBoxOrient: 'vertical', overflow: isExpanded ? 'visible' : 'hidden' } },
chunk.text || '')),
React.createElement('td', { className: 'text-sm text-muted' }, chunk.token_count),
React.createElement('td', { className: 'text-sm text-muted' }, chunk.character_count),
),
isExpanded ? React.createElement('tr', null,
React.createElement('td', { colSpan: 4 },
React.createElement('div', { className: 'chunk-detail' },
React.createElement('div', null, chunk.text || '(empty)'),
React.createElement('div', { className: 'meta' },
`chunk_id: ${chunk.chunk_id} | parent: ${chunk.parent_id || 'none'}`)
)
)
) : null
);
React.createElement(ChunkPreviewPanel, {
documents, strategies, addToast, formatFilter: 'all',
})
)
)
)
)
),
// -- 4. Questions Management --
@@ -1651,6 +1756,7 @@ function App() {
useEffect(() => {
api('/strategies').then(d => setStrategies(d.strategies || [])).catch(() => {});
api('/documents').then(d => setDocuments(d.items || [])).catch(() => {});
}, []);
const addToast = useCallback((message, type = 'success') => {
@@ -1666,6 +1772,7 @@ function App() {
const tabs = [
{ id: 'home', label: 'Home' },
{ id: 'documents', label: 'Documents' },
{ id: 'pdf', label: 'PDF' },
{ id: 'query', label: 'Query' },
{ id: 'benchmarks', label: 'Benchmarks' },
{ id: 'admin', label: 'Admin' },
@@ -1675,11 +1782,13 @@ function App() {
switch (activeTab) {
case 'home': return React.createElement(HomeTab);
case 'documents': return React.createElement(DocumentsTab, {
documents, setDocuments, strategies, addToast });
documents, setDocuments, strategies, addToast, formatFilter: 'word' });
case 'pdf': return React.createElement(PdfWorkspaceTab, {
documents, setDocuments, strategies, addToast, questionsFile });
case 'query': return React.createElement(QueryTab, {
documents, strategies, addToast });
documents, strategies, addToast, formatFilter: 'word' });
case 'benchmarks': return React.createElement(BenchmarksTab, {
documents, strategies, addToast, questionsFile });
documents, strategies, addToast, questionsFile, formatFilter: 'word' });
case 'admin': return React.createElement(AdminTab, {
addToast, documents, strategies, setActiveTab, setQuestionsFile });
default: return null;