chunking_strategies_evaluation/docs/phases.md

Phases & Steps

Implementation plan for the RAG Chunking Strategy Benchmarking Framework. Each step maps back to tasks in tasks.md.

---

Phase 1: Document Parsing + Storage

Foundation layer. Everything else depends on this.

Step	What	Files	Status
1	SQLite Storage Layer	src/storage/sqlite.py	DONE
2	Qdrant Storage Layer	src/storage/qdrant.py	DONE
3	Document Parser	src/documents/parser.py	DONE
4	Document Routes + Service	src/documents/routes.py, src/documents/service.py, src/documents/models.py	DONE

Endpoints delivered:

• POST /documents — upload .docx
• POST /documents/{id}/process — run strategies (stubbed)
• DELETE /documents/{id} — remove document + vectors
• GET /strategies — list all 5 strategies

---

Phase 2: Chunking Strategies

The core evaluation targets. Each strategy is an independent module.

Step	What	Files	Status
5	Base Strategy Interface	src/chunking/base.py	TODO
6	Five Strategy Implementations	src/chunking/strategies/recursive.py, markdown_structure.py, semantic.py, parent_child.py, contextual_structure.py	TODO
7	Embedding Service	src/chunking/embedding.py	TODO
8	Chunking Orchestration	src/chunking/service.py	TODO

ADR alignment:

• ADR 0001: Unified Chunk model (nullable fields)
• ADR 0002: Hard-fail on contextual enrichment failure
• ADR 0003: Per-strategy failure isolation
• ADR 0005: Hardcoded article-level parents
• ADR 0011: Embed full enriched text for contextual
• ADR 0012: Sentence-level semantic with min chunk size
• ADR 0013: Leaf-level markdown chunks only
• ADR 0015: Direct API usage, no LangChain/LlamaIndex

Delivers: The POST /documents/{id}/process endpoint goes from stub to real — chunking, embedding, and Qdrant storage for all 5 strategies.

---

Phase 3: Query Pipeline

Retrieval + generation. The bridge between storage and evaluation.

Step	What	Files	Status
9	Query Service	src/benchmarking/query_service.py	TODO
10	Query Routes	src/benchmarking/routes.py	TODO

Endpoints delivered:

• POST /queries — ask a question against a strategy
• GET /queries/{id} — retrieve past query

Pipeline: question → embed → vector search (Qdrant) → top_k chunks → gpt-4o-mini → answer + metadata.

---

Phase 4: Benchmarking + Evaluation

The reason the project exists. Compare strategies head-to-head.

Step	What	Files	Status
11	LLM-as-Judge Evaluation	src/benchmarking/evaluation.py	TODO
12	Benchmark Service	src/benchmarking/benchmark_service.py	TODO
13	Benchmark Routes	extends src/benchmarking/routes.py	TODO
14	HTML Report Template	src/benchmarking/templates/report.html	TODO

Endpoints delivered:

• POST /benchmarks — run benchmark (single or dataset)
• POST /benchmarks?dry_run=true — cost estimate only
• GET /benchmarks/{id} — retrieve experiment results
• GET /benchmarks/{id}/report — download HTML report

Evaluation metrics (ADR 0004):

• Context Relevance (1–10)
• Answer Similarity (1–10)
• Faithfulness (1–10)
• Hallucination (bool)

Modes:

• Single question
• Dataset (questions.json)
• Dry run (cost estimation)

---

Phase 5: Wiring + Verification

Polish, integration, and proof that it all works end-to-end.

Step	What	Files	Status
15	Wire All Routers + Logging	src/main.py	TODO
16	End-to-End Smoke Test	test suite	TODO

Verification checklist:

• Upload insurance .docx
• Process with all 5 strategies
• Ask 3 test questions
• Run mini benchmark (3 questions × 5 strategies)
• Verify HTML report renders
• Confirm per-strategy failure isolation (simulate one strategy failure)
• Confirm dry-run cost estimation matches actual cost within 10%