azk/chatbot_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ingestion): add DOCX/CSV/XLSX parsing and fixed-size chunking (ADR-0018)

Browse Source 
Adds src/application/ingestion/ -- Persian normalization, DOCX body
walk with structural data/layout table classification, CSV/XLSX row
rendering, and fixed-size token chunking (cl100k_base, 400/60/512) --
as pure functions per ADR-0015, tested against real production
documents (asia_data_sample, kept out of the repo). ADR-0018 records
where this diverges from ADR-0004 (fixed-size default, no invented
headings/tree, structural table classification, header-provable
labeling only). Plan 001's scope line is corrected from CSV-only to
DOCX/XLSX/CSV, and CLAUDE.md's stale project-status paragraph is
updated to match current implementation state.
This commit is contained in:
Ali Zarin Kolah
2026-08-18 10:22:17 +03:30
parent 80ed5b1577
commit 5cdfb70085
26 changed files with 2438 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
CLAUDE.md
View File

@@ -4,12 +4,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project status
This repo is currently ADR-driven and mostly pre-implementation: `src/` contains
only an empty `main.py`/`config.py` scaffold and empty `api/routers`,
`api/dependencies`, `db`, and `schemas` directories. Architecture decisions live
in `docs/adr/` (17 ADRs plus the 0000 template; 0001–0004 are `Accepted`,
0014 is `Superseded by 0017`, and the rest — 0005–0013 and 0015–0017 — are
`Proposed`). Implementation plans live in `docs/plans/`:
This repo is ADR-driven and early in implementation. Working today: the FastAPI
app factory and lifespan wiring (`src/bootstrap/`), `/healthz` and `/readyz`,
structlog config, Postgres/MinIO/Qdrant clients (`src/infrastructure/`), five
SQLAlchemy models with one Alembic migration, and document parsing plus
fixed-size chunking (`src/application/ingestion/`). Not built yet: API-key auth,
`POST /v1/files` (so `/v1` currently exposes no routes), repositories, embedding
adapters, Qdrant collection bootstrap, and `src/agent/`. That maps to plan 001
Phase 1 done, Phase 2 partly done, and the parsing half of Phase 5.
Architecture decisions live in `docs/adr/` (18 ADRs plus the 0000 template;
0001–0004 are `Accepted` — 0004 amended by 0018; 0014 is `Superseded by 0017`;
the rest — 0005–0013 and 0015–0018 — are `Proposed`). Implementation plans live
in `docs/plans/`:
`001-ingestion-vertical-slice.md` and
`002-point-crud-and-keyword-search.md`. **Read the relevant
ADR(s) before implementing anything** — the ADRs are the source of truth for