chatbot_v3/docs/plans/001-ingestion-vertical-slice.md

001. Ingestion vertical-slice implementation plan

Purpose

This plan turns the accepted architectural direction in the ADRs into the first working product slice: a tenant-scoped DOCX/XLSX/CSV upload is stored in MinIO, represented by durable Postgres records, parsed/chunked/embedded inline in the request (ADR-0017), and indexed as Qdrant points before the response returns.

This is an implementation plan, not an Architecture Decision Record. ADRs explain why major technologies and boundaries were chosen; this document defines the order, scope, and verification criteria for implementing them.

Architecture baseline

The first vertical slice uses these responsibilities:

System	Responsibility
FastAPI	HTTP boundary, validation, authentication, tenant derivation, and job creation.
Postgres	Tenant/auth data, source-file metadata, ingestion attempt records/progress, and audit.
MinIO	Private source-file bytes and retained derived ingestion blobs.
Ingestion service	Parsing, chunking, embedding, ingestion-generated Chunk/Point CRUD, and job status updates — inline in the request.
Qdrant	Tenant-filtered generated chunks and their vectors/payloads.

The controlling ADRs are:

• ADR-0008: FastAPI REST boundary and job-shaped file ingestion contract.
• ADR-0009: Postgres source files, jobs, audit, and migration conventions.
• ADR-0012: resource lifetime, dependency injection, and explicit transaction ownership.
• ADR-0013: MinIO object storage boundary.
• ADR-0017: inline ingestion, batched/bounded-concurrent embedding, anyio.to_thread.run_sync for blocking work, and request bounds. It supersedes ADR-0014, which remains the design to adopt when a broker becomes necessary.

The cited ADRs are currently proposed. Treat them as the implementation baseline only after the project owner accepts them; code should not silently diverge from them.

First release scope

In scope

• POST /v1/files for authenticated tenant-scoped DOCX, XLSX, and CSV upload. .doc is rejected with 415 pending an out-of-process conversion service (ADR-0018). An earlier revision of this plan scoped the slice to CSV only and placed DOCX out of scope; the real corpus is DOCX and XLSX, so ADR-0018 corrects that.
• File validation, size limits, content hashing, and streaming upload to MinIO.
• Alembic-managed Postgres schema for the minimal tenant/auth, source file, ingestion job, and job event records needed by this slice.
• Inline ingestion in POST /v1/files, with batched/bounded-concurrent embedding, thread-offloaded parsing, and enforced size/timeout/capacity bounds.
• DOCX/XLSX/CSV parsing into structural units and deterministic chunk creation (ADR-0018, implemented in src/application/ingestion/).
• Tenant-filtered Qdrant point upserts using deterministic point identifiers.
• Job status/progress persistence and GET /v1/files/{file_id} status lookup.
• Structured correlation logging at HTTP and ingestion-stage boundaries.
• Automated tests for the critical state transitions, redelivery, and tenant boundaries.

Explicitly out of scope

• Legacy .doc ingestion — rejected with 415 until an out-of-process conversion service exists (ADR-0018). DOCX and XLSX are in scope; they were listed here before ADR-0018 corrected the scope line.
• qa_pair structural detection and embedded-image captioning (ADR-0004), deferred by ADR-0018.
• The conversational LangGraph API and SSE streaming.
• Final reranker selection, GPU deployment, or unresolved model licensing from ADR-0005.
• Direct /v1/points CRUD endpoints beyond the reusable service layer required by ingestion.
• Full tenant erasure and hard-deletion workflow.
• A public download API or presigned object URLs.
• A message broker, transactional outbox, job queue, and separate worker/publisher processes (ADR-0014, deferred by ADR-0017).
• late_interaction (jina-colbert-v2) document vectors at ingest — populating them is ADR-0017's primary trigger to move ingestion back off the request.
• Exactly-once end-to-end processing. Job execution must instead be safe when a job runs more than once.

Required invariants

The implementation must preserve these rules from the ADRs:

1. tenant_id is derived from trusted authentication context; it is never accepted from the upload body, query parameters, MinIO metadata, or a dispatch payload as authority.
2. MinIO stores bytes; Postgres stores metadata, lifecycle state, job progress, audit records, and the queue.
3. Dispatch payloads contain stable IDs and correlation metadata only. They never contain file bytes, extracted text, chunks, embeddings, secrets, raw prompts, or raw model output.
4. Ingestion works from the persisted job and source-file records, not from request-supplied values.
5. Parsing, chunking, embedding, and generated Qdrant Chunk/Point CRUD live in the ingestion application service, not in the route handler. Blocking work runs through anyio.to_thread.run_sync, never directly on the event loop.
6. Qdrant reads and mutations are tenant-filtered. Ingestion-generated point IDs are deterministic so retrying a job does not create duplicate logical chunks.
7. No Postgres session or transaction is held open across parse/embed/upsert. The job row is committed running before the work and updated to a terminal status after it, in a second short transaction.
8. Application clients are built at FastAPI startup and closed at shutdown. No mutable clients are opened as import-time globals.
9. Embedding is batched per provider limits and concurrency-bounded by a semaphore; blocking work (parse, chunk, BM25, minio) runs through anyio.to_thread.run_sync with an explicit CapacityLimiter.

Required decisions before implementing the affected phase

The first implementation should use these defaults unless a later ADR changes them:

Source-file idempotency and replacement

• Use (tenant_id, domain, content_sha256) to recognize identical uploads.
• An identical active upload should return the existing source-file/job reference rather than create a duplicate ingestion.
• A changed upload creates a new ingestion job. A failed re-ingestion never removes a working index: the soft-delete sweep for a shortened file runs only after every upsert has succeeded. Because ADR-0001's point ids are deterministic, upserts overwrite in place, so an interrupted attempt can leave a prefix updated — it cannot empty or partially delete the index, and a retry converges. See ADR-0017, "Re-running an ingestion stays safe".
• Preserve the original filename in Postgres metadata. MinIO object keys remain internal ID-based paths.

This policy should be made explicit in ADR-0009 before implementing re-ingestion rather than becoming an accidental repository behavior.

File deletion

For the first release, file deletion should be soft and job-shaped:

1. mark the source file as deletion requested/soft deleted in Postgres;
2. soft-delete the related Qdrant points inline, recording the attempt;
3. retain the MinIO object until an explicit retention or hard-erasure workflow.

Hard deletion requires a later retention/erasure implementation covering MinIO, Qdrant, and the relevant Postgres data.

Ingestion bounds and operations

Before deploying an environment, define and document:

• INGESTION_MAX_CONCURRENCY and the thread-pool capacity limiter, and their relation to the process's CPU/memory budget;
• INGESTION_TIMEOUT_SECONDS, and the proxy/load-balancer/client read timeouts that must exceed it;
• INGESTION_EMBED_BATCH_SIZE and INGESTION_EMBED_CONCURRENCY, sized to the provider's rate limits and the self-hosted embedder's capacity;
• alert thresholds for p95 ingestion duration, 503/504 rates, and jobs left in running past the timeout;
• how failed ingestions are inspected and retried.

These are deployment/runbook settings, not new ADRs unless they change the reliability guarantee or system boundary.

Build order

Phase 1: Foundation and local dependencies

1. Add typed configuration in src/config.py for Postgres, MinIO, Qdrant, ingestion bounds, application limits, and logging.
2. Populate .env.example with non-secret local-development configuration.
3. Add application Docker Compose services for Postgres, MinIO, and Qdrant. Keep application MinIO buckets/credentials separate from Langfuse infrastructure.
4. Add direct Python dependencies and lock them with uv: SQLAlchemy async/Postgres driver, MinIO/S3 client, Qdrant client, and structured logging dependencies chosen by ADR-0011.
5. Add the pytest foundation from ADR-0016: async test configuration, boundary markers, and support for dependency-injected fakes. Add a lifespan smoke test before creating external clients.
6. Create FastAPI lifespan setup and typed dependency helpers without creating schema at startup.

Exit criteria: local infrastructure starts; readiness checks can report each required dependency; clients are opened/closed by process owners; fast unit tests run without Docker or live providers.

Phase 2: Database, migrations, and domain contracts

1. Define SQLAlchemy models and Alembic migrations for the minimum required tables: tenants, api_keys, source_files, ingestion_jobs, and ingestion_job_events, per ADR-0009.
2. Define Pydantic request/response schemas, including the terminal upload response (file_id, ingestion_job_id, status, chunks_indexed).
3. Implement explicit repositories/services with a request/job-lifetime AsyncSession; routes/services own commit/rollback boundaries as specified by ADR-0012.
4. Write the empty-database migration test before each schema revision. Create Testcontainers-based Postgres fixtures for tenants, hashed API keys, database sessions, and migrations. Do not use create_all() in test fixtures.

Exit criteria: migrations create the schema from an empty database; application startup performs no DDL; schema and repository tests verify tenant-scoped reads/writes and valid job transitions.

Phase 3: MinIO upload and durable job creation

1. Implement API-key authentication and AuthContext tenant derivation.
2. Implement POST /v1/files for DOCX, XLSX, and CSV, including streaming-size controls, file-type validation, SHA-256 calculation, and a private MinIO upload using an internal object key.
3. In one short Postgres transaction, persist source_files and create ingestion_jobs(status='running'), then commit and release the connection before any parse/embed work.
4. Return 201 Created with file_id, ingestion_job_id, terminal status, and chunks_indexed once ingestion completes.
5. Implement GET /v1/files/{file_id} with tenant filtering and a public status response that does not expose raw storage credentials or internal artifacts.
6. Add cleanup/compensation handling for a MinIO upload that succeeds while the database transaction fails.
7. Add unit/API tests for trusted tenant derivation, upload validation, idempotency, the terminal 201 Created response, and tenant-scoped status. Add MinIO adapter integration tests for server-derived private object paths and compensation behavior.

Exit criteria: an authenticated upload creates a private object and a running job row committed before any ingestion work; a tenant cannot retrieve another tenant's file status.

Phase 4: Bounded execution primitives

1. Add the async embedding ports and adapters in src/infrastructure/embedding/, with per-provider batching and an asyncio.Semaphore bounding in-flight batches.
2. Route blocking work (parse, chunk, BM25, minio) through anyio.to_thread.run_sync with an explicit CapacityLimiter created at startup, so ingestion cannot exhaust Starlette's thread pool.
3. Enforce the request bounds: INGESTION_MAX_CONCURRENCY (503 + Retry-After when exceeded), INGESTION_TIMEOUT_SECONDS around the whole work phase (504), and the size/chunk-count ceiling (413) checked before work starts.
4. Guarantee a bounded failure always writes a terminal job status — a timeout must never leave a job stuck in running.
5. Add unit tests for batching/concurrency limits, timeout-to-terminal-status, and capacity rejection, using scripted embedder fakes.

Exit criteria: embedding a few hundred chunks issues batched, concurrent requests rather than serial ones; exceeding any bound produces the right status code and a terminal job row.

Phase 5: Ingestion execution and Qdrant Chunk/Point CRUD

Carried forward from Phase 4 — the chunks collection must create the sparse vector with modifier="idf". The BM25 adapter computes only term-frequency saturation client-side; IDF comes from Qdrant's collection-wide statistics. Omit the modifier and there is no error and no warning — sparse scoring silently loses its IDF term and lexical retrieval degrades. See ADR-0005, "Benchmark outcome".

Collection creation must also use the pinned dimensions from ADR-0001: dense_nomic 768, dense_openai 3072.

1. Implement the ingestion service called by the route, using the application-lifetime database, MinIO, Qdrant, model, and logging clients.
2. Validate the persisted records before fetching the MinIO object.
3. Append progress events, parse the document, create deterministic chunks, embed them, and upsert tenant-scoped Qdrant points — without holding a Postgres session open across the work.
4. In a second short transaction, mark the job succeeded with counters or failed with a safe error summary, then return the terminal response.
5. Make a retried upload safe: no duplicate logical chunks, no incorrect counters, and no transition from a terminal state back to running.
6. Add unit tests for deterministic chunks, point IDs, and terminal job transitions. Add Testcontainers Qdrant and Postgres integration tests for tenant-filtered upserts, terminal state persistence, retrying an upload, and parser/Qdrant failure handling.

The chunks collection itself is provisioned by a deployment step — uv run python -m src.cli.qdrant_bootstrap — not by FastAPI startup, for the same reason ADR-0009 keeps Alembic out of startup and ADR-0012 makes LangGraph's .setup() a deployment step. See ADR-0001, "Collection provisioning".

Exit criteria: a successful upload returns 201 with a terminal status, and its points are retrievable only under the owning tenant's Qdrant filter. A forced failure mid-ingestion produces a failed job and the right HTTP status, and retrying the upload produces a correct final state without duplicate chunks.

Phase 6: Operations, integration tests, and documentation

1. Add an operator runbook covering local startup, migrations, MinIO bucket setup, the run command, ingestion-bound tuning, the proxy/client timeout requirement, and how to retry a failed ingestion. — docs/runbook.md.
2. Add a serialized Compose-based operational smoke test covering upload through indexed points against the running web process. Testcontainers remains the standard pytest mechanism for individual adapter integration tests. — scripts/smoke.sh driving tests/e2e/test_compose_smoke.py, which skips itself unless SMOKE_BASE_URL is set so uv run pytest never invokes Compose.
3. Add end-to-end tests for duplicate upload, retrying a failed upload, tenant isolation, capacity/timeout rejection, and failed parser/Qdrant behavior. — tests/e2e/test_ingestion_slice.py, on Testcontainers, in the default suite.
4. Add health/readiness checks that distinguish process health from dependency readiness. — /healthz and /readyz; /readyz additionally requires the chunks collection to exist, since a reachable but unbootstrapped Qdrant would 502 on the first upload.
5. Update the README with local-start instructions and links to ADRs, this plan, and the operations runbook.

Provisioning a tenant and its first API key turned out to be a prerequisite for 1 and 2 rather than a separate milestone: nothing over HTTP can create the first tenant, so src/cli/provision_tenant.py was added alongside the other two deployment-step commands.

Exit criteria: a new developer can start the stack, apply migrations, upload a a document, observe the job through completion, and understand how to investigate or retry a failure.

Definition of done for the vertical slice

The first slice is done when the following path works in local Compose and is covered by automated tests:

POST /v1/files (authenticated DOCX/XLSX/CSV upload)
  -> raw bytes stored privately in MinIO
  -> source file and running job committed in Postgres, connection released
  -> parse/chunk on threads, embed in bounded concurrent batches
  -> deterministic Qdrant points upserted
  -> Postgres records progress and terminal job status
  -> 201 Created returns the terminal result in the same request
  -> GET /v1/files/{file_id} reports that status within the owning tenant only

The next implementation work after this slice is direct Point CRUD, retrieval, and then the LangGraph conversational flow. Do not couple those later milestones to the initial ingestion path unless they are needed to preserve one of the invariants above.