chatbot_v3/docs/runbook.md

Operator runbook

How to start this service, configure its ingestion bounds, and investigate or retry a failed upload. Architecture rationale lives in docs/adr/; the implementation milestone is plan 001. This document covers operating what those describe.

The service is a single process with no background work. POST /v1/files parses, chunks, embeds, and indexes inline and returns a terminal result (ADR-0017). There is no queue, no worker, and no automatic retry — the caller owns the retry decision, which makes the request's duration a deployment constraint. That fact drives most of this document.

1. Prerequisites and local startup

Docker, and uv with Python 3.13.

cp .env.example .env      # non-secret local defaults; .env is gitignored
uv sync
docker compose up -d --wait

docker-compose.yml runs Postgres (127.0.0.1:5433), MinIO (127.0.0.1:9100, console 9101), and Qdrant (127.0.0.1:6343). It is the local development stack and says so in its header — it is not a production deployment.

2. Deployment steps

Two schema steps run before the application, never at startup: FastAPI performs no DDL, for Postgres (ADR-0009) or for Qdrant (ADR-0001, "Collection provisioning"). Both commands and their reasoning are in the README's Provisioning the datastores section:

uv run alembic upgrade head                   # Postgres schema
uv run python -m src.cli.qdrant_bootstrap     # the `chunks` collection

Both are idempotent. qdrant_bootstrap verifies an existing collection against the pinned schema and exits non-zero on a mismatch rather than leaving a silently degraded sparse index in place — the sparse vector's modifier="idf" and the pinned dense dimensions (768 / 3072) fail silently if wrong, which is why they are checked rather than assumed.

Run both again after every deploy that ships a migration or a collection-schema change.

3. MinIO bucket

Under Compose the bucket already exists: the app-minio service's entrypoint runs mkdir -p /data/${MINIO_BUCKET:-chatbot-source-files} before starting the server, so first boot creates it. Nothing else needs to be done locally.

Outside Compose, create the bucket named by MINIO_BUCKET before the first upload — the application never creates it. It must stay private; ADR-0013 keeps source bytes non-public and this slice ships no download API or presigned URLs.

4. Provisioning a tenant, an API key, and its domains

Nothing over HTTP can bootstrap a tenant: every /v1 route needs an API key, and a key cannot exist before its tenant. So the first key is issued by an operator command:

uv run python -m src.cli.provision_tenant \
  --slug acme --domain fire --domain life --scopes files:write,domains:read

It prints api_key=sk_... once. Postgres stores only its SHA-256 hash (ADR-0009), so a lost key is reissued by re-running the command, never recovered. Structured logs carry only the non-secret key_prefix — a plaintext key must never reach a log sink (ADR-0011).

Re-running with the same --slug reuses the tenant and any domains it already has, and issues an additional key. Both keys stay valid; this adds a key, it does not rotate one.

Scopes are the security boundary between uploading and managing the allowlist. Give an upload client files:write only. domains:write lets its holder create new domains, which is exactly what the allowlist exists to prevent an upload key from doing.

Domains after the first one

POST /v1/files rejects an unregistered or disabled domain with 400 (unknown_domain) before anything is written. Ongoing domain management is the /v1/domains API, under domains:read / domains:write:

curl -X POST http://localhost:8000/v1/domains \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"domain": "fire", "display_name": "Fire insurance"}'

The domain key itself is immutable — it is denormalized into every Qdrant point payload and into source_files, so renaming it is a migration, not an edit (ADR-0009). Disabling a domain blocks new uploads; it does not delete existing points.

5. Running the service

uv run fastapi dev src/main.py                                   # local, reload
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000           # deployed shape

Run more than one worker/replica only after reading §6: ingestion bounds are per process, so INGESTION_MAX_CONCURRENCY multiplies by the number of processes.

6. Ingestion bounds and tuning

Every bound is enforced server-side and maps to a status code. All are in .env.example. Ingestion is CPU- and network-bound in the request, so these are the numbers that decide whether the service degrades gracefully or falls over.

Setting	Bounds	On breach	Size it against
INGESTION_MAX_CONCURRENCY	Ingestions in flight per process	503 + Retry-After	Memory per in-flight upload (whole file plus its chunks and vectors are resident) and the embedder's capacity. Rejecting is deliberate: ADR-0017 refuses rather than queues.
INGESTION_THREAD_POOL_SIZE	Threads for blocking work (parse, chunk, hash, BM25, the sync minio SDK)	— (waits)	CPU cores. It exists to stop ingestion exhausting Starlette's own thread pool, so keep it below the total thread budget.
INGESTION_TIMEOUT_SECONDS	The whole work phase	504, job marked failed	The slowest legitimate document, plus headroom. See §7 — this must stay under every read timeout in front of it.
INGESTION_MAX_UPLOAD_SIZE_MB	Bytes accepted	413	Memory: the upload is read fully into the process before any work starts.
INGESTION_MAX_CHUNKS_PER_FILE	Chunks per file, checked before embedding	413	Embedder cost/time per chunk × INGESTION_TIMEOUT_SECONDS. This is the real defence against one pathological file eating a slot.
INGESTION_EMBED_BATCH_SIZE	Texts per embedder request	502 on embedder failure	The provider's per-request limits. Batch before parallelizing.
INGESTION_EMBED_CONCURRENCY	Concurrent embed batches	502	Provider rate limits and the self-hosted embedder's throughput. Never unbounded.
QDRANT_UPSERT_BATCH_SIZE / _CONCURRENCY	Points per upsert and concurrent upserts	502 (index_error)	Qdrant's ingest capacity; the batch size stays in ADR-0001's 64–256 band.

Two settings that look like tuning knobs but are not:

• EMBEDDING_NOMIC_KEEP_ALIVE holds the self-hosted model resident. A cold load of nomic-embed-text-v2-moe takes over 150 s — longer than any sane INGESTION_TIMEOUT_SECONDS — so an idle period followed by an upload would otherwise 504. The lifespan also warms both dense embedders at startup for the same reason.
• The BM25 analyzer and weights (EMBEDDING_SPARSE_*) are a measured artifact ported from the emet evaluation lab, verified token-for-token against it (ADR-0005). Re-benchmark; do not tune them in place.

7. The proxy and client read-timeout requirement

Every read timeout in front of this service must exceed INGESTION_TIMEOUT_SECONDS. That includes the reverse proxy / ingress, any load balancer, and the calling backend's own HTTP client.

If a proxy times out first, the client gets that proxy's error, the upload keeps running in the process, and the caller learns nothing about the outcome from the response. The job row still reaches a terminal status, so GET /v1/files/{file_id} remains the way to find out what happened — but the response contract is broken for that request. ADR-0017 names this the main cost of inline ingestion.

A workable local ordering: client read timeout > proxy read timeout > INGESTION_TIMEOUT_SECONDS.

8. Health and readiness

Endpoint	Question it answers	Use for
GET /healthz	Is the process alive?	Liveness probes / restart policy. Never depends on Postgres, MinIO, or Qdrant.
GET /readyz	Can it actually serve?	Load-balancer admission and post-deploy gating. 200 with each dependency true, 503 if any is false.

/readyz checks Postgres, MinIO, and Qdrant reachability and that the chunks collection exists. A reachable-but-unbootstrapped Qdrant reports {"qdrant": false} on purpose: uploads to it would fail with 502, so it is not ready, and this is how a skipped qdrant_bootstrap surfaces at deploy time instead of on a user's first upload.

9. Investigating a failure

Logs are structured (structlog, JSON in production) with stable event names — grep the event name, not prose (ADR-0011). Set LOG_FILE_PATH for a local JSON file sink alongside the console renderer; leave it unset in production, where stdout collection is preferred.

Correlate by request_id. Every request has one, echoed in the X-Request-Id response header and included in every error envelope, and bound into every log line emitted while handling that request. A client reporting a failed upload should quote it. tenant_id, file_id, and ingestion_job_id are the other join keys.

Events worth knowing:

Event	Level	Means
ingestion.job.started	info	Txn A committed; work phase beginning. Carries tenant_id, ingestion_job_id, file_id, domain, source_type.
ingestion.job.completed	info	Terminal success, with chunks_parsed, points_upserted, points_soft_deleted.
ingestion.job.failed	warning	Terminal failure. error_code says which stage: storage_upload_failed, parse_failed, chunk_limit_exceeded, embedding_failed, index_failed, timeout.
files.upload.duplicate	info	Identical content already ingested; the existing file/job was returned and nothing was re-ingested.
domain.rejected	warning	Upload refused before any row was written; reason is unregistered or disabled.
auth.failed	warning	reason is malformed_key, unknown_key, key_inactive, key_expired, or tenant_inactive. Never contains key material.
auth.succeeded	info	Carries tenant_id, api_key_id, actor_type.
lifespan.embedder.warm_failed	warning	An embedder was unreachable at startup. Boot continues by design — /readyz and the first upload are where this bites.
qdrant.bootstrap.schema_mismatch	error	The existing collection diverges from the pinned schema. The bootstrap exits non-zero; do not start the app against it.
api.unhandled_exception	error	A bug: an exception with no mapping to the error envelope. Always worth a look.

A 503 (ingestion_at_capacity) is rejected before a job row exists, so it appears in the access log and metrics, not in ingestion_jobs.

The durable record

Logs may roll; ingestion_jobs and ingestion_job_events do not. For one file:

SELECT id, status, error_code, error_message, points_created, points_soft_deleted,
       created_at, updated_at
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND source_file_id = :file_id
ORDER BY created_at DESC;

SELECT stage, level, message, details, created_at
FROM ingestion_job_events
WHERE tenant_id = :tenant_id AND ingestion_job_id = :ingestion_job_id
ORDER BY created_at;

Recent failures across a tenant:

SELECT error_code, count(*), max(created_at)
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND status = 'failed' AND created_at > now() - interval '1 day'
GROUP BY error_code ORDER BY 2 DESC;

GET /v1/files/{file_id} reports the same terminal status over HTTP, scoped to the owning tenant — a file belonging to another tenant returns 404, not 403.

10. Retrying a failed ingestion

Re-upload the same bytes. There is no retry endpoint and no automatic retry; the client owns that decision (ADR-0017).

What that guarantees:

• Identical content with a succeeded job is recognized by (tenant_id, domain, content_sha256) and returned as-is with 200 — no re-ingestion, no duplicate points.
• Identical content whose last job failed starts a fresh job against the same source_files row. A terminal job is never moved back to running.
• Point ids are deterministic from file_id + chunk_index (ADR-0001), so the retry overwrites in place — it cannot duplicate chunks.
• A failed attempt never empties or partially deletes a working index: the soft-delete sweep that retires a shortened file's leftover points runs only after every upsert has succeeded. An interrupted attempt can leave a prefix updated; a retry converges (ADR-0017, "Re-running an ingestion stays safe").

Fix the cause first — the error_code says where to look:

error_code	Usual cause
parse_failed	The file is corrupt or is not really the type its extension claims. Retrying identical bytes will fail identically.
chunk_limit_exceeded	The file is genuinely too large for one inline ingestion. Split it, or raise INGESTION_MAX_CHUNKS_PER_FILE knowing what §6 says about the timeout.
embedding_failed	The embedder is down, rate-limiting, or unauthenticated. Fix it, then retry — this one usually succeeds unchanged.
index_failed	Qdrant is down, or the collection is missing (run qdrant_bootstrap).
timeout	The work exceeded INGESTION_TIMEOUT_SECONDS. Check whether the embedder was cold (see lifespan.embedder.warm_failed and KEEP_ALIVE) before raising the bound.
storage_upload_failed	MinIO is unreachable or the bucket is missing (§3).

11. What to alert on

ADR-0017's own triggers for moving ingestion back off the request path. These are the numbers that say the inline design has stopped fitting:

• p95 ingestion duration approaching INGESTION_TIMEOUT_SECONDS.

• 503 and 504 rates ceasing to be negligible.

• Jobs stuck in running past the timeout — every handled failure writes a terminal status, so a non-zero count here means the process died mid-request:

  SELECT count(*) FROM ingestion_jobs
  WHERE status = 'running' AND created_at < now() - interval '5 minutes';
  

Also worth alerting: any qdrant.bootstrap.schema_mismatch, a sustained /readyz 503, and any api.unhandled_exception.

12. Verifying a deployment

./scripts/smoke.sh

Brings up Compose, runs both deployment steps, provisions a throwaway tenant, starts the web process, and drives an upload through to indexed Qdrant points against the running process — including asserting the structured log output from §9. It is the only Compose-based test; everything else runs on Testcontainers under uv run pytest (ADR-0016). Run it before a release.