# Operator runbook

How to start this service, configure its ingestion bounds, and investigate or
retry a failed upload. Architecture rationale lives in [`docs/adr/`](adr/); the
implementation milestone is
[plan 001](plans/001-ingestion-vertical-slice.md). This document covers
operating what those describe.

The service is a **single process with no background work**. `POST /v1/files`
parses, chunks, embeds, and indexes inline and returns a terminal result
(ADR-0017). There is no queue, no worker, and no automatic retry — the caller
owns the retry decision, which makes the request's duration a deployment
constraint. That fact drives most of this document.

## 1. Prerequisites and local startup

Docker, and [`uv`](https://docs.astral.sh/uv/) with Python 3.13.

```bash
cp .env.example .env      # non-secret local defaults; .env is gitignored
uv sync
docker compose up -d --wait
```

`docker-compose.yml` runs Postgres (`127.0.0.1:5433`), MinIO
(`127.0.0.1:9100`, console `9101`), and Qdrant (`127.0.0.1:6343`). It is the
local development stack and says so in its header — it is not a production
deployment.

## 2. Deployment steps

Two schema steps run **before** the application, never at startup: FastAPI
performs no DDL, for Postgres (ADR-0009) or for Qdrant (ADR-0001, "Collection
provisioning"). Both commands and their reasoning are in the README's
[Provisioning the datastores](../README.md#provisioning-the-datastores)
section:

```bash
uv run alembic upgrade head                   # Postgres schema
uv run python -m src.cli.qdrant_bootstrap     # the `chunks` collection
```

Both are idempotent. `qdrant_bootstrap` verifies an existing collection against
the pinned schema and **exits non-zero on a mismatch** rather than leaving a
silently degraded sparse index in place — the `sparse` vector's
`modifier="idf"` and the pinned dense dimensions (768 / 3072) fail silently if
wrong, which is why they are checked rather than assumed.

Run both again after every deploy that ships a migration or a collection-schema
change.

## 3. MinIO bucket

Under Compose the bucket already exists: the `app-minio` service's entrypoint
runs `mkdir -p /data/${MINIO_BUCKET:-chatbot-source-files}` before starting the
server, so first boot creates it. Nothing else needs to be done locally.

Outside Compose, create the bucket named by `MINIO_BUCKET` before the first
upload — the application never creates it. It must stay **private**; ADR-0013
keeps source bytes non-public and this slice ships no download API or presigned
URLs.

## 4. Provisioning a tenant, an API key, and its domains

Nothing over HTTP can bootstrap a tenant: every `/v1` route needs an API key,
and a key cannot exist before its tenant. So the first key is issued by an
operator command:

```bash
uv run python -m src.cli.provision_tenant \
  --slug acme --domain fire --domain life --scopes files:write,domains:read
```

It prints `api_key=sk_...` **once**. Postgres stores only its SHA-256 hash
(ADR-0009), so a lost key is reissued by re-running the command, never
recovered. Structured logs carry only the non-secret `key_prefix` — a plaintext
key must never reach a log sink (ADR-0011).

Re-running with the same `--slug` reuses the tenant and any domains it already
has, and issues an **additional** key. Both keys stay valid; this adds a key, it
does not rotate one.

Scopes are the security boundary between uploading, reading chunks, and
managing the allowlist. Give an upload client `files:write` only. `domains:write`
lets its holder create new domains, which is exactly what the allowlist exists to
prevent an upload key from doing, and `points:read` lets its holder read the text
of every chunk of every file — so an upload-only key gets neither.

| Scope | Grants |
|---|---|
| `files:write` | Upload a document and read its ingestion status. |
| `points:read` | Read, list, count, and keyword-search this tenant's points, including `GET /v1/files/{file_id}/points`. |
| `points:write` | Create, edit, reorder, and soft-delete points (plan 002 Phases 3-5; no route uses it yet). |
| `domains:read` / `domains:write` | Inspect and manage the domain allowlist. |
| `admin` | Satisfies every scope check. |

The command's `--scopes` default issues all of the above except `admin`, which
suits a first operator key; narrow it explicitly for per-client keys.

### Domains after the first one

`POST /v1/files` rejects an unregistered or disabled `domain` with `400`
(`unknown_domain`) before anything is written. Ongoing domain management is the
`/v1/domains` API, under `domains:read` / `domains:write`:

```bash
curl -X POST http://localhost:8000/v1/domains \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"domain": "fire", "display_name": "Fire insurance"}'
```

The `domain` key itself is immutable — it is denormalized into every Qdrant
point payload and into `source_files`, so renaming it is a migration, not an
edit (ADR-0009). Disabling a domain blocks new uploads; it does not delete
existing points.

## 5. Running the service

```bash
uv run fastapi dev src/main.py                                   # local, reload
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000           # deployed shape
```

Run more than one worker/replica only after reading §6: ingestion bounds are
**per process**, so `INGESTION_MAX_CONCURRENCY` multiplies by the number of
processes.

## 6. Ingestion bounds and tuning

Every bound is enforced server-side and maps to a status code. All are in
`.env.example`. Ingestion is CPU- and network-bound in the request, so these are
the numbers that decide whether the service degrades gracefully or falls over.

| Setting | Bounds | On breach | Size it against |
|---|---|---|---|
| `INGESTION_MAX_CONCURRENCY` | Ingestions in flight **per process** | `503` + `Retry-After` | Memory per in-flight upload (whole file plus its chunks and vectors are resident) and the embedder's capacity. Rejecting is deliberate: ADR-0017 refuses rather than queues. |
| `INGESTION_THREAD_POOL_SIZE` | Threads for blocking work (parse, chunk, hash, BM25, the sync `minio` SDK) | — (waits) | CPU cores. It exists to stop ingestion exhausting Starlette's own thread pool, so keep it below the total thread budget. |
| `INGESTION_TIMEOUT_SECONDS` | The whole work phase | `504`, job marked `failed` | The slowest legitimate document, plus headroom. See §7 — this must stay under every read timeout in front of it. |
| `INGESTION_MAX_UPLOAD_SIZE_MB` | Bytes accepted | `413` | Memory: the upload is read fully into the process before any work starts. |
| `INGESTION_MAX_CHUNKS_PER_FILE` | Chunks per file, checked before embedding | `413` | Embedder cost/time per chunk × `INGESTION_TIMEOUT_SECONDS`. This is the real defence against one pathological file eating a slot. |
| `INGESTION_EMBED_BATCH_SIZE` | Texts per embedder request | `502` on embedder failure | The provider's per-request limits. Batch before parallelizing. |
| `INGESTION_EMBED_CONCURRENCY` | Concurrent embed batches | `502` | Provider rate limits and the self-hosted embedder's throughput. Never unbounded. |
| `QDRANT_UPSERT_BATCH_SIZE` / `_CONCURRENCY` | Points per upsert and concurrent upserts | `502` (`index_error`) | Qdrant's ingest capacity; the batch size stays in ADR-0001's 64–256 band. |

Two settings that look like tuning knobs but are not:

- **`EMBEDDING_NOMIC_KEEP_ALIVE`** holds the self-hosted model resident. A cold
  load of `nomic-embed-text-v2-moe` takes over 150 s — longer than any sane
  `INGESTION_TIMEOUT_SECONDS` — so an idle period followed by an upload would
  otherwise `504`. The lifespan also warms both dense embedders at startup for
  the same reason.
- **The BM25 analyzer and weights** (`EMBEDDING_SPARSE_*`) are a measured
  artifact ported from the `emet` evaluation lab, verified token-for-token
  against it (ADR-0005). Re-benchmark; do not tune them in place.

## 7. The proxy and client read-timeout requirement

**Every read timeout in front of this service must exceed
`INGESTION_TIMEOUT_SECONDS`.** That includes the reverse proxy / ingress, any
load balancer, and the calling backend's own HTTP client.

If a proxy times out first, the client gets that proxy's error, the upload keeps
running in the process, and the caller learns nothing about the outcome from the
response. The job row still reaches a terminal status, so
`GET /v1/files/{file_id}` remains the way to find out what happened — but the
response contract is broken for that request. ADR-0017 names this the main cost
of inline ingestion.

A workable local ordering: client read timeout > proxy read timeout >
`INGESTION_TIMEOUT_SECONDS`.

## 8. Health and readiness

| Endpoint | Question it answers | Use for |
|---|---|---|
| `GET /healthz` | Is the process alive? | Liveness probes / restart policy. Never depends on Postgres, MinIO, or Qdrant. |
| `GET /readyz` | Can it actually serve? | Load-balancer admission and post-deploy gating. `200` with each dependency `true`, `503` if any is `false`. |

`/readyz` checks Postgres, MinIO, and Qdrant reachability **and** that the
`chunks` collection exists. A reachable-but-unbootstrapped Qdrant reports
`{"qdrant": false}` on purpose: uploads to it would fail with `502`, so it is
not ready, and this is how a skipped `qdrant_bootstrap` surfaces at deploy time
instead of on a user's first upload.

## 9. Investigating a failure

Logs are structured (`structlog`, JSON in production) with stable event names —
grep the event name, not prose (ADR-0011). Set `LOG_FILE_PATH` for a local
JSON file sink alongside the console renderer; leave it unset in production,
where stdout collection is preferred.

**Correlate by `request_id`.** Every request has one, echoed in the
`X-Request-Id` response header and included in every error envelope, and bound
into every log line emitted while handling that request. A client reporting a
failed upload should quote it. `tenant_id`, `file_id`, and `ingestion_job_id`
are the other join keys.

Events worth knowing:

| Event | Level | Means |
|---|---|---|
| `ingestion.job.started` | info | Txn A committed; work phase beginning. Carries `tenant_id`, `ingestion_job_id`, `file_id`, `domain`, `source_type`. |
| `ingestion.job.completed` | info | Terminal success, with `chunks_parsed`, `points_upserted`, `points_soft_deleted`. |
| `ingestion.job.failed` | warning | Terminal failure. **`error_code` says which stage**: `storage_upload_failed`, `parse_failed`, `chunk_limit_exceeded`, `embedding_failed`, `index_failed`, `timeout`. |
| `files.upload.duplicate` | info | Identical content already ingested; the existing file/job was returned and nothing was re-ingested. |
| `domain.rejected` | warning | Upload refused before any row was written; `reason` is `unregistered` or `disabled`. |
| `auth.failed` | warning | `reason` is `malformed_key`, `unknown_key`, `key_inactive`, `key_expired`, or `tenant_inactive`. Never contains key material. |
| `auth.succeeded` | info | Carries `tenant_id`, `api_key_id`, `actor_type`. |
| `lifespan.embedder.warm_failed` | warning | An embedder was unreachable at startup. Boot continues by design — `/readyz` and the first upload are where this bites. |
| `qdrant.bootstrap.schema_mismatch` | error | The existing collection diverges from the pinned schema. The bootstrap exits non-zero; do not start the app against it. |
| `api.unhandled_exception` | error | A bug: an exception with no mapping to the error envelope. Always worth a look. |

A `503` (`ingestion_at_capacity`) is rejected before a job row exists, so it
appears in the access log and metrics, not in `ingestion_jobs`.

### The durable record

Logs may roll; `ingestion_jobs` and `ingestion_job_events` do not. For one file:

```sql
SELECT id, status, error_code, error_message, points_created, points_soft_deleted,
       created_at, updated_at
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND source_file_id = :file_id
ORDER BY created_at DESC;

SELECT stage, level, message, details, created_at
FROM ingestion_job_events
WHERE tenant_id = :tenant_id AND ingestion_job_id = :ingestion_job_id
ORDER BY created_at;
```

Recent failures across a tenant:

```sql
SELECT error_code, count(*), max(created_at)
FROM ingestion_jobs
WHERE tenant_id = :tenant_id AND status = 'failed' AND created_at > now() - interval '1 day'
GROUP BY error_code ORDER BY 2 DESC;
```

`GET /v1/files/{file_id}` reports the same terminal status over HTTP, scoped to
the owning tenant — a file belonging to another tenant returns `404`, not `403`.

## 10. Retrying a failed ingestion

**Re-upload the same bytes.** There is no retry endpoint and no automatic retry;
the client owns that decision (ADR-0017).

What that guarantees:

- Identical content with a **succeeded** job is recognized by
  `(tenant_id, domain, content_sha256)` and returned as-is with `200` — no
  re-ingestion, no duplicate points.
- Identical content whose last job **failed** starts a fresh job against the
  same `source_files` row. A terminal job is never moved back to `running`.
- Point ids are deterministic from `file_id` + `chunk_index` (ADR-0001), so the
  retry **overwrites in place** — it cannot duplicate chunks.
- A failed attempt never empties or partially deletes a working index: the
  soft-delete sweep that retires a shortened file's leftover points runs only
  after every upsert has succeeded. An interrupted attempt can leave a prefix
  updated; a retry converges (ADR-0017, "Re-running an ingestion stays safe").

Fix the cause first — the `error_code` says where to look:

| `error_code` | Usual cause |
|---|---|
| `parse_failed` | The file is corrupt or is not really the type its extension claims. Retrying identical bytes will fail identically. |
| `chunk_limit_exceeded` | The file is genuinely too large for one inline ingestion. Split it, or raise `INGESTION_MAX_CHUNKS_PER_FILE` knowing what §6 says about the timeout. |
| `embedding_failed` | The embedder is down, rate-limiting, or unauthenticated. Fix it, then retry — this one usually succeeds unchanged. |
| `index_failed` | Qdrant is down, or the collection is missing (run `qdrant_bootstrap`). |
| `timeout` | The work exceeded `INGESTION_TIMEOUT_SECONDS`. Check whether the embedder was cold (see `lifespan.embedder.warm_failed` and `KEEP_ALIVE`) before raising the bound. |
| `storage_upload_failed` | MinIO is unreachable or the bucket is missing (§3). |

## 11. What to alert on

ADR-0017's own triggers for moving ingestion back off the request path. These
are the numbers that say the inline design has stopped fitting:

- **p95 ingestion duration** approaching `INGESTION_TIMEOUT_SECONDS`.
- **`503` and `504` rates** ceasing to be negligible.
- **Jobs stuck in `running` past the timeout** — every handled failure writes a
  terminal status, so a non-zero count here means the process died mid-request:

  ```sql
  SELECT count(*) FROM ingestion_jobs
  WHERE status = 'running' AND created_at < now() - interval '5 minutes';
  ```

Also worth alerting: any `qdrant.bootstrap.schema_mismatch`, a sustained
`/readyz` `503`, and any `api.unhandled_exception`.

## 12. Verifying a deployment

```bash
./scripts/smoke.sh
```

Brings up Compose, runs both deployment steps, provisions a throwaway tenant,
starts the web process, and drives an upload through to indexed Qdrant points
against the **running process** — including asserting the structured log output
from §9. It is the only Compose-based test; everything else runs on
Testcontainers under `uv run pytest` (ADR-0016). Run it before a release.