Ali Zarinkolah
5251990444
Why: - points:read and points:write are named in ADR-0009 but were absent from DEFAULT_SCOPES, so a provisioned key could not reach the read paths that follow. The runbook documented only files:write and domains:write, which understated what a default key can now do. Changes: - Add points:read and points:write to DEFAULT_SCOPES. - Document the full scope table in the runbook, calling out that points:read grants the text of every chunk of every file -- so an upload-only key gets files:write alone. Impact: - Keys issued before this change keep their existing scopes; provisioning does not backfill. Reissue or widen an existing key explicitly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
304 lines
15 KiB
Markdown
304 lines
15 KiB
Markdown
# Operator runbook
|
||
|
||
How to start this service, configure its ingestion bounds, and investigate or
|
||
retry a failed upload. Architecture rationale lives in [`docs/adr/`](adr/); the
|
||
implementation milestone is
|
||
[plan 001](plans/001-ingestion-vertical-slice.md). This document covers
|
||
operating what those describe.
|
||
|
||
The service is a **single process with no background work**. `POST /v1/files`
|
||
parses, chunks, embeds, and indexes inline and returns a terminal result
|
||
(ADR-0017). There is no queue, no worker, and no automatic retry — the caller
|
||
owns the retry decision, which makes the request's duration a deployment
|
||
constraint. That fact drives most of this document.
|
||
|
||
## 1. Prerequisites and local startup
|
||
|
||
Docker, and [`uv`](https://docs.astral.sh/uv/) with Python 3.13.
|
||
|
||
```bash
|
||
cp .env.example .env # non-secret local defaults; .env is gitignored
|
||
uv sync
|
||
docker compose up -d --wait
|
||
```
|
||
|
||
`docker-compose.yml` runs Postgres (`127.0.0.1:5433`), MinIO
|
||
(`127.0.0.1:9100`, console `9101`), and Qdrant (`127.0.0.1:6343`). It is the
|
||
local development stack and says so in its header — it is not a production
|
||
deployment.
|
||
|
||
## 2. Deployment steps
|
||
|
||
Two schema steps run **before** the application, never at startup: FastAPI
|
||
performs no DDL, for Postgres (ADR-0009) or for Qdrant (ADR-0001, "Collection
|
||
provisioning"). Both commands and their reasoning are in the README's
|
||
[Provisioning the datastores](../README.md#provisioning-the-datastores)
|
||
section:
|
||
|
||
```bash
|
||
uv run alembic upgrade head # Postgres schema
|
||
uv run python -m src.cli.qdrant_bootstrap # the `chunks` collection
|
||
```
|
||
|
||
Both are idempotent. `qdrant_bootstrap` verifies an existing collection against
|
||
the pinned schema and **exits non-zero on a mismatch** rather than leaving a
|
||
silently degraded sparse index in place — the `sparse` vector's
|
||
`modifier="idf"` and the pinned dense dimensions (768 / 3072) fail silently if
|
||
wrong, which is why they are checked rather than assumed.
|
||
|
||
Run both again after every deploy that ships a migration or a collection-schema
|
||
change.
|
||
|
||
## 3. MinIO bucket
|
||
|
||
Under Compose the bucket already exists: the `app-minio` service's entrypoint
|
||
runs `mkdir -p /data/${MINIO_BUCKET:-chatbot-source-files}` before starting the
|
||
server, so first boot creates it. Nothing else needs to be done locally.
|
||
|
||
Outside Compose, create the bucket named by `MINIO_BUCKET` before the first
|
||
upload — the application never creates it. It must stay **private**; ADR-0013
|
||
keeps source bytes non-public and this slice ships no download API or presigned
|
||
URLs.
|
||
|
||
## 4. Provisioning a tenant, an API key, and its domains
|
||
|
||
Nothing over HTTP can bootstrap a tenant: every `/v1` route needs an API key,
|
||
and a key cannot exist before its tenant. So the first key is issued by an
|
||
operator command:
|
||
|
||
```bash
|
||
uv run python -m src.cli.provision_tenant \
|
||
--slug acme --domain fire --domain life --scopes files:write,domains:read
|
||
```
|
||
|
||
It prints `api_key=sk_...` **once**. Postgres stores only its SHA-256 hash
|
||
(ADR-0009), so a lost key is reissued by re-running the command, never
|
||
recovered. Structured logs carry only the non-secret `key_prefix` — a plaintext
|
||
key must never reach a log sink (ADR-0011).
|
||
|
||
Re-running with the same `--slug` reuses the tenant and any domains it already
|
||
has, and issues an **additional** key. Both keys stay valid; this adds a key, it
|
||
does not rotate one.
|
||
|
||
Scopes are the security boundary between uploading, reading chunks, and
|
||
managing the allowlist. Give an upload client `files:write` only. `domains:write`
|
||
lets its holder create new domains, which is exactly what the allowlist exists to
|
||
prevent an upload key from doing, and `points:read` lets its holder read the text
|
||
of every chunk of every file — so an upload-only key gets neither.
|
||
|
||
| Scope | Grants |
|
||
|---|---|
|
||
| `files:write` | Upload a document and read its ingestion status. |
|
||
| `points:read` | Read, list, count, and keyword-search this tenant's points, including `GET /v1/files/{file_id}/points`. |
|
||
| `points:write` | Create, edit, reorder, and soft-delete points (plan 002 Phases 3-5; no route uses it yet). |
|
||
| `domains:read` / `domains:write` | Inspect and manage the domain allowlist. |
|
||
| `admin` | Satisfies every scope check. |
|
||
|
||
The command's `--scopes` default issues all of the above except `admin`, which
|
||
suits a first operator key; narrow it explicitly for per-client keys.
|
||
|
||
### Domains after the first one
|
||
|
||
`POST /v1/files` rejects an unregistered or disabled `domain` with `400`
|
||
(`unknown_domain`) before anything is written. Ongoing domain management is the
|
||
`/v1/domains` API, under `domains:read` / `domains:write`:
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8000/v1/domains \
|
||
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
|
||
-d '{"domain": "fire", "display_name": "Fire insurance"}'
|
||
```
|
||
|
||
The `domain` key itself is immutable — it is denormalized into every Qdrant
|
||
point payload and into `source_files`, so renaming it is a migration, not an
|
||
edit (ADR-0009). Disabling a domain blocks new uploads; it does not delete
|
||
existing points.
|
||
|
||
## 5. Running the service
|
||
|
||
```bash
|
||
uv run fastapi dev src/main.py # local, reload
|
||
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000 # deployed shape
|
||
```
|
||
|
||
Run more than one worker/replica only after reading §6: ingestion bounds are
|
||
**per process**, so `INGESTION_MAX_CONCURRENCY` multiplies by the number of
|
||
processes.
|
||
|
||
## 6. Ingestion bounds and tuning
|
||
|
||
Every bound is enforced server-side and maps to a status code. All are in
|
||
`.env.example`. Ingestion is CPU- and network-bound in the request, so these are
|
||
the numbers that decide whether the service degrades gracefully or falls over.
|
||
|
||
| Setting | Bounds | On breach | Size it against |
|
||
|---|---|---|---|
|
||
| `INGESTION_MAX_CONCURRENCY` | Ingestions in flight **per process** | `503` + `Retry-After` | Memory per in-flight upload (whole file plus its chunks and vectors are resident) and the embedder's capacity. Rejecting is deliberate: ADR-0017 refuses rather than queues. |
|
||
| `INGESTION_THREAD_POOL_SIZE` | Threads for blocking work (parse, chunk, hash, BM25, the sync `minio` SDK) | — (waits) | CPU cores. It exists to stop ingestion exhausting Starlette's own thread pool, so keep it below the total thread budget. |
|
||
| `INGESTION_TIMEOUT_SECONDS` | The whole work phase | `504`, job marked `failed` | The slowest legitimate document, plus headroom. See §7 — this must stay under every read timeout in front of it. |
|
||
| `INGESTION_MAX_UPLOAD_SIZE_MB` | Bytes accepted | `413` | Memory: the upload is read fully into the process before any work starts. |
|
||
| `INGESTION_MAX_CHUNKS_PER_FILE` | Chunks per file, checked before embedding | `413` | Embedder cost/time per chunk × `INGESTION_TIMEOUT_SECONDS`. This is the real defence against one pathological file eating a slot. |
|
||
| `INGESTION_EMBED_BATCH_SIZE` | Texts per embedder request | `502` on embedder failure | The provider's per-request limits. Batch before parallelizing. |
|
||
| `INGESTION_EMBED_CONCURRENCY` | Concurrent embed batches | `502` | Provider rate limits and the self-hosted embedder's throughput. Never unbounded. |
|
||
| `QDRANT_UPSERT_BATCH_SIZE` / `_CONCURRENCY` | Points per upsert and concurrent upserts | `502` (`index_error`) | Qdrant's ingest capacity; the batch size stays in ADR-0001's 64–256 band. |
|
||
|
||
Two settings that look like tuning knobs but are not:
|
||
|
||
- **`EMBEDDING_NOMIC_KEEP_ALIVE`** holds the self-hosted model resident. A cold
|
||
load of `nomic-embed-text-v2-moe` takes over 150 s — longer than any sane
|
||
`INGESTION_TIMEOUT_SECONDS` — so an idle period followed by an upload would
|
||
otherwise `504`. The lifespan also warms both dense embedders at startup for
|
||
the same reason.
|
||
- **The BM25 analyzer and weights** (`EMBEDDING_SPARSE_*`) are a measured
|
||
artifact ported from the `emet` evaluation lab, verified token-for-token
|
||
against it (ADR-0005). Re-benchmark; do not tune them in place.
|
||
|
||
## 7. The proxy and client read-timeout requirement
|
||
|
||
**Every read timeout in front of this service must exceed
|
||
`INGESTION_TIMEOUT_SECONDS`.** That includes the reverse proxy / ingress, any
|
||
load balancer, and the calling backend's own HTTP client.
|
||
|
||
If a proxy times out first, the client gets that proxy's error, the upload keeps
|
||
running in the process, and the caller learns nothing about the outcome from the
|
||
response. The job row still reaches a terminal status, so
|
||
`GET /v1/files/{file_id}` remains the way to find out what happened — but the
|
||
response contract is broken for that request. ADR-0017 names this the main cost
|
||
of inline ingestion.
|
||
|
||
A workable local ordering: client read timeout > proxy read timeout >
|
||
`INGESTION_TIMEOUT_SECONDS`.
|
||
|
||
## 8. Health and readiness
|
||
|
||
| Endpoint | Question it answers | Use for |
|
||
|---|---|---|
|
||
| `GET /healthz` | Is the process alive? | Liveness probes / restart policy. Never depends on Postgres, MinIO, or Qdrant. |
|
||
| `GET /readyz` | Can it actually serve? | Load-balancer admission and post-deploy gating. `200` with each dependency `true`, `503` if any is `false`. |
|
||
|
||
`/readyz` checks Postgres, MinIO, and Qdrant reachability **and** that the
|
||
`chunks` collection exists. A reachable-but-unbootstrapped Qdrant reports
|
||
`{"qdrant": false}` on purpose: uploads to it would fail with `502`, so it is
|
||
not ready, and this is how a skipped `qdrant_bootstrap` surfaces at deploy time
|
||
instead of on a user's first upload.
|
||
|
||
## 9. Investigating a failure
|
||
|
||
Logs are structured (`structlog`, JSON in production) with stable event names —
|
||
grep the event name, not prose (ADR-0011). Set `LOG_FILE_PATH` for a local
|
||
JSON file sink alongside the console renderer; leave it unset in production,
|
||
where stdout collection is preferred.
|
||
|
||
**Correlate by `request_id`.** Every request has one, echoed in the
|
||
`X-Request-Id` response header and included in every error envelope, and bound
|
||
into every log line emitted while handling that request. A client reporting a
|
||
failed upload should quote it. `tenant_id`, `file_id`, and `ingestion_job_id`
|
||
are the other join keys.
|
||
|
||
Events worth knowing:
|
||
|
||
| Event | Level | Means |
|
||
|---|---|---|
|
||
| `ingestion.job.started` | info | Txn A committed; work phase beginning. Carries `tenant_id`, `ingestion_job_id`, `file_id`, `domain`, `source_type`. |
|
||
| `ingestion.job.completed` | info | Terminal success, with `chunks_parsed`, `points_upserted`, `points_soft_deleted`. |
|
||
| `ingestion.job.failed` | warning | Terminal failure. **`error_code` says which stage**: `storage_upload_failed`, `parse_failed`, `chunk_limit_exceeded`, `embedding_failed`, `index_failed`, `timeout`. |
|
||
| `files.upload.duplicate` | info | Identical content already ingested; the existing file/job was returned and nothing was re-ingested. |
|
||
| `domain.rejected` | warning | Upload refused before any row was written; `reason` is `unregistered` or `disabled`. |
|
||
| `auth.failed` | warning | `reason` is `malformed_key`, `unknown_key`, `key_inactive`, `key_expired`, or `tenant_inactive`. Never contains key material. |
|
||
| `auth.succeeded` | info | Carries `tenant_id`, `api_key_id`, `actor_type`. |
|
||
| `lifespan.embedder.warm_failed` | warning | An embedder was unreachable at startup. Boot continues by design — `/readyz` and the first upload are where this bites. |
|
||
| `qdrant.bootstrap.schema_mismatch` | error | The existing collection diverges from the pinned schema. The bootstrap exits non-zero; do not start the app against it. |
|
||
| `api.unhandled_exception` | error | A bug: an exception with no mapping to the error envelope. Always worth a look. |
|
||
|
||
A `503` (`ingestion_at_capacity`) is rejected before a job row exists, so it
|
||
appears in the access log and metrics, not in `ingestion_jobs`.
|
||
|
||
### The durable record
|
||
|
||
Logs may roll; `ingestion_jobs` and `ingestion_job_events` do not. For one file:
|
||
|
||
```sql
|
||
SELECT id, status, error_code, error_message, points_created, points_soft_deleted,
|
||
created_at, updated_at
|
||
FROM ingestion_jobs
|
||
WHERE tenant_id = :tenant_id AND source_file_id = :file_id
|
||
ORDER BY created_at DESC;
|
||
|
||
SELECT stage, level, message, details, created_at
|
||
FROM ingestion_job_events
|
||
WHERE tenant_id = :tenant_id AND ingestion_job_id = :ingestion_job_id
|
||
ORDER BY created_at;
|
||
```
|
||
|
||
Recent failures across a tenant:
|
||
|
||
```sql
|
||
SELECT error_code, count(*), max(created_at)
|
||
FROM ingestion_jobs
|
||
WHERE tenant_id = :tenant_id AND status = 'failed' AND created_at > now() - interval '1 day'
|
||
GROUP BY error_code ORDER BY 2 DESC;
|
||
```
|
||
|
||
`GET /v1/files/{file_id}` reports the same terminal status over HTTP, scoped to
|
||
the owning tenant — a file belonging to another tenant returns `404`, not `403`.
|
||
|
||
## 10. Retrying a failed ingestion
|
||
|
||
**Re-upload the same bytes.** There is no retry endpoint and no automatic retry;
|
||
the client owns that decision (ADR-0017).
|
||
|
||
What that guarantees:
|
||
|
||
- Identical content with a **succeeded** job is recognized by
|
||
`(tenant_id, domain, content_sha256)` and returned as-is with `200` — no
|
||
re-ingestion, no duplicate points.
|
||
- Identical content whose last job **failed** starts a fresh job against the
|
||
same `source_files` row. A terminal job is never moved back to `running`.
|
||
- Point ids are deterministic from `file_id` + `chunk_index` (ADR-0001), so the
|
||
retry **overwrites in place** — it cannot duplicate chunks.
|
||
- A failed attempt never empties or partially deletes a working index: the
|
||
soft-delete sweep that retires a shortened file's leftover points runs only
|
||
after every upsert has succeeded. An interrupted attempt can leave a prefix
|
||
updated; a retry converges (ADR-0017, "Re-running an ingestion stays safe").
|
||
|
||
Fix the cause first — the `error_code` says where to look:
|
||
|
||
| `error_code` | Usual cause |
|
||
|---|---|
|
||
| `parse_failed` | The file is corrupt or is not really the type its extension claims. Retrying identical bytes will fail identically. |
|
||
| `chunk_limit_exceeded` | The file is genuinely too large for one inline ingestion. Split it, or raise `INGESTION_MAX_CHUNKS_PER_FILE` knowing what §6 says about the timeout. |
|
||
| `embedding_failed` | The embedder is down, rate-limiting, or unauthenticated. Fix it, then retry — this one usually succeeds unchanged. |
|
||
| `index_failed` | Qdrant is down, or the collection is missing (run `qdrant_bootstrap`). |
|
||
| `timeout` | The work exceeded `INGESTION_TIMEOUT_SECONDS`. Check whether the embedder was cold (see `lifespan.embedder.warm_failed` and `KEEP_ALIVE`) before raising the bound. |
|
||
| `storage_upload_failed` | MinIO is unreachable or the bucket is missing (§3). |
|
||
|
||
## 11. What to alert on
|
||
|
||
ADR-0017's own triggers for moving ingestion back off the request path. These
|
||
are the numbers that say the inline design has stopped fitting:
|
||
|
||
- **p95 ingestion duration** approaching `INGESTION_TIMEOUT_SECONDS`.
|
||
- **`503` and `504` rates** ceasing to be negligible.
|
||
- **Jobs stuck in `running` past the timeout** — every handled failure writes a
|
||
terminal status, so a non-zero count here means the process died mid-request:
|
||
|
||
```sql
|
||
SELECT count(*) FROM ingestion_jobs
|
||
WHERE status = 'running' AND created_at < now() - interval '5 minutes';
|
||
```
|
||
|
||
Also worth alerting: any `qdrant.bootstrap.schema_mismatch`, a sustained
|
||
`/readyz` `503`, and any `api.unhandled_exception`.
|
||
|
||
## 12. Verifying a deployment
|
||
|
||
```bash
|
||
./scripts/smoke.sh
|
||
```
|
||
|
||
Brings up Compose, runs both deployment steps, provisions a throwaway tenant,
|
||
starts the web process, and drives an upload through to indexed Qdrant points
|
||
against the **running process** — including asserting the structured log output
|
||
from §9. It is the only Compose-based test; everything else runs on
|
||
Testcontainers under `uv run pytest` (ADR-0016). Run it before a release.
|