# 0014. Durable job dispatch with NATS JetStream

## Status

Proposed

## Context

ADR-0008 makes file ingestion job-shaped: `POST /v1/files` returns `202 Accepted`
with `file_id`, `ingestion_job_id`, and a queued/running status. ADR-0009 makes
Postgres the durable source of truth for `source_files`, `ingestion_jobs`, and
`ingestion_job_events`. ADR-0013 stores uploaded file bytes in MinIO so ingestion
workers can fetch them after the request completes.

What is still missing is the delivery mechanism that tells workers which durable
jobs are ready to process.

The broker decision has to preserve existing boundaries:

- Postgres remains authoritative for job status, progress, audit, tenancy, and
  retention-sensitive state.
- MinIO stores file bytes and derived blobs, not queue state.
- Broker messages carry ids and correlation metadata, not raw files or chunks.
- Chat runs are not queued by this service: ADR-0007/0008 allow one in-flight run
  per `thread_id` and return `409 Conflict` for concurrent runs.
- Application-lifetime clients are created and closed by lifespan/startup owners
  under ADR-0012, not import-time globals.

## Decision

Use **NATS JetStream** as the durable pub/sub and job-dispatch broker for
asynchronous ingestion and maintenance work.

NATS core pub/sub alone is not enough for ingestion dispatch because ingestion
jobs need durable delivery, acknowledgement, redelivery, and worker restart
recovery. JetStream provides those broker-level mechanics while Postgres remains
the source of truth for application state.

### Start with ingestion jobs

The first broker-backed workflow is source-file ingestion:

```text
POST /v1/files
  -> authenticate and resolve tenant
  -> store source bytes in MinIO (ADR-0013)
  -> create/update source_files row
  -> create ingestion_jobs row with status='queued'
  -> commit the transaction
  -> publish a JetStream message containing ingestion_job_id
  -> return 202 Accepted
```

A dedicated ingestion worker process consumes the message:

```text
JetStream message
  -> load ingestion_jobs/source_files from Postgres
  -> bind logging context from durable ids
  -> fetch source object from MinIO
  -> parse, chunk, embed, and mutate Qdrant points
  -> append ingestion_job_events
  -> mark ingestion_jobs succeeded/failed/cancelled
  -> ack or negatively acknowledge the message
```

### Worker-owned ingestion chunk CRUD

For file ingestion and re-ingestion, generated chunk/point CRUD is performed by
the ingestion worker, not by the FastAPI publisher/request path.

The publisher only owns the lightweight durable handoff:

- authenticate the caller and derive tenant context;
- store uploaded bytes in MinIO;
- create or update `source_files`;
- create `ingestion_jobs(status='queued')`;
- commit the Postgres transaction;
- publish the JetStream message containing durable ids.

The ingestion worker owns the expensive and retryable side effects:

- parsing source files;
- chunking;
- embedding and reranking-vector generation when applicable;
- creating, updating, reordering, and soft-deleting generated Qdrant points;
- appending `ingestion_job_events`;
- updating ingestion job status, counters, and failure summaries.

Shared chunk/point mutation logic should live in a service/repository layer that
can be reused by both direct API routes and ingestion workers. Direct
`/v1/points` CRUD remains a synchronous API responsibility unless the operation
is explicitly bulk or job-shaped. Ingestion-generated mutations are asynchronous
worker responsibility because they must be safe to retry and resume by
`ingestion_job_id`.

### Keep messages small and versioned

Broker payloads carry identifiers and correlation metadata only. A typical
message is:

```json
{
  "type": "ingestion.job.created",
  "version": 1,
  "tenant_id": "...",
  "file_id": "...",
  "ingestion_job_id": "...",
  "request_id": "...",
  "api_key_id": "..."
}
```

Do not put raw file bytes, extracted text, chunks, embeddings, raw prompts, model
outputs, or secrets in JetStream messages.

Workers must treat message metadata as routing/correlation input, not as the only
authority. Before doing tenant-scoped work, workers reload the durable job and
file rows from Postgres and verify the ids are consistent.

### Use explicit subjects, streams, and durable consumers

Use stable, dot-separated subjects. Initial subjects:

```text
ingestion.job.created
ingestion.job.retry_requested
maintenance.retention.requested
maintenance.erasure.requested
```

Create an application-owned JetStream stream for durable work messages, for
example:

```text
CHATBOT_JOBS
subjects: ingestion.job.*, maintenance.*
```

Workers use durable consumers or queue groups so multiple worker processes can
share work without processing every message independently.

Ack messages only after the worker has persisted the resulting job state and
progress events to Postgres. Redelivery must be safe: ingestion processors should
be idempotent by `ingestion_job_id` and Qdrant point ids/upsert semantics.

### Run subscribers as worker processes, not FastAPI side effects

FastAPI may create a JetStream publisher/client during application lifespan so
routes can publish job notifications after durable state is committed.

Subscribers should run in separate worker process entrypoints. Do not start a
subscriber inside every FastAPI web worker by default: with multiple
Uvicorn/Gunicorn workers, each process has its own lifespan and could create
surprising duplicate consumers or resource pressure.

Worker processes create their own application-lifetime resources at startup:

- SQLAlchemy engine/sessionmaker;
- MinIO/S3 object-storage client;
- Qdrant client;
- NATS/JetStream client;
- observability/logging clients;
- ingestion model clients.

Each message gets its own request/job-lifetime SQLAlchemy `AsyncSession` and
explicit transaction boundaries, following ADR-0012.

### Handle commit/publish consistency deliberately

The initial implementation may publish the JetStream message after the Postgres
transaction commits. This avoids workers observing uncommitted jobs, but it means
a process crash between commit and publish can leave a job in `queued` state with
no broker message.

To handle that failure mode, workers or a small scheduler should periodically
scan Postgres for queued/stale ingestion jobs and publish or process them again.
This keeps Postgres authoritative and makes JetStream a wakeup/delivery
mechanism rather than the only source of pending work.

If stricter guarantees become necessary, introduce a transactional outbox table
in Postgres via a future ADR/migration:

```text
same DB transaction:
  insert/update ingestion_jobs
  insert outbox_events

publisher process:
  read unpublished outbox_events
  publish to JetStream
  mark outbox event published
```

Do not add DDL at FastAPI startup; all outbox schema changes must go through
Alembic as required by ADR-0009.

### Configure retries, dead letters, and observability

JetStream consumers should configure explicit acknowledgement, redelivery, and
maximum delivery behavior. Poison messages or repeatedly failing jobs should end
as durable Postgres failures with an `ingestion_job_events` error entry, not an
infinite invisible retry loop.

At worker ingress, bind structured logging context from durable identifiers:

- `tenant_id`;
- `request_id`;
- `api_key_id`;
- `file_id`;
- `ingestion_job_id`;
- JetStream stream/subject/sequence when available.

Use stable event names such as:

- `ingestion.job.dispatched`;
- `ingestion.job.received`;
- `ingestion.job.completed`;
- `ingestion.job.failed`;
- `broker.publish.failed`;
- `broker.message.redelivered`.

### Do not queue chat runs through JetStream yet

ADR-0007 and ADR-0008 intentionally reject queueing concurrent chat runs per
thread: one run may be in flight, and concurrent runs return `409 Conflict`.
This ADR does not change that decision.

A future ADR may introduce broker-backed work for chat-adjacent tasks such as
summarization, retention, erasure, or offline evaluation, but synchronous chat run
execution remains owned by the FastAPI/LangGraph boundary unless superseded.

## Consequences

### Positive
- File ingestion can move out of HTTP request handling into scalable worker
  processes without changing the REST contract from ADR-0008.
- JetStream gives durable delivery, ack/redelivery, and worker-group semantics
  without adopting Kafka-level operational complexity.
- Postgres remains the auditable source of truth for jobs, progress, tenant
  ownership, and failure state.
- MinIO, Postgres, Qdrant, and the broker each have distinct responsibilities:
  bytes, metadata/state, vector search, and work delivery.
- Separate worker processes avoid surprising subscriptions in every FastAPI web
  worker and make GPU/model-heavy ingestion resources easier to size.

### Negative
- Adds another external service to run, secure, monitor, and back up where stream
  durability is required.
- Developers must reason about redelivery and idempotency; handlers may run more
  than once for the same `ingestion_job_id`.
- Publishing after commit has a crash window unless a stale-job scanner or future
  transactional outbox is implemented.
- NATS/JetStream stream, consumer, retention, and dead-letter configuration must
  be managed explicitly per environment.
- The app now needs worker deployment/runbook conventions in addition to the
  FastAPI web process.

## Alternatives Considered

- **NATS core pub/sub only**: rejected. Core pub/sub is useful for ephemeral
  events, but ingestion dispatch needs durable delivery and acknowledgement.
  JetStream is the NATS feature that provides those semantics.
- **Redis Streams**: rejected for this decision. Redis Streams would work for an
  MVP and has simple local deployment, but NATS JetStream is a cleaner dedicated
  broker for durable pub/sub and worker coordination without also mixing cache
  responsibilities into the same service.
- **RabbitMQ**: rejected for now. It is mature for work queues and routing, but
  the project does not currently need its exchange/routing complexity, and NATS
  JetStream keeps the broker footprint lighter.
- **Kafka**: rejected. Kafka is an excellent durable event log at high scale, but
  its operational overhead is unnecessary for this service's initial ingestion
  and maintenance jobs.
- **FastAPI `BackgroundTasks` or in-process asyncio tasks**: rejected as the
  durable design. They are simple, but work can be lost on process restart and
  they do not coordinate multiple workers cleanly.
- **Database polling only**: rejected as the main dispatch mechanism. Polling
  Postgres for queued jobs is a useful repair path, but relying on polling alone
  adds latency and unnecessary database load once a broker is available.
- **Use the broker as the source of truth for job state**: rejected. Broker state
  is delivery state. Application job state, progress, audit, and tenant ownership
  remain in Postgres.