chatbot_v3/docs/adr/0014-durable-job-dispatch-with-nats-jetstream.md

0014. Durable job dispatch with NATS JetStream

Status

Proposed

Context

ADR-0008 makes file ingestion job-shaped: POST /v1/files returns 202 Accepted with file_id, ingestion_job_id, and a queued/running status. ADR-0009 makes Postgres the durable source of truth for source_files, ingestion_jobs, and ingestion_job_events. ADR-0013 stores uploaded file bytes in MinIO so ingestion workers can fetch them after the request completes.

What is still missing is the delivery mechanism that tells workers which durable jobs are ready to process.

The broker decision has to preserve existing boundaries:

• Postgres remains authoritative for job status, progress, audit, tenancy, and retention-sensitive state.
• MinIO stores file bytes and derived blobs, not queue state.
• Broker messages carry ids and correlation metadata, not raw files or chunks.
• Chat runs are not queued by this service: ADR-0007/0008 allow one in-flight run per thread_id and return 409 Conflict for concurrent runs.
• Application-lifetime clients are created and closed by lifespan/startup owners under ADR-0012, not import-time globals.

Decision

Use NATS JetStream as the durable pub/sub and job-dispatch broker for asynchronous ingestion and maintenance work.

NATS core pub/sub alone is not enough for ingestion dispatch because ingestion jobs need durable delivery, acknowledgement, redelivery, and worker restart recovery. JetStream provides those broker-level mechanics while Postgres remains the source of truth for application state.

Start with ingestion jobs

The first broker-backed workflow is source-file ingestion:

POST /v1/files
  -> authenticate and resolve tenant
  -> store source bytes in MinIO (ADR-0013)
  -> create/update source_files row
  -> create ingestion_jobs row with status='queued'
  -> commit the transaction
  -> publish a JetStream message containing ingestion_job_id
  -> return 202 Accepted

A dedicated ingestion worker process consumes the message:

JetStream message
  -> load ingestion_jobs/source_files from Postgres
  -> bind logging context from durable ids
  -> fetch source object from MinIO
  -> parse, chunk, embed, and mutate Qdrant points
  -> append ingestion_job_events
  -> mark ingestion_jobs succeeded/failed/cancelled
  -> ack or negatively acknowledge the message

Worker-owned ingestion chunk CRUD

For file ingestion and re-ingestion, generated chunk/point CRUD is performed by the ingestion worker, not by the FastAPI publisher/request path.

The publisher only owns the lightweight durable handoff:

• authenticate the caller and derive tenant context;
• store uploaded bytes in MinIO;
• create or update source_files;
• create ingestion_jobs(status='queued');
• commit the Postgres transaction;
• publish the JetStream message containing durable ids.

The ingestion worker owns the expensive and retryable side effects:

• parsing source files;
• chunking;
• embedding and reranking-vector generation when applicable;
• creating, updating, reordering, and soft-deleting generated Qdrant points;
• appending ingestion_job_events;
• updating ingestion job status, counters, and failure summaries.

Shared chunk/point mutation logic should live in a service/repository layer that can be reused by both direct API routes and ingestion workers. Direct /v1/points CRUD remains a synchronous API responsibility unless the operation is explicitly bulk or job-shaped. Ingestion-generated mutations are asynchronous worker responsibility because they must be safe to retry and resume by ingestion_job_id.

Keep messages small and versioned

Broker payloads carry identifiers and correlation metadata only. A typical message is:

{
  "type": "ingestion.job.created",
  "version": 1,
  "tenant_id": "...",
  "file_id": "...",
  "ingestion_job_id": "...",
  "request_id": "...",
  "api_key_id": "..."
}

Do not put raw file bytes, extracted text, chunks, embeddings, raw prompts, model outputs, or secrets in JetStream messages.

Workers must treat message metadata as routing/correlation input, not as the only authority. Before doing tenant-scoped work, workers reload the durable job and file rows from Postgres and verify the ids are consistent.

Use explicit subjects, streams, and durable consumers

Use stable, dot-separated subjects. Initial subjects:

ingestion.job.created
ingestion.job.retry_requested
maintenance.retention.requested
maintenance.erasure.requested

Create an application-owned JetStream stream for durable work messages, for example:

CHATBOT_JOBS
subjects: ingestion.job.*, maintenance.*

Workers use durable consumers or queue groups so multiple worker processes can share work without processing every message independently.

Ack messages only after the worker has persisted the resulting job state and progress events to Postgres. Redelivery must be safe: ingestion processors should be idempotent by ingestion_job_id and Qdrant point ids/upsert semantics.

Run subscribers as worker processes, not FastAPI side effects

FastAPI may create a JetStream publisher/client during application lifespan so routes can publish job notifications after durable state is committed.

Subscribers should run in separate worker process entrypoints. Do not start a subscriber inside every FastAPI web worker by default: with multiple Uvicorn/Gunicorn workers, each process has its own lifespan and could create surprising duplicate consumers or resource pressure.

Worker processes create their own application-lifetime resources at startup:

• SQLAlchemy engine/sessionmaker;
• MinIO/S3 object-storage client;
• Qdrant client;
• NATS/JetStream client;
• observability/logging clients;
• ingestion model clients.

Each message gets its own request/job-lifetime SQLAlchemy AsyncSession and explicit transaction boundaries, following ADR-0012.

Handle commit/publish consistency deliberately

The initial implementation may publish the JetStream message after the Postgres transaction commits. This avoids workers observing uncommitted jobs, but it means a process crash between commit and publish can leave a job in queued state with no broker message.

To handle that failure mode, workers or a small scheduler should periodically scan Postgres for queued/stale ingestion jobs and publish or process them again. This keeps Postgres authoritative and makes JetStream a wakeup/delivery mechanism rather than the only source of pending work.

If stricter guarantees become necessary, introduce a transactional outbox table in Postgres via a future ADR/migration:

same DB transaction:
  insert/update ingestion_jobs
  insert outbox_events

publisher process:
  read unpublished outbox_events
  publish to JetStream
  mark outbox event published

Do not add DDL at FastAPI startup; all outbox schema changes must go through Alembic as required by ADR-0009.

Configure retries, dead letters, and observability

JetStream consumers should configure explicit acknowledgement, redelivery, and maximum delivery behavior. Poison messages or repeatedly failing jobs should end as durable Postgres failures with an ingestion_job_events error entry, not an infinite invisible retry loop.

At worker ingress, bind structured logging context from durable identifiers:

• tenant_id;
• request_id;
• api_key_id;
• file_id;
• ingestion_job_id;
• JetStream stream/subject/sequence when available.

Use stable event names such as:

• ingestion.job.dispatched;
• ingestion.job.received;
• ingestion.job.completed;
• ingestion.job.failed;
• broker.publish.failed;
• broker.message.redelivered.

Do not queue chat runs through JetStream yet

ADR-0007 and ADR-0008 intentionally reject queueing concurrent chat runs per thread: one run may be in flight, and concurrent runs return 409 Conflict. This ADR does not change that decision.

A future ADR may introduce broker-backed work for chat-adjacent tasks such as summarization, retention, erasure, or offline evaluation, but synchronous chat run execution remains owned by the FastAPI/LangGraph boundary unless superseded.

Consequences

Positive

• File ingestion can move out of HTTP request handling into scalable worker processes without changing the REST contract from ADR-0008.
• JetStream gives durable delivery, ack/redelivery, and worker-group semantics without adopting Kafka-level operational complexity.
• Postgres remains the auditable source of truth for jobs, progress, tenant ownership, and failure state.
• MinIO, Postgres, Qdrant, and the broker each have distinct responsibilities: bytes, metadata/state, vector search, and work delivery.
• Separate worker processes avoid surprising subscriptions in every FastAPI web worker and make GPU/model-heavy ingestion resources easier to size.

Negative

• Adds another external service to run, secure, monitor, and back up where stream durability is required.
• Developers must reason about redelivery and idempotency; handlers may run more than once for the same ingestion_job_id.
• Publishing after commit has a crash window unless a stale-job scanner or future transactional outbox is implemented.
• NATS/JetStream stream, consumer, retention, and dead-letter configuration must be managed explicitly per environment.
• The app now needs worker deployment/runbook conventions in addition to the FastAPI web process.

Alternatives Considered

• NATS core pub/sub only: rejected. Core pub/sub is useful for ephemeral events, but ingestion dispatch needs durable delivery and acknowledgement. JetStream is the NATS feature that provides those semantics.
• Redis Streams: rejected for this decision. Redis Streams would work for an MVP and has simple local deployment, but NATS JetStream is a cleaner dedicated broker for durable pub/sub and worker coordination without also mixing cache responsibilities into the same service.
• RabbitMQ: rejected for now. It is mature for work queues and routing, but the project does not currently need its exchange/routing complexity, and NATS JetStream keeps the broker footprint lighter.
• Kafka: rejected. Kafka is an excellent durable event log at high scale, but its operational overhead is unnecessary for this service's initial ingestion and maintenance jobs.
• FastAPI BackgroundTasks or in-process asyncio tasks: rejected as the durable design. They are simple, but work can be lost on process restart and they do not coordinate multiple workers cleanly.
• Database polling only: rejected as the main dispatch mechanism. Polling Postgres for queued jobs is a useful repair path, but relying on polling alone adds latency and unnecessary database load once a broker is available.
• Use the broker as the source of truth for job state: rejected. Broker state is delivery state. Application job state, progress, audit, and tenant ownership remain in Postgres.