chatbot_v3/docs/adr/0013-s3-compatible-object-storage-with-minio.md

# 0013. S3-compatible object storage with MinIO

## Status

Proposed

## Context

ADR-0008 defines source-file upload endpoints and returns a job-shaped ingestion
response. ADR-0009 models `source_files` and leaves room for a `storage_uri`
where original files are retained. The service now needs a clear place to store
uploaded file bytes and any derived ingestion artifacts without putting large
binary/text payloads in Postgres or broker messages.

The storage decision has to satisfy several constraints:

- Uploaded files may be larger than is appropriate for Postgres rows.
- Ingestion workers need to fetch the exact file bytes after the HTTP request has
  completed.
- Postgres should remain the source of truth for tenant metadata, file metadata,
  ingestion job state, audit records, and retention decisions.
- Broker messages should carry durable identifiers, not raw file content.
- Local/self-hosted deployment should stay practical and avoid coupling this app
  to Langfuse's internal object-storage services.

## Decision

Use **MinIO** as the application's S3-compatible object store for uploaded source
files and derived ingestion blobs.

MinIO stores bytes. Postgres stores metadata and authoritative state.

### Store source-file bytes in MinIO

When `POST /v1/files` receives an upload, the service stores the raw file in an
application-owned MinIO bucket before or within the same application flow that
creates the `source_files` and `ingestion_jobs` rows.

`source_files` keeps the durable pointer and metadata, including fields already
sketched in ADR-0009:

- `tenant_id`;
- `domain`;
- `source_filename`;
- `source_type`;
- `content_sha256`;
- `byte_size`;
- `storage_uri` or equivalent bucket/key fields;
- lifecycle status and audit timestamps.

Object keys are internal identifiers, not trusted user filenames. Prefer a shape
like:

```text
tenants/{tenant_id}/source-files/{file_id}/original
```

or, when content-addressing is useful:

```text
tenants/{tenant_id}/source-files/{file_id}/{content_sha256}
```

The original filename is stored as metadata in Postgres, not used as the object
key authority.

### Keep MinIO separate from authoritative state

MinIO is not the source of truth for:

- tenant identity or authorization;
- file ownership;
- ingestion job status;
- ingestion progress events;
- audit logs;
- LangGraph checkpoints or Store memory;
- Qdrant point state;
- broker delivery state.

Workers always load `source_files` and `ingestion_jobs` from Postgres by id, then
use the stored object pointer to fetch bytes from MinIO.

### Use MinIO for retained blobs, not queue payloads

Broker messages reference ids such as `file_id` and `ingestion_job_id`; they do
not contain raw file bytes, extracted text, chunks, embeddings, or large parser
outputs.

MinIO may also store derived blobs when retention policy allows it, for example:

- converted `.doc` -> `.docx` outputs from ADR-0004;
- extracted text snapshots used for debugging failed ingestion;
- parser diagnostics or quarantined uploads;
- exported artifacts generated by maintenance jobs.

Those derived objects still need Postgres metadata if they are user-visible,
auditable, or subject to retention/erasure policy.

### Use application-owned buckets and credentials

The app must not casually reuse Langfuse's internal MinIO bucket, credentials, or
lifecycle. Langfuse object storage belongs to the Langfuse stack from ADR-0010;
this service needs its own application bucket(s) and credentials.

A local deployment may run MinIO in the same Docker environment, but with a
separate bucket such as:

```text
chatbot-source-files
```

Production may use MinIO or another S3-compatible service behind the same object
storage interface, provided it preserves tenant isolation, encryption, backup,
and retention requirements.

### Manage the client as an application-lifetime resource

The object-storage client is an application-lifetime external client under
ADR-0012. It is created by the FastAPI lifespan or worker process startup and
closed on shutdown when the client library requires explicit cleanup.

Lower-level ingestion services receive an object-storage client or interface
explicitly. They do not import a module-level mutable MinIO singleton.

For async code, object-storage operations must not block the event loop. Use an
async-capable S3 client, or isolate synchronous SDK calls in the appropriate
threadpool boundary.

### Enforce privacy, retention, and integrity rules

- Compute and store `content_sha256` for uploaded bytes.
- Enforce allowed content types/extensions and maximum file size before retaining
  files, as required by ADR-0008.
- Prefer private buckets; generate short-lived presigned URLs only for explicit
  internal workflows that need them.
- Apply server-side encryption and backups according to deployment policy.
- Deletion and tenant-erasure workflows must delete both Postgres metadata and
  MinIO objects, while preserving audit records according to tenant/legal
  retention requirements.

## Consequences

### Positive
- Large uploaded files and derived blobs stay out of Postgres rows and broker
  payloads.
- Ingestion can run in separate workers after the HTTP request finishes because
  workers can fetch the retained object by durable id.
- The service keeps S3-compatible portability while using MinIO for local and
  self-hosted deployments.
- `source_files.storage_uri` from ADR-0009 now has a concrete storage backend.
- Separating app MinIO from Langfuse MinIO avoids accidental coupling to
  Langfuse's internal lifecycle and credentials.

### Negative
- Adds another external dependency to configure, secure, back up, and monitor.
- Retention and tenant erasure now have to coordinate Postgres rows, Qdrant
  points, and MinIO objects.
- Upload flows need cleanup handling for partial failures, such as object upload
  succeeding but the Postgres transaction failing.
- Async FastAPI handlers need care if the chosen S3/MinIO client is synchronous.
- Object keys and bucket policies become part of the security boundary; mistakes
  can expose cross-tenant files.

## Alternatives Considered

- **Store uploaded files directly in Postgres**: rejected. It simplifies
  transactional consistency, but bloats the relational database, complicates
  backups, and mixes large binary/blob storage with metadata and audit queries.
- **Store only extracted chunks in Qdrant and discard originals immediately**:
  rejected as the default. It makes failed-ingestion debugging, reprocessing,
  parser upgrades, and retention review harder. Tenants may still configure
  shorter retention later.
- **Use the broker message as file transport**: rejected. Brokers should carry
  small work identifiers and metadata; raw files, extracted text, chunks, and
  embeddings belong in object storage or the appropriate durable store.
- **Reuse Langfuse's internal MinIO service/bucket**: rejected. Langfuse object
  storage is part of the observability stack, not this application's file store.
  Sharing it would couple credentials, retention, backups, and operational
  lifecycle across unrelated systems.
- **Bind directly to a cloud-only S3 provider**: rejected for now. The project is
  self-hosted/local-development oriented, and MinIO gives an S3-compatible API
  while keeping deployment portable.