chatbot_v3/docs/adr/0013-s3-compatible-object-storage-with-minio.md

0013. S3-compatible object storage with MinIO

Status

Proposed

Context

ADR-0008 defines source-file upload endpoints and returns a job-shaped ingestion response. ADR-0009 models source_files and leaves room for a storage_uri where original files are retained. The service now needs a clear place to store uploaded file bytes and any derived ingestion artifacts without putting large binary/text payloads in Postgres or broker messages.

The storage decision has to satisfy several constraints:

• Uploaded files may be larger than is appropriate for Postgres rows.
• Ingestion workers need to fetch the exact file bytes after the HTTP request has completed.
• Postgres should remain the source of truth for tenant metadata, file metadata, ingestion job state, audit records, and retention decisions.
• Broker messages should carry durable identifiers, not raw file content.
• Local/self-hosted deployment should stay practical and avoid coupling this app to Langfuse's internal object-storage services.

Decision

Use MinIO as the application's S3-compatible object store for uploaded source files and derived ingestion blobs.

MinIO stores bytes. Postgres stores metadata and authoritative state.

Store source-file bytes in MinIO

When POST /v1/files receives an upload, the service stores the raw file in an application-owned MinIO bucket before or within the same application flow that creates the source_files and ingestion_jobs rows.

source_files keeps the durable pointer and metadata, including fields already sketched in ADR-0009:

• tenant_id;
• domain;
• source_filename;
• source_type;
• content_sha256;
• byte_size;
• storage_uri or equivalent bucket/key fields;
• lifecycle status and audit timestamps.

Object keys are internal identifiers, not trusted user filenames. Prefer a shape like:

tenants/{tenant_id}/source-files/{file_id}/original

or, when content-addressing is useful:

tenants/{tenant_id}/source-files/{file_id}/{content_sha256}

The original filename is stored as metadata in Postgres, not used as the object key authority.

Keep MinIO separate from authoritative state

MinIO is not the source of truth for:

• tenant identity or authorization;
• file ownership;
• ingestion job status;
• ingestion progress events;
• audit logs;
• LangGraph checkpoints or Store memory;
• Qdrant point state;
• broker delivery state.

Workers always load source_files and ingestion_jobs from Postgres by id, then use the stored object pointer to fetch bytes from MinIO.

Use MinIO for retained blobs, not queue payloads

Broker messages reference ids such as file_id and ingestion_job_id; they do not contain raw file bytes, extracted text, chunks, embeddings, or large parser outputs.

MinIO may also store derived blobs when retention policy allows it, for example:

• converted .doc -> .docx outputs from ADR-0004;
• extracted text snapshots used for debugging failed ingestion;
• parser diagnostics or quarantined uploads;
• exported artifacts generated by maintenance jobs.

Those derived objects still need Postgres metadata if they are user-visible, auditable, or subject to retention/erasure policy.

Use application-owned buckets and credentials

The app must not casually reuse Langfuse's internal MinIO bucket, credentials, or lifecycle. Langfuse object storage belongs to the Langfuse stack from ADR-0010; this service needs its own application bucket(s) and credentials.

A local deployment may run MinIO in the same Docker environment, but with a separate bucket such as:

chatbot-source-files

Production may use MinIO or another S3-compatible service behind the same object storage interface, provided it preserves tenant isolation, encryption, backup, and retention requirements.

Manage the client as an application-lifetime resource

The object-storage client is an application-lifetime external client under ADR-0012. It is created by the FastAPI lifespan or worker process startup and closed on shutdown when the client library requires explicit cleanup.

Lower-level ingestion services receive an object-storage client or interface explicitly. They do not import a module-level mutable MinIO singleton.

For async code, object-storage operations must not block the event loop. Use an async-capable S3 client, or isolate synchronous SDK calls in the appropriate threadpool boundary.

Enforce privacy, retention, and integrity rules

• Compute and store content_sha256 for uploaded bytes.
• Enforce allowed content types/extensions and maximum file size before retaining files, as required by ADR-0008.
• Prefer private buckets; generate short-lived presigned URLs only for explicit internal workflows that need them.
• Apply server-side encryption and backups according to deployment policy.
• Deletion and tenant-erasure workflows must delete both Postgres metadata and MinIO objects, while preserving audit records according to tenant/legal retention requirements.

Consequences

Positive

• Large uploaded files and derived blobs stay out of Postgres rows and broker payloads.
• Ingestion can run in separate workers after the HTTP request finishes because workers can fetch the retained object by durable id.
• The service keeps S3-compatible portability while using MinIO for local and self-hosted deployments.
• source_files.storage_uri from ADR-0009 now has a concrete storage backend.
• Separating app MinIO from Langfuse MinIO avoids accidental coupling to Langfuse's internal lifecycle and credentials.

Negative

• Adds another external dependency to configure, secure, back up, and monitor.
• Retention and tenant erasure now have to coordinate Postgres rows, Qdrant points, and MinIO objects.
• Upload flows need cleanup handling for partial failures, such as object upload succeeding but the Postgres transaction failing.
• Async FastAPI handlers need care if the chosen S3/MinIO client is synchronous.
• Object keys and bucket policies become part of the security boundary; mistakes can expose cross-tenant files.

Alternatives Considered

• Store uploaded files directly in Postgres: rejected. It simplifies transactional consistency, but bloats the relational database, complicates backups, and mixes large binary/blob storage with metadata and audit queries.
• Store only extracted chunks in Qdrant and discard originals immediately: rejected as the default. It makes failed-ingestion debugging, reprocessing, parser upgrades, and retention review harder. Tenants may still configure shorter retention later.
• Use the broker message as file transport: rejected. Brokers should carry small work identifiers and metadata; raw files, extracted text, chunks, and embeddings belong in object storage or the appropriate durable store.
• Reuse Langfuse's internal MinIO service/bucket: rejected. Langfuse object storage is part of the observability stack, not this application's file store. Sharing it would couple credentials, retention, backups, and operational lifecycle across unrelated systems.
• Bind directly to a cloud-only S3 provider: rejected for now. The project is self-hosted/local-development oriented, and MinIO gives an S3-compatible API while keeping deployment portable.