docs(architecture): define application resource ownership
Why: - Establish explicit ownership and lifetime rules for application, request/job, and operation-scoped resources. Changes: - Define FastAPI lifespan ownership for engines, clients, graphs, and pools. - Define dependency-managed sessions and request context. - Require explicit transaction boundaries and dependency passing. - Prohibit shared global SQLAlchemy sessions and import-time network clients. Impact: - Application resources are created and closed by their process owner. - Request and job resources must not be shared across concurrent units of work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
182
docs/adr/0013-s3-compatible-object-storage-with-minio.md
Normal file
182
docs/adr/0013-s3-compatible-object-storage-with-minio.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# 0013. S3-compatible object storage with MinIO
|
||||
|
||||
## Status
|
||||
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0008 defines source-file upload endpoints and returns a job-shaped ingestion
|
||||
response. ADR-0009 models `source_files` and leaves room for a `storage_uri`
|
||||
where original files are retained. The service now needs a clear place to store
|
||||
uploaded file bytes and any derived ingestion artifacts without putting large
|
||||
binary/text payloads in Postgres or broker messages.
|
||||
|
||||
The storage decision has to satisfy several constraints:
|
||||
|
||||
- Uploaded files may be larger than is appropriate for Postgres rows.
|
||||
- Ingestion workers need to fetch the exact file bytes after the HTTP request has
|
||||
completed.
|
||||
- Postgres should remain the source of truth for tenant metadata, file metadata,
|
||||
ingestion job state, audit records, and retention decisions.
|
||||
- Broker messages should carry durable identifiers, not raw file content.
|
||||
- Local/self-hosted deployment should stay practical and avoid coupling this app
|
||||
to Langfuse's internal object-storage services.
|
||||
|
||||
## Decision
|
||||
|
||||
Use **MinIO** as the application's S3-compatible object store for uploaded source
|
||||
files and derived ingestion blobs.
|
||||
|
||||
MinIO stores bytes. Postgres stores metadata and authoritative state.
|
||||
|
||||
### Store source-file bytes in MinIO
|
||||
|
||||
When `POST /v1/files` receives an upload, the service stores the raw file in an
|
||||
application-owned MinIO bucket before or within the same application flow that
|
||||
creates the `source_files` and `ingestion_jobs` rows.
|
||||
|
||||
`source_files` keeps the durable pointer and metadata, including fields already
|
||||
sketched in ADR-0009:
|
||||
|
||||
- `tenant_id`;
|
||||
- `domain`;
|
||||
- `source_filename`;
|
||||
- `source_type`;
|
||||
- `content_sha256`;
|
||||
- `byte_size`;
|
||||
- `storage_uri` or equivalent bucket/key fields;
|
||||
- lifecycle status and audit timestamps.
|
||||
|
||||
Object keys are internal identifiers, not trusted user filenames. Prefer a shape
|
||||
like:
|
||||
|
||||
```text
|
||||
tenants/{tenant_id}/source-files/{file_id}/original
|
||||
```
|
||||
|
||||
or, when content-addressing is useful:
|
||||
|
||||
```text
|
||||
tenants/{tenant_id}/source-files/{file_id}/{content_sha256}
|
||||
```
|
||||
|
||||
The original filename is stored as metadata in Postgres, not used as the object
|
||||
key authority.
|
||||
|
||||
### Keep MinIO separate from authoritative state
|
||||
|
||||
MinIO is not the source of truth for:
|
||||
|
||||
- tenant identity or authorization;
|
||||
- file ownership;
|
||||
- ingestion job status;
|
||||
- ingestion progress events;
|
||||
- audit logs;
|
||||
- LangGraph checkpoints or Store memory;
|
||||
- Qdrant point state;
|
||||
- broker delivery state.
|
||||
|
||||
Workers always load `source_files` and `ingestion_jobs` from Postgres by id, then
|
||||
use the stored object pointer to fetch bytes from MinIO.
|
||||
|
||||
### Use MinIO for retained blobs, not queue payloads
|
||||
|
||||
Broker messages reference ids such as `file_id` and `ingestion_job_id`; they do
|
||||
not contain raw file bytes, extracted text, chunks, embeddings, or large parser
|
||||
outputs.
|
||||
|
||||
MinIO may also store derived blobs when retention policy allows it, for example:
|
||||
|
||||
- converted `.doc` -> `.docx` outputs from ADR-0004;
|
||||
- extracted text snapshots used for debugging failed ingestion;
|
||||
- parser diagnostics or quarantined uploads;
|
||||
- exported artifacts generated by maintenance jobs.
|
||||
|
||||
Those derived objects still need Postgres metadata if they are user-visible,
|
||||
auditable, or subject to retention/erasure policy.
|
||||
|
||||
### Use application-owned buckets and credentials
|
||||
|
||||
The app must not casually reuse Langfuse's internal MinIO bucket, credentials, or
|
||||
lifecycle. Langfuse object storage belongs to the Langfuse stack from ADR-0010;
|
||||
this service needs its own application bucket(s) and credentials.
|
||||
|
||||
A local deployment may run MinIO in the same Docker environment, but with a
|
||||
separate bucket such as:
|
||||
|
||||
```text
|
||||
chatbot-source-files
|
||||
```
|
||||
|
||||
Production may use MinIO or another S3-compatible service behind the same object
|
||||
storage interface, provided it preserves tenant isolation, encryption, backup,
|
||||
and retention requirements.
|
||||
|
||||
### Manage the client as an application-lifetime resource
|
||||
|
||||
The object-storage client is an application-lifetime external client under
|
||||
ADR-0012. It is created by the FastAPI lifespan or worker process startup and
|
||||
closed on shutdown when the client library requires explicit cleanup.
|
||||
|
||||
Lower-level ingestion services receive an object-storage client or interface
|
||||
explicitly. They do not import a module-level mutable MinIO singleton.
|
||||
|
||||
For async code, object-storage operations must not block the event loop. Use an
|
||||
async-capable S3 client, or isolate synchronous SDK calls in the appropriate
|
||||
threadpool boundary.
|
||||
|
||||
### Enforce privacy, retention, and integrity rules
|
||||
|
||||
- Compute and store `content_sha256` for uploaded bytes.
|
||||
- Enforce allowed content types/extensions and maximum file size before retaining
|
||||
files, as required by ADR-0008.
|
||||
- Prefer private buckets; generate short-lived presigned URLs only for explicit
|
||||
internal workflows that need them.
|
||||
- Apply server-side encryption and backups according to deployment policy.
|
||||
- Deletion and tenant-erasure workflows must delete both Postgres metadata and
|
||||
MinIO objects, while preserving audit records according to tenant/legal
|
||||
retention requirements.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Large uploaded files and derived blobs stay out of Postgres rows and broker
|
||||
payloads.
|
||||
- Ingestion can run in separate workers after the HTTP request finishes because
|
||||
workers can fetch the retained object by durable id.
|
||||
- The service keeps S3-compatible portability while using MinIO for local and
|
||||
self-hosted deployments.
|
||||
- `source_files.storage_uri` from ADR-0009 now has a concrete storage backend.
|
||||
- Separating app MinIO from Langfuse MinIO avoids accidental coupling to
|
||||
Langfuse's internal lifecycle and credentials.
|
||||
|
||||
### Negative
|
||||
- Adds another external dependency to configure, secure, back up, and monitor.
|
||||
- Retention and tenant erasure now have to coordinate Postgres rows, Qdrant
|
||||
points, and MinIO objects.
|
||||
- Upload flows need cleanup handling for partial failures, such as object upload
|
||||
succeeding but the Postgres transaction failing.
|
||||
- Async FastAPI handlers need care if the chosen S3/MinIO client is synchronous.
|
||||
- Object keys and bucket policies become part of the security boundary; mistakes
|
||||
can expose cross-tenant files.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- **Store uploaded files directly in Postgres**: rejected. It simplifies
|
||||
transactional consistency, but bloats the relational database, complicates
|
||||
backups, and mixes large binary/blob storage with metadata and audit queries.
|
||||
- **Store only extracted chunks in Qdrant and discard originals immediately**:
|
||||
rejected as the default. It makes failed-ingestion debugging, reprocessing,
|
||||
parser upgrades, and retention review harder. Tenants may still configure
|
||||
shorter retention later.
|
||||
- **Use the broker message as file transport**: rejected. Brokers should carry
|
||||
small work identifiers and metadata; raw files, extracted text, chunks, and
|
||||
embeddings belong in object storage or the appropriate durable store.
|
||||
- **Reuse Langfuse's internal MinIO service/bucket**: rejected. Langfuse object
|
||||
storage is part of the observability stack, not this application's file store.
|
||||
Sharing it would couple credentials, retention, backups, and operational
|
||||
lifecycle across unrelated systems.
|
||||
- **Bind directly to a cloud-only S3 provider**: rejected for now. The project is
|
||||
self-hosted/local-development oriented, and MinIO gives an S3-compatible API
|
||||
while keeping deployment portable.
|
||||
Reference in New Issue
Block a user