diff --git a/docs/adr/0010-langfuse-observability-and-prompt-management.md b/docs/adr/0010-langfuse-observability-and-prompt-management.md new file mode 100644 index 0000000..8b53bfb --- /dev/null +++ b/docs/adr/0010-langfuse-observability-and-prompt-management.md @@ -0,0 +1,455 @@ +# 0010. Langfuse observability, prompt management, and evaluation workflow + +## Status + +Proposed + +## Context + +The chatbot now has architectural decisions for the LangGraph conversation +([0006](0006-conversational-agent-graph.md)), LangGraph thread persistence +([0007](0007-agent-persistence-threads-and-memory.md)), the FastAPI REST +boundary ([0008](0008-rest-api-and-fastapi-boundary.md)), and Postgres tables +for tenants, API keys, ingestion jobs, graph runs, LLM calls, payload retention, +and feedback ([0009](0009-postgres-sqlalchemy-alembic-schema.md)). Those +systems record *what happened* internally, but they do not provide the day-to-day +LLM observability and prompt iteration workflow needed to run an insurance +chatbot safely. + +We need to answer questions like: + +- Which graph node failed — `triage`, `retrieve`, `generate`, or `verify`? +- Which retrieved chunks did the model see before it answered or escalated? +- Which prompt version produced a bad answer? +- Are prompt changes improving groundedness or just reducing escalations? +- Which tenants, domains, or routes are most expensive? +- Which failures should be fixed by better retrieval, better source content, + better prompts, or stricter handoff policy? + +Langfuse is a good fit for the observability and prompt/evaluation layer. It +provides traces, nested observations, generation token/cost tracking, prompt +version management, scores/user feedback, datasets, experiments, evaluators, +and annotation workflows. It should not replace the application database: +Postgres remains the source of truth for API-key authentication, tenant/audit +records, ingestion jobs, graph run rows, billing/cost ledgers, and retention +policy enforcement. + +Current Langfuse guidance relevant to this project: + +- Use the current Langfuse Python v3 SDK. +- For LangGraph/LangChain, use `from langfuse.langchain import CallbackHandler` + and pass the handler in the graph invocation config. +- Use one trace per user-facing chatbot run. +- Use stable, low-cardinality trace and observation names. +- Model calls should appear as generation observations so model, token, latency, + and cost information can attach correctly. +- Conversation grouping should use Langfuse sessions; in this service, + LangGraph `thread_id` is the Langfuse `session_id`. +- Prompt management uses prompt versions and labels (for example `development`, + `staging`, `production`, tenant-specific labels, and automatically maintained + `latest`). Labels can represent environments, tenants, or experiments. +- Scores represent explicit feedback and implicit/evaluator signals. +- Sensitive data should be masked before export, especially in insurance. + +## Decision + +### Langfuse is the observability plane, not the transactional database + +Adopt Langfuse for: + +- traces and nested observations; +- prompt versions, variables, labels, diffs, and trace links; +- user feedback and evaluator scores; +- datasets and experiments; +- annotation queues and systematic error analysis; +- interactive debugging and dashboards. + +Keep Postgres ([0009](0009-postgres-sqlalchemy-alembic-schema.md)) as the +internal system of record for: + +- tenants and API keys; +- request and point audit records; +- source files and ingestion jobs; +- `graph_runs` and `run_feedback`; +- `llm_calls`, `llm_pricing`, and optional `llm_call_payloads`; +- tenant-specific retention, erasure, and billing/reporting queries. + +The two systems are correlated with stable identifiers: + +| Identifier | Origin | Use | +|---|---|---| +| `request_id` | FastAPI middleware | Correlate API logs, Postgres audit rows, Langfuse trace metadata. | +| `tenant_id` | API-key dependency | Tenant filtering/cost attribution; never client-supplied. | +| `thread_id` | REST path / LangGraph config | LangGraph thread id and Langfuse `session_id`. | +| `run_id` | `graph_runs.id` | One `POST /v1/threads/{thread_id}/runs` execution. | +| `llm_call_id` | `llm_calls.id` | Optional correlation from a Langfuse generation to Postgres usage ledger. | +| `langfuse_trace_id` | Langfuse | Stored on `graph_runs.metadata` initially, or promoted to a typed column later if frequently queried. | +| `langfuse_observation_id` | Langfuse | Correlated by putting `llm_call_id` in Langfuse observation metadata; add a typed Postgres column later only if needed. | + +### SDK and invocation pattern + +Use the current Langfuse Python SDK v3 integration for LangGraph/LangChain: + +```python +from langfuse import get_client, propagate_attributes +from langfuse.langchain import CallbackHandler + +langfuse = get_client() +trace_tags = [environment, "chatbot", "threads:runs"] +trace_metadata = { + "request_id": request_id, + "tenant_id": str(tenant_id), + "api_key_id": str(api_key_id), + "thread_id": thread_id, + "run_id": str(run_id), + "graph_version": graph_version, + "retrieval_config_version": retrieval_config_version, + "prompt_label": prompt_label, +} + +with propagate_attributes( + trace_name="chat-run", + user_id=external_user_id, + session_id=thread_id, + tags=trace_tags, + metadata=trace_metadata, +): + langfuse_handler = CallbackHandler() + result = graph.invoke( + graph_input, + config={ + "callbacks": [langfuse_handler], + "configurable": { + "thread_id": thread_id, + "tenant_id": str(tenant_id), + "user_id": external_user_id, + }, + }, + ) +``` + +If the implementation sets Langfuse attributes through LangChain/LangGraph +`config["metadata"]` instead of or in addition to `propagate_attributes`, use the +current integration keys `langfuse_user_id`, `langfuse_session_id`, and +`langfuse_tags` for user, session, and tag propagation. + +For non-LangChain/LangGraph work that should appear in the same trace — FastAPI +request spans, Qdrant retrieval, custom reranking metadata, or explicit audit +steps — use Langfuse/OpenTelemetry spans around that work rather than relying +only on automatic LLM callbacks. + +Short-lived scripts and tests call `langfuse.flush()` or `langfuse.shutdown()` +before exit. Long-lived FastAPI workers flush during graceful shutdown. + +### Trace shape + +Create one Langfuse trace per `POST /v1/threads/{thread_id}/runs` call. + +Trace-level fields: + +| Field | Value | +|---|---| +| name | `chat-run` | +| input | The latest user message, after request-level redaction. | +| output | Final assistant answer, clarifying question, or escalation message. | +| user_id | External `user_id` supplied by the main backend. | +| session_id | LangGraph `thread_id`. | +| tags | Environment, feature (`chatbot`), route (`threads:runs`), tenant segment if safe, terminal status. | +| metadata | `request_id`, `tenant_id`, `api_key_id`, `thread_id`, `run_id`, graph version, retrieval config version, prompt label. | + +Observation hierarchy mirrors ADR-0006's graph instead of producing a flat list: + +| Observation | Type | Contents | +|---|---|---| +| `load-memory` | span | Store lookup latency and memory keys, not sensitive values. | +| `triage` | generation | Intent classification prompt/output. | +| `contextualize` | generation | Query rewrite input/output. | +| `retrieve` | retriever/span | Qdrant collection, vector names, filters, top-k, returned point/chunk IDs, scores, and source metadata. Avoid raw chunk text unless redacted. | +| `grade` | generation | Relevance verdicts over retrieved chunks. | +| `clarify` | generation | Grounded clarification question when used. | +| `generate` | generation | Final answer generation with citation IDs. | +| `verify` | generation | Groundedness/citation verification result. | +| `escalate` | span | Escalation reason and handoff context summary. | +| `write-memory` | span | Memory write keys and outcome, not sensitive facts. | + +Observation names stay stable and low-cardinality. Dynamic values such as tenant +slug, file id, model name, retry number, point id, and user text go in metadata +or input/output fields, not in observation names. + +Every generation should include model name, input tokens, output tokens, +latency, and cost when available. Langfuse provides the interactive view; +`llm_calls` in Postgres remains the billing/audit ledger. + +### Prompt management and tenant-specific variants + +Manage prompts in Langfuse for graph nodes whose behavior will be tuned over +time: + +- `insurance-chatbot/triage` +- `insurance-chatbot/contextualize` +- `insurance-chatbot/grade-retrieval` +- `insurance-chatbot/clarify` +- `insurance-chatbot/generate-answer` +- `insurance-chatbot/verify-grounding` +- `insurance-chatbot/summarize-thread` +- `insurance-chatbot/extract-memory` + +Prompt text, versions, diffs, labels, and rollback live in Langfuse. Tenant +identity and the policy that chooses which prompt label/name to fetch live in the +application and, where tenant-specific configuration is needed, Postgres +`tenants.settings`. + +Prompt names are stable and low-cardinality. The initial tenant-customization +model uses shared prompt names plus tenant-specific labels: + +| Label | Use | +|---|---| +| `development` | Local/dev experiments. | +| `staging` | Pre-production validation and experiments. | +| `production` | Default live traffic prompt when no tenant-specific override exists. | +| `tenant-{tenant_slug}-{environment}` | Tenant-specific prompt version for one environment, for example `tenant-acme-production`. | +| `latest` | Automatically points to newest version; useful for discovery, not production runtime. | + +Langfuse labels can represent environments, tenants, or experiments. The service +therefore uses labels to support different live prompts per tenant without +creating an application-owned prompt-version table. + +Tenant-specific prompt resolution is handled in the application with an explicit +fallback order: + +1. derive `tenant_slug` from the authenticated `AuthContext`, never from a + request body/query parameter; +2. derive the environment label from trusted configuration, for example + `production` or `staging`; +3. try the tenant/environment label, for example `tenant-acme-production`; +4. fall back to the environment label, for example `production`; +5. fail closed or use a code-packaged emergency prompt only if the configured + prompt cannot be loaded. + +Indicative resolver shape: + +```python +prompt_name = "insurance-chatbot/generate-answer" +environment_label = settings.langfuse_prompt_label # e.g. "production" +tenant_label = f"tenant-{auth.tenant_slug}-{environment_label}" + +prompt = prompt_resolver.get_with_fallback( + name=prompt_name, + labels=[tenant_label, environment_label], +) +compiled_prompt = prompt.compile( + language=language, + user_message=user_message, + conversation_summary=conversation_summary, + retrieved_chunks=retrieved_chunks, + tenant_policy=tenant_policy, + handoff_policy=handoff_policy, +) +``` + +The exact SDK call and not-found error handling should follow the current +Langfuse Python SDK, but the architectural rule is stable: select the tenant +label from trusted server-side tenant context, not from caller input. + +This keeps prompt content, prompt versions, prompt diffs, and label movement in +Langfuse while keeping tenant identity and resolver policy in the application. +If tenant-specific prompts become large and independently owned, an acceptable +future variant is tenant-namespaced prompt names such as +`insurance-chatbot/tenants/acme/generate-answer` with ordinary `production` and +`staging` labels. The initial design prefers shared prompt names plus +tenant-specific labels because it keeps graph-node prompts easier to compare +across tenants. + +Postgres may store prompt resolver configuration in `tenants.settings`, for +example an enabled flag, preferred prompt namespace, explicit label override, +default language, tenant policy reference, or handoff policy reference. Postgres +must not become the source of truth for prompt text/version history; that +belongs in Langfuse. + +Prompt variables carry dynamic context, for example: + +- `{{language}}` +- `{{user_message}}` +- `{{conversation_summary}}` +- `{{retrieved_chunks}}` +- `{{tenant_policy}}` +- `{{handoff_policy}}` +- `{{citation_contract}}` +- `{{memory_profile}}` + +Prompt versions are linked to traces so failures can be traced back to the exact +prompt version that produced them. Rollback is performed by moving the relevant +label — `production` for the default prompt, or `tenant-acme-production` for one +tenant's override — back to a known-good version, not by redeploying code. + +Prompt changes are evaluated before promotion using Langfuse datasets and +experiments. A prompt version should not be promoted to `production` or to a +tenant-specific production label only because one manual test looked good. + +### Privacy, redaction, and retention + +Insurance conversations can contain PII, policy details, claim information, and +financial/health-adjacent facts. Langfuse is therefore sensitive infrastructure. + +Rules: + +- Mask/redact before export from the application process. Prefer current SDK + masking support such as `mask_otel_spans` where applicable, plus deterministic + field/path redaction for known sensitive fields. +- Never put API keys, database URLs, provider secrets, raw auth headers, or + unredacted request bodies in trace metadata. +- Do not trace raw uploaded files or full document text. +- Retrieval observations should prefer point/chunk IDs, file IDs, scores, + titles, and short redacted snippets over full retrieved chunks. +- Raw LLM input/output retention is governed by ADR-0009's + `llm_call_payloads` tenant policy, not by accidental Langfuse trace capture. +- Langfuse project retention/deletion settings must align with insurance and + tenant contractual retention requirements. +- If self-hosted Langfuse is used later, server-side ingestion masking can add + defense in depth, but it does not replace client-side masking because data + can still reach pre-masking ingestion infrastructure. + +### Feedback and scores + +Record explicit and implicit feedback as Langfuse scores while mirroring durable +feedback to Postgres `run_feedback` when the application needs it. + +Explicit feedback: + +| Signal | Langfuse score name | Notes | +|---|---|---| +| Thumbs up/down | `user-thumbs` | Boolean score, attached to the trace/run. | +| Optional comment | score comment/metadata | Also stored in `run_feedback.comment` if enabled. | +| Report bad answer | `user-report` | Useful for annotation queues and support review. | + +Implicit feedback: + +| Signal | Score/tag | Meaning | +|---|---|---| +| Handoff triggered | `handoff-triggered` | The run ended in `status=escalate`. | +| Regeneration/retry | `user-retry` or metadata | Negative/diagnostic signal. | +| Citation opened | `citation-opened` | User inspected a cited source. | +| Answer copied | `answer-copied` | Weak positive signal if frontend exposes it. | + +The FastAPI response for a run should include enough trace/run identity for the +backend or frontend to submit feedback later, but no Langfuse secret key is ever +sent to a browser. Browser-side feedback, if added, may use only a Langfuse +public key or, preferably, route feedback through the backend. + +### Evaluation and quality workflow + +Use Langfuse beyond basic tracing: + +1. **Datasets** — maintain representative insurance questions, expected + citation behavior, ambiguous questions, out-of-scope cases, and handoff + cases. +2. **Experiments** — compare prompt versions, retrieval parameters, reranking + thresholds, model choices, and verification thresholds before deployment. +3. **LLM-as-judge evaluators** — score groundedness, citation correctness, + ambiguity handling, escalation correctness, tone/language, and refusal + quality. Judges are aids, not absolute truth. +4. **Annotation queues** — send low-score traces, user reports, ungrounded + answers, and unnecessary escalations to human review. +5. **Error analysis** — periodically sample traces, open-code failures, cluster + failure modes, and decide whether the fix belongs in retrieval, source + content, prompt text, graph policy, or human handoff rules. +6. **Dashboards** — track per-tenant cost, per-node token spend, latency, + escalation rate, clarification rate, retrieval-insufficient rate, and + verification-failure rate. + +### Boundary with ADR-0009 Postgres tables + +Langfuse overlaps with parts of ADR-0009, especially LLM-call visibility, +feedback, and run debugging. This overlap is intentional, but the systems have +different authority. + +| Need | Langfuse | Postgres | Decision | +|---|---|---|---| +| Prompt text, versions, diffs, labels, and rollback | Yes | No | Use Langfuse as the prompt source of truth. | +| Tenant-specific prompt content | Yes | No | Use Langfuse prompt labels or, later, tenant-namespaced prompt names. | +| Tenant prompt-routing policy | Partly | Yes | Store trusted tenant config in `tenants.settings` or app config; fetch prompt content from Langfuse. | +| Interactive debugging of one bad answer | Yes | Partly | Use Langfuse trace; correlate to `graph_runs.id`. | +| Trace/session grouping | Yes | Partly | Use Langfuse `session_id=thread_id`; keep `thread_id` on `graph_runs` for ledger/audit. | +| LLM token/cost visibility | Yes | Yes | Langfuse for observability; Postgres `llm_calls`/`llm_pricing` for billing and audit. | +| Durable graph-run ledger | No | Yes | Keep `graph_runs`. Langfuse traces are not the application run table. | +| User feedback | Yes | Yes | Store scores in Langfuse and durable feedback in `run_feedback` when needed by the app. | +| Error analysis and annotation queues | Yes | Usually no | Use Langfuse; avoid building an annotation workflow in Postgres initially. | +| Datasets, experiments, and evaluators | Yes | Usually no | Use Langfuse; copy to Postgres only if a product/compliance need appears. | +| Tenants and API keys | No | Yes | Keep `tenants` and `api_keys` in Postgres. | +| API request audit logs | Not enough | Yes | Keep `api_request_logs` in Postgres. | +| Point/file mutation audit | No | Yes | Keep `point_audit_events`, `source_files`, and `ingestion_jobs` in Postgres. | +| Controlled raw/redacted payload retention | Partly | Yes | Postgres `llm_call_payloads` plus tenant policy owns retention; Langfuse must be redacted/configured to comply. | +| Legal retention and tenant erasure policy | Partly | Yes | Postgres owns application policy; Langfuse project retention/deletion settings must align with it. | + +Do not remove ADR-0009 tables because Langfuse exists. Langfuse is optimized for +observability, prompt iteration, feedback analysis, datasets, experiments, and +annotation workflows. Postgres is optimized for auth, audit, retention, billing, +and application consistency. + +## Consequences + +### Positive +- Every chat run becomes inspectable as a graph-shaped trace rather than a flat + log line or a pile of model-call rows. +- Prompt versions can be changed, compared, rolled back, and linked to the + traces they produced without redeploying application code for every prompt + edit. +- Tenant-specific prompt labels let one tenant receive a customized prompt while + other tenants continue using the default `production` version. +- Scores, datasets, experiments, and annotation queues create a path from user + feedback to systematic quality improvement. +- Per-node token/cost/latency data exposes where the graph is expensive or + slow: retrieval, generation, verification, or memory. +- Correlating Langfuse IDs with Postgres IDs preserves both workflows: + interactive debugging and durable audit/billing. + +### Negative +- Langfuse adds another service and another set of credentials, retention + policies, access controls, and operational dashboards. +- Prompt management introduces runtime dependency on prompt fetch/caching + behavior. The application needs safe fallbacks if Langfuse is temporarily + unavailable. +- Tenant-specific prompt labels add routing complexity. The resolver must avoid + caller-controlled labels and must have a clear fallback when a tenant override + does not exist. +- Observability can become a privacy leak if masking is incomplete. Insurance + content makes this a high-consequence risk. +- There is intentional duplication between Langfuse generation usage and + Postgres `llm_calls`; implementations must avoid reconciling them as if one + were authoritative for the other's purpose. +- LLM-as-judge evaluators can be wrong. They should guide review and experiments, + not replace human analysis for high-risk policy/coverage answers. + +## Alternatives Considered + +- **Use Postgres tables only for observability**: rejected. `llm_calls` and + audit tables are good for billing and compliance queries, but poor for + nested trace inspection, prompt-version attribution, annotation queues, and + experiment workflows. +- **Use Langfuse as the only usage/cost store**: rejected. Tenant billing, + erasure, audit, and internal consistency need application-owned tables with + controlled retention and foreign keys to tenants/API keys. +- **Use Langfuse instead of ADR-0009 operational tables**: rejected. Langfuse can + help with traces, prompt versions, scores, datasets, experiments, and + annotation queues, but it should not own API-key authentication, request audit, + point/file mutation audit, ingestion jobs, or the internal billing ledger. +- **Store tenant prompt text and versions in Postgres**: rejected for the initial + design. It would duplicate Langfuse prompt-management features and require the + project to build its own versioning, diffs, labels, rollback, and trace links. + Postgres stores tenant prompt-routing configuration only. +- **Keep prompts only in code**: rejected for prompts that will be tuned often. + Code-only prompts make rollback and prompt/version-to-trace analysis harder. + Small static strings and schema definitions may still live in code. +- **Fetch prompts by `latest` in production**: rejected. `latest` changes every + time a new version is created. Production uses a configured label such as + `production`; rollback is moving the label. +- **Trace full retrieved chunks and raw files for easier debugging**: rejected. + It would make debugging convenient but creates unnecessary privacy and storage + risk. IDs, scores, metadata, and redacted snippets are enough by default. +- **One Langfuse project per tenant**: rejected for the initial design. A single + project per environment with tenant metadata/tags is easier to operate and + compare. Revisit per-tenant projects only if contractual isolation requires + it. +- **Skip Langfuse scores until evals are mature**: rejected. Simple user-thumbs + and handoff-triggered scores are cheap and immediately useful for filtering + traces and building the first annotation queues. diff --git a/docs/adr/0011-python-structured-logging-with-structlog.md b/docs/adr/0011-python-structured-logging-with-structlog.md new file mode 100644 index 0000000..b316195 --- /dev/null +++ b/docs/adr/0011-python-structured-logging-with-structlog.md @@ -0,0 +1,425 @@ +# 0011. Python structured logging with structlog and request context + +## Status + +Proposed + +## Context + +The service needs application-level Python logging in addition to the durable +Postgres records from ADR-0009 and the LLM/agent traces from ADR-0010. + +Postgres audit tables answer durable business questions such as which tenant API +key called an endpoint, which point mutation was requested, which ingestion job +ran, and what usage ledger rows were produced. Langfuse answers LLM observability +questions such as which graph node or prompt version produced an answer. +Structured Python logs answer operational questions while the service is running: + +- Which request failed and where? +- Which logs belong to one FastAPI request or LangGraph run? +- Which tenant, API key, thread, run, file, point, or ingestion job was involved? +- Which dependency was slow or unavailable? +- Which fallback path or retry was used? + +FastAPI, LangGraph, SQLAlchemy, Qdrant, Langfuse, and HTTP clients can all emit +logs from asynchronous code. Since async tasks can interleave on the same event +loop, relying on process-global mutable variables is unsafe. The logging context +must be request-scoped and safe across async task switching. Python +`contextvars`, exposed through `structlog.contextvars`, provide this behavior. + +The user has used a previous `log.py` based on `logging`, `structlog`, +`logging.config.dictConfig`, `ProcessorFormatter`, JSON rendering, stdlib log +capture, and manual `ContextVar` fields such as `session_id` and `property_id`. +This service should keep the same core idea but adapt the field names to the +current architecture: + +- `thread_id` instead of `session_id` for LangGraph conversations, matching + ADR-0007 and ADR-0008; +- `tenant_id`, `tenant_slug`, `api_key_id`, and `actor_type` from `AuthContext`; +- `request_id` from FastAPI middleware; +- `run_id` from `graph_runs` for chat runs; +- `file_id`, `point_id`, and `ingestion_job_id` for ingestion and point work; +- `langfuse_trace_id` when available for cross-navigation to ADR-0010 traces. + +## Decision + +### Use structlog as the application logging interface + +Use `structlog` for application logs and integrate it with Python stdlib logging +so framework/library logs are formatted consistently. + +Application code imports loggers with: + +```python +import structlog + +logger = structlog.get_logger(__name__) +``` + +Log events use stable event names and structured fields: + +```python +logger.info( + "graph.run.completed", + status="answered", + duration_ms=duration_ms, + retrieved_chunk_count=len(retrieved_chunks), +) +``` + +Do not build log messages by interpolating operational metadata into prose. +Prefer fields over long strings because fields are queryable. + +### Emit JSON logs by default in production + +Production logs are JSON on stdout so process managers, container runtimes, and +log collectors can ingest them directly. Local development may use a colored +console renderer controlled by configuration. + +File logging is optional and mainly for local development. If enabled, it must +use explicit rotation settings such as `maxBytes` and `backupCount`. Do not rely +on a default `RotatingFileHandler` with no rotation parameters. In containerized +production, stdout/stderr collection is preferred over writing `logs/app.log` +inside the application container. + +### Configure stdlib and structlog together + +The logging setup should happen once during process startup, before the FastAPI +app begins serving requests. + +Indicative configuration shape: + +```python +import logging +import logging.config +import sys + +import structlog + + +def configure_logging(*, log_level: str, json_logs: bool) -> None: + shared_processors = [ + structlog.contextvars.merge_contextvars, + structlog.stdlib.add_log_level, + structlog.stdlib.add_logger_name, + structlog.processors.TimeStamper(fmt="iso", utc=True), + structlog.processors.StackInfoRenderer(), + ] + + structlog.configure( + processors=[ + *shared_processors, + structlog.processors.format_exc_info, + structlog.stdlib.ProcessorFormatter.wrap_for_formatter, + ], + logger_factory=structlog.stdlib.LoggerFactory(), + wrapper_class=structlog.stdlib.BoundLogger, + cache_logger_on_first_use=True, + ) + + renderer = ( + structlog.processors.JSONRenderer() + if json_logs + else structlog.dev.ConsoleRenderer(colors=True) + ) + + logging.config.dictConfig( + { + "version": 1, + "disable_existing_loggers": False, + "formatters": { + "default": { + "()": structlog.stdlib.ProcessorFormatter, + "processors": [ + structlog.stdlib.ProcessorFormatter.remove_processors_meta, + renderer, + ], + "foreign_pre_chain": [ + structlog.stdlib.ExtraAdder(), + *shared_processors, + ], + }, + }, + "handlers": { + "console": { + "class": "logging.StreamHandler", + "level": log_level, + "formatter": "default", + "stream": sys.stdout, + }, + }, + "loggers": { + "": { + "handlers": ["console"], + "level": log_level, + "propagate": False, + }, + "uvicorn": { + "handlers": ["console"], + "level": log_level, + "propagate": False, + }, + "uvicorn.access": { + "handlers": ["console"], + "level": log_level, + "propagate": False, + }, + "sqlalchemy.engine": { + "handlers": ["console"], + "level": "WARNING", + "propagate": False, + }, + "watchfiles": { + "handlers": ["console"], + "level": "INFO", + "propagate": False, + }, + }, + } + ) +``` + +Notes: + +- Use the logger name `sqlalchemy.engine`, not `sqlalchemy.engin`. +- SQL statement logging is too noisy and can leak values; keep it `WARNING` by + default in production and enable `INFO`/`DEBUG` only in controlled debugging. +- `structlog.stdlib.ExtraAdder()` keeps useful fields from stdlib log records. +- `structlog.contextvars.merge_contextvars` ensures request-bound fields appear + on both structlog and stdlib logs processed through the formatter. + +### Bind request context with contextvars + +At FastAPI ingress, clear stale context, bind request identifiers, and return the +request id to callers. This makes it possible to select all logs from one request +or one graph run even when async tasks interleave. + +Indicative middleware: + +```python +from time import perf_counter +from uuid import uuid4 + +import structlog +from fastapi import Request +from starlette.types import ASGIApp + +REQUEST_ID_HEADER = "X-Request-ID" + + +async def logging_context_middleware(request: Request, call_next: ASGIApp): + structlog.contextvars.clear_contextvars() + + request_id = request.headers.get(REQUEST_ID_HEADER) or str(uuid4()) + route = request.scope.get("route") + path_template = getattr(route, "path", request.url.path) + + structlog.contextvars.bind_contextvars( + request_id=request_id, + method=request.method, + path_template=path_template, + ) + + logger = structlog.get_logger("app.http") + started = perf_counter() + logger.info("request.started") + + try: + response = await call_next(request) + except Exception: + logger.exception( + "request.failed", + duration_ms=round((perf_counter() - started) * 1000, 2), + ) + raise + + response.headers[REQUEST_ID_HEADER] = request_id + logger.info( + "request.completed", + status_code=response.status_code, + duration_ms=round((perf_counter() - started) * 1000, 2), + ) + return response +``` + +After API-key authentication succeeds, the auth dependency or route handler binds +trusted tenant/auth fields: + +```python +structlog.contextvars.bind_contextvars( + tenant_id=str(auth.tenant_id), + tenant_slug=auth.tenant_slug, + api_key_id=str(auth.api_key_id), + actor_type=auth.actor_type, +) +``` + +Route handlers bind route-specific fields when they become known: + +```python +structlog.contextvars.bind_contextvars( + external_user_id=body.user_id, + thread_id=thread_id, + run_id=str(run_id), +) +``` + +Use these canonical context keys: + +| Field | Source | Notes | +|---|---|---| +| `request_id` | FastAPI middleware | Primary log correlation id; also appears in ADR-0008/0009 records. | +| `tenant_id` | `AuthContext` | Trusted server-side tenant id; never request body/query. | +| `tenant_slug` | `AuthContext` | Useful for filtering; avoid if contractual policy treats it as sensitive. | +| `api_key_id` | `AuthContext` | Non-secret id only. Never log raw API keys or auth headers. | +| `actor_type` | `AuthContext` | `backend`, `admin`, or `worker`. | +| `external_user_id` | Main backend | May be high-cardinality; acceptable in logs, not metrics labels. | +| `thread_id` | REST path | LangGraph thread id. | +| `run_id` | `graph_runs.id` | One chat run. | +| `ingestion_job_id` | `ingestion_jobs.id` | File ingestion correlation. | +| `file_id` | `source_files.id` | Source-file correlation. | +| `point_id` | Qdrant point id | Point mutation/read correlation. | +| `langfuse_trace_id` | Langfuse | Cross-link to ADR-0010 trace when available. | + +Use `structlog.contextvars.clear_contextvars()` at request/task ingress to avoid +leaking a previous request's context into reused workers. + +### Bind context explicitly for jobs and background work + +Context variables work across normal async task switching, but background jobs, +worker processes, scheduled jobs, and threadpool work should bind context at +their own entry point from durable identifiers. + +Examples: + +- ingestion worker binds `tenant_id`, `ingestion_job_id`, `file_id`, and + `request_id` if inherited from the upload request; +- LangGraph run execution binds `thread_id`, `run_id`, `tenant_id`, and + `external_user_id` before invoking the graph; +- point batch workers bind `tenant_id`, `api_request_log_id`, and operation + metadata before processing each batch. + +If code crosses a boundary where contextvars may not propagate automatically, +pass the identifiers explicitly and bind them again at the boundary. + +### Log levels and event naming + +Use log levels consistently: + +| Level | Use | +|---|---| +| `DEBUG` | Local diagnostics, disabled by default in production. | +| `INFO` | Normal lifecycle events: request started/completed, graph run completed, ingestion job completed. | +| `WARNING` | Recoverable anomalies: fallback prompt used, retry scheduled, insufficient retrieval before clarification/escalation. | +| `ERROR` | Failed operations requiring attention: unhandled exception, dependency outage, ingestion failure. | + +Do not log expected user behavior at `ERROR`. For example, a user asking an +ambiguous question that leads to clarification is an `INFO` event; a retriever +being unavailable is an `ERROR` event. + +Use stable dot-separated event names: + +- `request.started` +- `request.completed` +- `request.failed` +- `auth.succeeded` +- `auth.failed` +- `graph.run.started` +- `graph.run.completed` +- `graph.run.escalated` +- `retrieval.completed` +- `retrieval.insufficient` +- `llm.call.completed` +- `llm.call.failed` +- `ingestion.job.started` +- `ingestion.job.completed` +- `point.mutation.completed` + +Do not include dynamic values in logger names or event names. Put dynamic values +in structured fields. + +### Security and privacy rules + +Logs must not contain secrets or raw sensitive payloads. + +Never log: + +- plaintext API keys; +- `Authorization` headers; +- database URLs or provider credentials; +- raw uploaded file contents; +- full retrieved chunks by default; +- raw user messages, raw prompts, or raw model outputs by default; +- embeddings or vectors. + +Prefer: + +- ids (`request_id`, `thread_id`, `run_id`, `file_id`, `point_id`); +- hashes (`input_message_hash`, `output_message_hash`, `content_sha256`); +- counts, sizes, durations, and status codes; +- short redacted summaries only when useful and allowed by tenant policy. + +The same redaction policy used for ADR-0009 `llm_call_payloads` and ADR-0010 +Langfuse tracing should guide log redaction. Logging should be safe even when +log aggregation has broader access than the application database. + +### Relationship to Postgres and Langfuse + +Structured logs complement but do not replace ADR-0009 and ADR-0010. + +| Question | System of record | +|---|---| +| What happened operationally inside this process? | Structured logs. | +| Which API key called which endpoint and what durable side effect occurred? | Postgres audit tables from ADR-0009. | +| Which graph node, prompt version, retrieved chunks, and model calls produced an answer? | Langfuse traces from ADR-0010. | +| What should be used for tenant billing and compliance reports? | Postgres `llm_calls`, `llm_pricing`, `api_request_logs`, and audit tables. | +| What should be used for interactive debugging of one LLM answer? | Langfuse trace, linked from logs/Postgres by ids. | + +Logs may contain `request_id`, `run_id`, and `langfuse_trace_id` so engineers can +navigate across all three systems. + +## Consequences + +### Positive +- Logs become queryable by `request_id`, `thread_id`, `run_id`, `tenant_id`, + `file_id`, and `ingestion_job_id`. +- Contextvars prevent async task interleaving from mixing request context. +- Stdlib/framework logs and application logs share one JSON structure. +- Production logs are compatible with common log collectors and container + runtimes. +- Logs, Postgres audit rows, and Langfuse traces can be correlated without + duplicating each system's purpose. + +### Negative +- Logging setup is more complex than plain `logging.basicConfig()`. +- Developers must learn to use structured fields instead of prose-only log + messages. +- Context must be rebound at worker/background-task boundaries. +- Too much logging can increase cost and leak sensitive data if redaction rules + are not followed. +- JSON logs are less pleasant locally unless a console renderer is enabled for + development. + +## Alternatives Considered + +- **Use Python stdlib logging only**: rejected. Stdlib logging can work, but + `structlog` gives cleaner structured context, contextvars integration, and + consistent event dictionaries across application and framework logs. +- **Use manual `ContextVar` fields only**: rejected as the default. Manual + context variables work, but `structlog.contextvars.bind_contextvars()` and + `merge_contextvars` provide a standard way to bind arbitrary request fields + without maintaining one `ContextVar` per field. Manual `ContextVar`s may still + be used for special cases. +- **Use `session_id` as the primary chat correlation field**: rejected for this + service. ADR-0007 standardizes on `thread_id`, which maps to LangGraph threads + and Langfuse sessions. If the main backend calls the same concept a session, + it is translated to `thread_id` at this service boundary. +- **Write only to `logs/app.log`**: rejected for production. File logging is + useful locally, but stdout JSON is the better default for deployed services. +- **Use Langfuse for all observability**: rejected. Langfuse is excellent for + LLM/agent traces, prompt versions, scores, and evals, but it is not a + replacement for process logs covering FastAPI middleware, auth, SQLAlchemy, + Qdrant calls, worker lifecycle, and non-LLM failures. +- **Use Postgres audit tables as logs**: rejected. ADR-0009 tables are durable + business/audit records. They should not receive high-volume operational debug + logs.