azk/chatbot_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(observability): define Langfuse and structured logging strategy

Browse Source 
Why:
- Establish separate observability systems for LLM tracing, prompt iteration, evaluation, operational logs, and durable application audit records.

Changes:
- Define Langfuse traces, prompt labels, feedback scores, evaluation workflows, redaction rules, and correlation identifiers.
- Define structlog-based JSON logging, request context propagation, event naming, log levels, and privacy requirements.

Impact:
- Langfuse remains the LLM observability plane, while Postgres remains the durable audit and billing source of truth.
- Application logs must avoid secrets and raw sensitive payloads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Ali Zarin Kolah
2026-08-10 11:45:19 +03:30
parent 96ff2ec137
commit e9caeaa4d8
2 changed files with 880 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

455
docs/adr/0010-langfuse-observability-and-prompt-management.md Normal file
View File

@@ -0,0 +1,455 @@
# 0010. Langfuse observability, prompt management, and evaluation workflow
## Status
Proposed
## Context
The chatbot now has architectural decisions for the LangGraph conversation
([0006](0006-conversational-agent-graph.md)), LangGraph thread persistence
([0007](0007-agent-persistence-threads-and-memory.md)), the FastAPI REST
boundary ([0008](0008-rest-api-and-fastapi-boundary.md)), and Postgres tables
for tenants, API keys, ingestion jobs, graph runs, LLM calls, payload retention,
and feedback ([0009](0009-postgres-sqlalchemy-alembic-schema.md)). Those
systems record *what happened* internally, but they do not provide the day-to-day
LLM observability and prompt iteration workflow needed to run an insurance
chatbot safely.
We need to answer questions like:
- Which graph node failed — `triage`, `retrieve`, `generate`, or `verify`?
- Which retrieved chunks did the model see before it answered or escalated?
- Which prompt version produced a bad answer?
- Are prompt changes improving groundedness or just reducing escalations?
- Which tenants, domains, or routes are most expensive?
- Which failures should be fixed by better retrieval, better source content,
better prompts, or stricter handoff policy?
Langfuse is a good fit for the observability and prompt/evaluation layer. It
provides traces, nested observations, generation token/cost tracking, prompt
version management, scores/user feedback, datasets, experiments, evaluators,
and annotation workflows. It should not replace the application database:
Postgres remains the source of truth for API-key authentication, tenant/audit
records, ingestion jobs, graph run rows, billing/cost ledgers, and retention
policy enforcement.
Current Langfuse guidance relevant to this project:
- Use the current Langfuse Python v3 SDK.
- For LangGraph/LangChain, use `from langfuse.langchain import CallbackHandler`
and pass the handler in the graph invocation config.
- Use one trace per user-facing chatbot run.
- Use stable, low-cardinality trace and observation names.
- Model calls should appear as generation observations so model, token, latency,
and cost information can attach correctly.
- Conversation grouping should use Langfuse sessions; in this service,
LangGraph `thread_id` is the Langfuse `session_id`.
- Prompt management uses prompt versions and labels (for example `development`,
`staging`, `production`, tenant-specific labels, and automatically maintained
`latest`). Labels can represent environments, tenants, or experiments.
- Scores represent explicit feedback and implicit/evaluator signals.
- Sensitive data should be masked before export, especially in insurance.
## Decision
### Langfuse is the observability plane, not the transactional database
Adopt Langfuse for:
- traces and nested observations;
- prompt versions, variables, labels, diffs, and trace links;
- user feedback and evaluator scores;
- datasets and experiments;
- annotation queues and systematic error analysis;
- interactive debugging and dashboards.
Keep Postgres ([0009](0009-postgres-sqlalchemy-alembic-schema.md)) as the
internal system of record for:
- tenants and API keys;
- request and point audit records;
- source files and ingestion jobs;
- `graph_runs` and `run_feedback`;
- `llm_calls`, `llm_pricing`, and optional `llm_call_payloads`;
- tenant-specific retention, erasure, and billing/reporting queries.
The two systems are correlated with stable identifiers:
| Identifier | Origin | Use |
|---|---|---|
| `request_id` | FastAPI middleware | Correlate API logs, Postgres audit rows, Langfuse trace metadata. |
| `tenant_id` | API-key dependency | Tenant filtering/cost attribution; never client-supplied. |
| `thread_id` | REST path / LangGraph config | LangGraph thread id and Langfuse `session_id`. |
| `run_id` | `graph_runs.id` | One `POST /v1/threads/{thread_id}/runs` execution. |
| `llm_call_id` | `llm_calls.id` | Optional correlation from a Langfuse generation to Postgres usage ledger. |
| `langfuse_trace_id` | Langfuse | Stored on `graph_runs.metadata` initially, or promoted to a typed column later if frequently queried. |
| `langfuse_observation_id` | Langfuse | Correlated by putting `llm_call_id` in Langfuse observation metadata; add a typed Postgres column later only if needed. |
### SDK and invocation pattern
Use the current Langfuse Python SDK v3 integration for LangGraph/LangChain:
```python
from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
langfuse = get_client()
trace_tags = [environment, "chatbot", "threads:runs"]
trace_metadata = {
"request_id": request_id,
"tenant_id": str(tenant_id),
"api_key_id": str(api_key_id),
"thread_id": thread_id,
"run_id": str(run_id),
"graph_version": graph_version,
"retrieval_config_version": retrieval_config_version,
"prompt_label": prompt_label,
}
with propagate_attributes(
trace_name="chat-run",
user_id=external_user_id,
session_id=thread_id,
tags=trace_tags,
metadata=trace_metadata,
):
langfuse_handler = CallbackHandler()
result = graph.invoke(
graph_input,
config={
"callbacks": [langfuse_handler],
"configurable": {
"thread_id": thread_id,
"tenant_id": str(tenant_id),
"user_id": external_user_id,
},
},
)
```
If the implementation sets Langfuse attributes through LangChain/LangGraph
`config["metadata"]` instead of or in addition to `propagate_attributes`, use the
current integration keys `langfuse_user_id`, `langfuse_session_id`, and
`langfuse_tags` for user, session, and tag propagation.
For non-LangChain/LangGraph work that should appear in the same trace — FastAPI
request spans, Qdrant retrieval, custom reranking metadata, or explicit audit
steps — use Langfuse/OpenTelemetry spans around that work rather than relying
only on automatic LLM callbacks.
Short-lived scripts and tests call `langfuse.flush()` or `langfuse.shutdown()`
before exit. Long-lived FastAPI workers flush during graceful shutdown.
### Trace shape
Create one Langfuse trace per `POST /v1/threads/{thread_id}/runs` call.
Trace-level fields:
| Field | Value |
|---|---|
| name | `chat-run` |
| input | The latest user message, after request-level redaction. |
| output | Final assistant answer, clarifying question, or escalation message. |
| user_id | External `user_id` supplied by the main backend. |
| session_id | LangGraph `thread_id`. |
| tags | Environment, feature (`chatbot`), route (`threads:runs`), tenant segment if safe, terminal status. |
| metadata | `request_id`, `tenant_id`, `api_key_id`, `thread_id`, `run_id`, graph version, retrieval config version, prompt label. |
Observation hierarchy mirrors ADR-0006's graph instead of producing a flat list:
| Observation | Type | Contents |
|---|---|---|
| `load-memory` | span | Store lookup latency and memory keys, not sensitive values. |
| `triage` | generation | Intent classification prompt/output. |
| `contextualize` | generation | Query rewrite input/output. |
| `retrieve` | retriever/span | Qdrant collection, vector names, filters, top-k, returned point/chunk IDs, scores, and source metadata. Avoid raw chunk text unless redacted. |
| `grade` | generation | Relevance verdicts over retrieved chunks. |
| `clarify` | generation | Grounded clarification question when used. |
| `generate` | generation | Final answer generation with citation IDs. |
| `verify` | generation | Groundedness/citation verification result. |
| `escalate` | span | Escalation reason and handoff context summary. |
| `write-memory` | span | Memory write keys and outcome, not sensitive facts. |
Observation names stay stable and low-cardinality. Dynamic values such as tenant
slug, file id, model name, retry number, point id, and user text go in metadata
or input/output fields, not in observation names.
Every generation should include model name, input tokens, output tokens,
latency, and cost when available. Langfuse provides the interactive view;
`llm_calls` in Postgres remains the billing/audit ledger.
### Prompt management and tenant-specific variants
Manage prompts in Langfuse for graph nodes whose behavior will be tuned over
time:
- `insurance-chatbot/triage`
- `insurance-chatbot/contextualize`
- `insurance-chatbot/grade-retrieval`
- `insurance-chatbot/clarify`
- `insurance-chatbot/generate-answer`
- `insurance-chatbot/verify-grounding`
- `insurance-chatbot/summarize-thread`
- `insurance-chatbot/extract-memory`
Prompt text, versions, diffs, labels, and rollback live in Langfuse. Tenant
identity and the policy that chooses which prompt label/name to fetch live in the
application and, where tenant-specific configuration is needed, Postgres
`tenants.settings`.
Prompt names are stable and low-cardinality. The initial tenant-customization
model uses shared prompt names plus tenant-specific labels:
| Label | Use |
|---|---|
| `development` | Local/dev experiments. |
| `staging` | Pre-production validation and experiments. |
| `production` | Default live traffic prompt when no tenant-specific override exists. |
| `tenant-{tenant_slug}-{environment}` | Tenant-specific prompt version for one environment, for example `tenant-acme-production`. |
| `latest` | Automatically points to newest version; useful for discovery, not production runtime. |
Langfuse labels can represent environments, tenants, or experiments. The service
therefore uses labels to support different live prompts per tenant without
creating an application-owned prompt-version table.
Tenant-specific prompt resolution is handled in the application with an explicit
fallback order:
1. derive `tenant_slug` from the authenticated `AuthContext`, never from a
request body/query parameter;
2. derive the environment label from trusted configuration, for example
`production` or `staging`;
3. try the tenant/environment label, for example `tenant-acme-production`;
4. fall back to the environment label, for example `production`;
5. fail closed or use a code-packaged emergency prompt only if the configured
prompt cannot be loaded.
Indicative resolver shape:
```python
prompt_name = "insurance-chatbot/generate-answer"
environment_label = settings.langfuse_prompt_label # e.g. "production"
tenant_label = f"tenant-{auth.tenant_slug}-{environment_label}"
prompt = prompt_resolver.get_with_fallback(
name=prompt_name,
labels=[tenant_label, environment_label],
)
compiled_prompt = prompt.compile(
language=language,
user_message=user_message,
conversation_summary=conversation_summary,
retrieved_chunks=retrieved_chunks,
tenant_policy=tenant_policy,
handoff_policy=handoff_policy,
)
```
The exact SDK call and not-found error handling should follow the current
Langfuse Python SDK, but the architectural rule is stable: select the tenant
label from trusted server-side tenant context, not from caller input.
This keeps prompt content, prompt versions, prompt diffs, and label movement in
Langfuse while keeping tenant identity and resolver policy in the application.
If tenant-specific prompts become large and independently owned, an acceptable
future variant is tenant-namespaced prompt names such as
`insurance-chatbot/tenants/acme/generate-answer` with ordinary `production` and
`staging` labels. The initial design prefers shared prompt names plus
tenant-specific labels because it keeps graph-node prompts easier to compare
across tenants.
Postgres may store prompt resolver configuration in `tenants.settings`, for
example an enabled flag, preferred prompt namespace, explicit label override,
default language, tenant policy reference, or handoff policy reference. Postgres
must not become the source of truth for prompt text/version history; that
belongs in Langfuse.
Prompt variables carry dynamic context, for example:
- `{{language}}`
- `{{user_message}}`
- `{{conversation_summary}}`
- `{{retrieved_chunks}}`
- `{{tenant_policy}}`
- `{{handoff_policy}}`
- `{{citation_contract}}`
- `{{memory_profile}}`
Prompt versions are linked to traces so failures can be traced back to the exact
prompt version that produced them. Rollback is performed by moving the relevant
label — `production` for the default prompt, or `tenant-acme-production` for one
tenant's override — back to a known-good version, not by redeploying code.
Prompt changes are evaluated before promotion using Langfuse datasets and
experiments. A prompt version should not be promoted to `production` or to a
tenant-specific production label only because one manual test looked good.
### Privacy, redaction, and retention
Insurance conversations can contain PII, policy details, claim information, and
financial/health-adjacent facts. Langfuse is therefore sensitive infrastructure.
Rules:
- Mask/redact before export from the application process. Prefer current SDK
masking support such as `mask_otel_spans` where applicable, plus deterministic
field/path redaction for known sensitive fields.
- Never put API keys, database URLs, provider secrets, raw auth headers, or
unredacted request bodies in trace metadata.
- Do not trace raw uploaded files or full document text.
- Retrieval observations should prefer point/chunk IDs, file IDs, scores,
titles, and short redacted snippets over full retrieved chunks.
- Raw LLM input/output retention is governed by ADR-0009's
`llm_call_payloads` tenant policy, not by accidental Langfuse trace capture.
- Langfuse project retention/deletion settings must align with insurance and
tenant contractual retention requirements.
- If self-hosted Langfuse is used later, server-side ingestion masking can add
defense in depth, but it does not replace client-side masking because data
can still reach pre-masking ingestion infrastructure.
### Feedback and scores
Record explicit and implicit feedback as Langfuse scores while mirroring durable
feedback to Postgres `run_feedback` when the application needs it.
Explicit feedback:
| Signal | Langfuse score name | Notes |
|---|---|---|
| Thumbs up/down | `user-thumbs` | Boolean score, attached to the trace/run. |
| Optional comment | score comment/metadata | Also stored in `run_feedback.comment` if enabled. |
| Report bad answer | `user-report` | Useful for annotation queues and support review. |
Implicit feedback:
| Signal | Score/tag | Meaning |
|---|---|---|
| Handoff triggered | `handoff-triggered` | The run ended in `status=escalate`. |
| Regeneration/retry | `user-retry` or metadata | Negative/diagnostic signal. |
| Citation opened | `citation-opened` | User inspected a cited source. |
| Answer copied | `answer-copied` | Weak positive signal if frontend exposes it. |
The FastAPI response for a run should include enough trace/run identity for the
backend or frontend to submit feedback later, but no Langfuse secret key is ever
sent to a browser. Browser-side feedback, if added, may use only a Langfuse
public key or, preferably, route feedback through the backend.
### Evaluation and quality workflow
Use Langfuse beyond basic tracing:
1. **Datasets** — maintain representative insurance questions, expected
citation behavior, ambiguous questions, out-of-scope cases, and handoff
cases.
2. **Experiments** — compare prompt versions, retrieval parameters, reranking
thresholds, model choices, and verification thresholds before deployment.
3. **LLM-as-judge evaluators** — score groundedness, citation correctness,
ambiguity handling, escalation correctness, tone/language, and refusal
quality. Judges are aids, not absolute truth.
4. **Annotation queues** — send low-score traces, user reports, ungrounded
answers, and unnecessary escalations to human review.
5. **Error analysis** — periodically sample traces, open-code failures, cluster
failure modes, and decide whether the fix belongs in retrieval, source
content, prompt text, graph policy, or human handoff rules.
6. **Dashboards** — track per-tenant cost, per-node token spend, latency,
escalation rate, clarification rate, retrieval-insufficient rate, and
verification-failure rate.
### Boundary with ADR-0009 Postgres tables
Langfuse overlaps with parts of ADR-0009, especially LLM-call visibility,
feedback, and run debugging. This overlap is intentional, but the systems have
different authority.
| Need | Langfuse | Postgres | Decision |
|---|---|---|---|
| Prompt text, versions, diffs, labels, and rollback | Yes | No | Use Langfuse as the prompt source of truth. |
| Tenant-specific prompt content | Yes | No | Use Langfuse prompt labels or, later, tenant-namespaced prompt names. |
| Tenant prompt-routing policy | Partly | Yes | Store trusted tenant config in `tenants.settings` or app config; fetch prompt content from Langfuse. |
| Interactive debugging of one bad answer | Yes | Partly | Use Langfuse trace; correlate to `graph_runs.id`. |
| Trace/session grouping | Yes | Partly | Use Langfuse `session_id=thread_id`; keep `thread_id` on `graph_runs` for ledger/audit. |
| LLM token/cost visibility | Yes | Yes | Langfuse for observability; Postgres `llm_calls`/`llm_pricing` for billing and audit. |
| Durable graph-run ledger | No | Yes | Keep `graph_runs`. Langfuse traces are not the application run table. |
| User feedback | Yes | Yes | Store scores in Langfuse and durable feedback in `run_feedback` when needed by the app. |
| Error analysis and annotation queues | Yes | Usually no | Use Langfuse; avoid building an annotation workflow in Postgres initially. |
| Datasets, experiments, and evaluators | Yes | Usually no | Use Langfuse; copy to Postgres only if a product/compliance need appears. |
| Tenants and API keys | No | Yes | Keep `tenants` and `api_keys` in Postgres. |
| API request audit logs | Not enough | Yes | Keep `api_request_logs` in Postgres. |
| Point/file mutation audit | No | Yes | Keep `point_audit_events`, `source_files`, and `ingestion_jobs` in Postgres. |
| Controlled raw/redacted payload retention | Partly | Yes | Postgres `llm_call_payloads` plus tenant policy owns retention; Langfuse must be redacted/configured to comply. |
| Legal retention and tenant erasure policy | Partly | Yes | Postgres owns application policy; Langfuse project retention/deletion settings must align with it. |
Do not remove ADR-0009 tables because Langfuse exists. Langfuse is optimized for
observability, prompt iteration, feedback analysis, datasets, experiments, and
annotation workflows. Postgres is optimized for auth, audit, retention, billing,
and application consistency.
## Consequences
### Positive
- Every chat run becomes inspectable as a graph-shaped trace rather than a flat
log line or a pile of model-call rows.
- Prompt versions can be changed, compared, rolled back, and linked to the
traces they produced without redeploying application code for every prompt
edit.
- Tenant-specific prompt labels let one tenant receive a customized prompt while
other tenants continue using the default `production` version.
- Scores, datasets, experiments, and annotation queues create a path from user
feedback to systematic quality improvement.
- Per-node token/cost/latency data exposes where the graph is expensive or
slow: retrieval, generation, verification, or memory.
- Correlating Langfuse IDs with Postgres IDs preserves both workflows:
interactive debugging and durable audit/billing.
### Negative
- Langfuse adds another service and another set of credentials, retention
policies, access controls, and operational dashboards.
- Prompt management introduces runtime dependency on prompt fetch/caching
behavior. The application needs safe fallbacks if Langfuse is temporarily
unavailable.
- Tenant-specific prompt labels add routing complexity. The resolver must avoid
caller-controlled labels and must have a clear fallback when a tenant override
does not exist.
- Observability can become a privacy leak if masking is incomplete. Insurance
content makes this a high-consequence risk.
- There is intentional duplication between Langfuse generation usage and
Postgres `llm_calls`; implementations must avoid reconciling them as if one
were authoritative for the other's purpose.
- LLM-as-judge evaluators can be wrong. They should guide review and experiments,
not replace human analysis for high-risk policy/coverage answers.
## Alternatives Considered
- **Use Postgres tables only for observability**: rejected. `llm_calls` and
audit tables are good for billing and compliance queries, but poor for
nested trace inspection, prompt-version attribution, annotation queues, and
experiment workflows.
- **Use Langfuse as the only usage/cost store**: rejected. Tenant billing,
erasure, audit, and internal consistency need application-owned tables with
controlled retention and foreign keys to tenants/API keys.
- **Use Langfuse instead of ADR-0009 operational tables**: rejected. Langfuse can
help with traces, prompt versions, scores, datasets, experiments, and
annotation queues, but it should not own API-key authentication, request audit,
point/file mutation audit, ingestion jobs, or the internal billing ledger.
- **Store tenant prompt text and versions in Postgres**: rejected for the initial
design. It would duplicate Langfuse prompt-management features and require the
project to build its own versioning, diffs, labels, rollback, and trace links.
Postgres stores tenant prompt-routing configuration only.
- **Keep prompts only in code**: rejected for prompts that will be tuned often.
Code-only prompts make rollback and prompt/version-to-trace analysis harder.
Small static strings and schema definitions may still live in code.
- **Fetch prompts by `latest` in production**: rejected. `latest` changes every
time a new version is created. Production uses a configured label such as
`production`; rollback is moving the label.
- **Trace full retrieved chunks and raw files for easier debugging**: rejected.
It would make debugging convenient but creates unnecessary privacy and storage
risk. IDs, scores, metadata, and redacted snippets are enough by default.
- **One Langfuse project per tenant**: rejected for the initial design. A single
project per environment with tenant metadata/tags is easier to operate and
compare. Revisit per-tenant projects only if contractual isolation requires
it.
- **Skip Langfuse scores until evals are mature**: rejected. Simple user-thumbs
and handoff-triggered scores are cheap and immediately useful for filtering
traces and building the first annotation queues.

425
docs/adr/0011-python-structured-logging-with-structlog.md Normal file
View File

@@ -0,0 +1,425 @@
# 0011. Python structured logging with structlog and request context
## Status
Proposed
## Context
The service needs application-level Python logging in addition to the durable
Postgres records from ADR-0009 and the LLM/agent traces from ADR-0010.
Postgres audit tables answer durable business questions such as which tenant API
key called an endpoint, which point mutation was requested, which ingestion job
ran, and what usage ledger rows were produced. Langfuse answers LLM observability
questions such as which graph node or prompt version produced an answer.
Structured Python logs answer operational questions while the service is running:
- Which request failed and where?
- Which logs belong to one FastAPI request or LangGraph run?
- Which tenant, API key, thread, run, file, point, or ingestion job was involved?
- Which dependency was slow or unavailable?
- Which fallback path or retry was used?
FastAPI, LangGraph, SQLAlchemy, Qdrant, Langfuse, and HTTP clients can all emit
logs from asynchronous code. Since async tasks can interleave on the same event
loop, relying on process-global mutable variables is unsafe. The logging context
must be request-scoped and safe across async task switching. Python
`contextvars`, exposed through `structlog.contextvars`, provide this behavior.
The user has used a previous `log.py` based on `logging`, `structlog`,
`logging.config.dictConfig`, `ProcessorFormatter`, JSON rendering, stdlib log
capture, and manual `ContextVar` fields such as `session_id` and `property_id`.
This service should keep the same core idea but adapt the field names to the
current architecture:
- `thread_id` instead of `session_id` for LangGraph conversations, matching
ADR-0007 and ADR-0008;
- `tenant_id`, `tenant_slug`, `api_key_id`, and `actor_type` from `AuthContext`;
- `request_id` from FastAPI middleware;
- `run_id` from `graph_runs` for chat runs;
- `file_id`, `point_id`, and `ingestion_job_id` for ingestion and point work;
- `langfuse_trace_id` when available for cross-navigation to ADR-0010 traces.
## Decision
### Use structlog as the application logging interface
Use `structlog` for application logs and integrate it with Python stdlib logging
so framework/library logs are formatted consistently.
Application code imports loggers with:
```python
import structlog
logger = structlog.get_logger(__name__)
```
Log events use stable event names and structured fields:
```python
logger.info(
"graph.run.completed",
status="answered",
duration_ms=duration_ms,
retrieved_chunk_count=len(retrieved_chunks),
)
```
Do not build log messages by interpolating operational metadata into prose.
Prefer fields over long strings because fields are queryable.
### Emit JSON logs by default in production
Production logs are JSON on stdout so process managers, container runtimes, and
log collectors can ingest them directly. Local development may use a colored
console renderer controlled by configuration.
File logging is optional and mainly for local development. If enabled, it must
use explicit rotation settings such as `maxBytes` and `backupCount`. Do not rely
on a default `RotatingFileHandler` with no rotation parameters. In containerized
production, stdout/stderr collection is preferred over writing `logs/app.log`
inside the application container.
### Configure stdlib and structlog together
The logging setup should happen once during process startup, before the FastAPI
app begins serving requests.
Indicative configuration shape:
```python
import logging
import logging.config
import sys
import structlog
def configure_logging(*, log_level: str, json_logs: bool) -> None:
shared_processors = [
structlog.contextvars.merge_contextvars,
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
structlog.processors.TimeStamper(fmt="iso", utc=True),
structlog.processors.StackInfoRenderer(),
]
structlog.configure(
processors=[
*shared_processors,
structlog.processors.format_exc_info,
structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
],
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
renderer = (
structlog.processors.JSONRenderer()
if json_logs
else structlog.dev.ConsoleRenderer(colors=True)
)
logging.config.dictConfig(
{
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"default": {
"()": structlog.stdlib.ProcessorFormatter,
"processors": [
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
renderer,
],
"foreign_pre_chain": [
structlog.stdlib.ExtraAdder(),
*shared_processors,
],
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": log_level,
"formatter": "default",
"stream": sys.stdout,
},
},
"loggers": {
"": {
"handlers": ["console"],
"level": log_level,
"propagate": False,
},
"uvicorn": {
"handlers": ["console"],
"level": log_level,
"propagate": False,
},
"uvicorn.access": {
"handlers": ["console"],
"level": log_level,
"propagate": False,
},
"sqlalchemy.engine": {
"handlers": ["console"],
"level": "WARNING",
"propagate": False,
},
"watchfiles": {
"handlers": ["console"],
"level": "INFO",
"propagate": False,
},
},
}
)
```
Notes:
- Use the logger name `sqlalchemy.engine`, not `sqlalchemy.engin`.
- SQL statement logging is too noisy and can leak values; keep it `WARNING` by
default in production and enable `INFO`/`DEBUG` only in controlled debugging.
- `structlog.stdlib.ExtraAdder()` keeps useful fields from stdlib log records.
- `structlog.contextvars.merge_contextvars` ensures request-bound fields appear
on both structlog and stdlib logs processed through the formatter.
### Bind request context with contextvars
At FastAPI ingress, clear stale context, bind request identifiers, and return the
request id to callers. This makes it possible to select all logs from one request
or one graph run even when async tasks interleave.
Indicative middleware:
```python
from time import perf_counter
from uuid import uuid4
import structlog
from fastapi import Request
from starlette.types import ASGIApp
REQUEST_ID_HEADER = "X-Request-ID"
async def logging_context_middleware(request: Request, call_next: ASGIApp):
structlog.contextvars.clear_contextvars()
request_id = request.headers.get(REQUEST_ID_HEADER) or str(uuid4())
route = request.scope.get("route")
path_template = getattr(route, "path", request.url.path)
structlog.contextvars.bind_contextvars(
request_id=request_id,
method=request.method,
path_template=path_template,
)
logger = structlog.get_logger("app.http")
started = perf_counter()
logger.info("request.started")
try:
response = await call_next(request)
except Exception:
logger.exception(
"request.failed",
duration_ms=round((perf_counter() - started) * 1000, 2),
)
raise
response.headers[REQUEST_ID_HEADER] = request_id
logger.info(
"request.completed",
status_code=response.status_code,
duration_ms=round((perf_counter() - started) * 1000, 2),
)
return response
```
After API-key authentication succeeds, the auth dependency or route handler binds
trusted tenant/auth fields:
```python
structlog.contextvars.bind_contextvars(
tenant_id=str(auth.tenant_id),
tenant_slug=auth.tenant_slug,
api_key_id=str(auth.api_key_id),
actor_type=auth.actor_type,
)
```
Route handlers bind route-specific fields when they become known:
```python
structlog.contextvars.bind_contextvars(
external_user_id=body.user_id,
thread_id=thread_id,
run_id=str(run_id),
)
```
Use these canonical context keys:
| Field | Source | Notes |
|---|---|---|
| `request_id` | FastAPI middleware | Primary log correlation id; also appears in ADR-0008/0009 records. |
| `tenant_id` | `AuthContext` | Trusted server-side tenant id; never request body/query. |
| `tenant_slug` | `AuthContext` | Useful for filtering; avoid if contractual policy treats it as sensitive. |
| `api_key_id` | `AuthContext` | Non-secret id only. Never log raw API keys or auth headers. |
| `actor_type` | `AuthContext` | `backend`, `admin`, or `worker`. |
| `external_user_id` | Main backend | May be high-cardinality; acceptable in logs, not metrics labels. |
| `thread_id` | REST path | LangGraph thread id. |
| `run_id` | `graph_runs.id` | One chat run. |
| `ingestion_job_id` | `ingestion_jobs.id` | File ingestion correlation. |
| `file_id` | `source_files.id` | Source-file correlation. |
| `point_id` | Qdrant point id | Point mutation/read correlation. |
| `langfuse_trace_id` | Langfuse | Cross-link to ADR-0010 trace when available. |
Use `structlog.contextvars.clear_contextvars()` at request/task ingress to avoid
leaking a previous request's context into reused workers.
### Bind context explicitly for jobs and background work
Context variables work across normal async task switching, but background jobs,
worker processes, scheduled jobs, and threadpool work should bind context at
their own entry point from durable identifiers.
Examples:
- ingestion worker binds `tenant_id`, `ingestion_job_id`, `file_id`, and
`request_id` if inherited from the upload request;
- LangGraph run execution binds `thread_id`, `run_id`, `tenant_id`, and
`external_user_id` before invoking the graph;
- point batch workers bind `tenant_id`, `api_request_log_id`, and operation
metadata before processing each batch.
If code crosses a boundary where contextvars may not propagate automatically,
pass the identifiers explicitly and bind them again at the boundary.
### Log levels and event naming
Use log levels consistently:
| Level | Use |
|---|---|
| `DEBUG` | Local diagnostics, disabled by default in production. |
| `INFO` | Normal lifecycle events: request started/completed, graph run completed, ingestion job completed. |
| `WARNING` | Recoverable anomalies: fallback prompt used, retry scheduled, insufficient retrieval before clarification/escalation. |
| `ERROR` | Failed operations requiring attention: unhandled exception, dependency outage, ingestion failure. |
Do not log expected user behavior at `ERROR`. For example, a user asking an
ambiguous question that leads to clarification is an `INFO` event; a retriever
being unavailable is an `ERROR` event.
Use stable dot-separated event names:
- `request.started`
- `request.completed`
- `request.failed`
- `auth.succeeded`
- `auth.failed`
- `graph.run.started`
- `graph.run.completed`
- `graph.run.escalated`
- `retrieval.completed`
- `retrieval.insufficient`
- `llm.call.completed`
- `llm.call.failed`
- `ingestion.job.started`
- `ingestion.job.completed`
- `point.mutation.completed`
Do not include dynamic values in logger names or event names. Put dynamic values
in structured fields.
### Security and privacy rules
Logs must not contain secrets or raw sensitive payloads.
Never log:
- plaintext API keys;
- `Authorization` headers;
- database URLs or provider credentials;
- raw uploaded file contents;
- full retrieved chunks by default;
- raw user messages, raw prompts, or raw model outputs by default;
- embeddings or vectors.
Prefer:
- ids (`request_id`, `thread_id`, `run_id`, `file_id`, `point_id`);
- hashes (`input_message_hash`, `output_message_hash`, `content_sha256`);
- counts, sizes, durations, and status codes;
- short redacted summaries only when useful and allowed by tenant policy.
The same redaction policy used for ADR-0009 `llm_call_payloads` and ADR-0010
Langfuse tracing should guide log redaction. Logging should be safe even when
log aggregation has broader access than the application database.
### Relationship to Postgres and Langfuse
Structured logs complement but do not replace ADR-0009 and ADR-0010.
| Question | System of record |
|---|---|
| What happened operationally inside this process? | Structured logs. |
| Which API key called which endpoint and what durable side effect occurred? | Postgres audit tables from ADR-0009. |
| Which graph node, prompt version, retrieved chunks, and model calls produced an answer? | Langfuse traces from ADR-0010. |
| What should be used for tenant billing and compliance reports? | Postgres `llm_calls`, `llm_pricing`, `api_request_logs`, and audit tables. |
| What should be used for interactive debugging of one LLM answer? | Langfuse trace, linked from logs/Postgres by ids. |
Logs may contain `request_id`, `run_id`, and `langfuse_trace_id` so engineers can
navigate across all three systems.
## Consequences
### Positive
- Logs become queryable by `request_id`, `thread_id`, `run_id`, `tenant_id`,
`file_id`, and `ingestion_job_id`.
- Contextvars prevent async task interleaving from mixing request context.
- Stdlib/framework logs and application logs share one JSON structure.
- Production logs are compatible with common log collectors and container
runtimes.
- Logs, Postgres audit rows, and Langfuse traces can be correlated without
duplicating each system's purpose.
### Negative
- Logging setup is more complex than plain `logging.basicConfig()`.
- Developers must learn to use structured fields instead of prose-only log
messages.
- Context must be rebound at worker/background-task boundaries.
- Too much logging can increase cost and leak sensitive data if redaction rules
are not followed.
- JSON logs are less pleasant locally unless a console renderer is enabled for
development.
## Alternatives Considered
- **Use Python stdlib logging only**: rejected. Stdlib logging can work, but
`structlog` gives cleaner structured context, contextvars integration, and
consistent event dictionaries across application and framework logs.
- **Use manual `ContextVar` fields only**: rejected as the default. Manual
context variables work, but `structlog.contextvars.bind_contextvars()` and
`merge_contextvars` provide a standard way to bind arbitrary request fields
without maintaining one `ContextVar` per field. Manual `ContextVar`s may still
be used for special cases.
- **Use `session_id` as the primary chat correlation field**: rejected for this
service. ADR-0007 standardizes on `thread_id`, which maps to LangGraph threads
and Langfuse sessions. If the main backend calls the same concept a session,
it is translated to `thread_id` at this service boundary.
- **Write only to `logs/app.log`**: rejected for production. File logging is
useful locally, but stdout JSON is the better default for deployed services.
- **Use Langfuse for all observability**: rejected. Langfuse is excellent for
LLM/agent traces, prompt versions, scores, and evals, but it is not a
replacement for process logs covering FastAPI middleware, auth, SQLAlchemy,
Qdrant calls, worker lifecycle, and non-LLM failures.
- **Use Postgres audit tables as logs**: rejected. ADR-0009 tables are durable
business/audit records. They should not receive high-volume operational debug
logs.