docs(observability): define Langfuse and structured logging strategy

Why: - Establish separate observability systems for LLM tracing, prompt iteration, evaluation, operational logs, and durable application audit records. Changes: - Define Langfuse traces, prompt labels, feedback scores, evaluation workflows, redaction rules, and correlation identifiers. - Define structlog-based JSON logging, request context propagation, event naming, log levels, and privacy requirements. Impact: - Langfuse remains the LLM observability plane, while Postgres remains the durable audit and billing source of truth. - Application logs must avoid secrets and raw sensitive payloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 11:45:19 +03:30
parent 96ff2ec137
commit e9caeaa4d8
2 changed files with 880 additions and 0 deletions
--- a/docs/adr/0010-langfuse-observability-and-prompt-management.md
+++ b/docs/adr/0010-langfuse-observability-and-prompt-management.md
@@ -0,0 +1,455 @@
+# 0010. Langfuse observability, prompt management, and evaluation workflow
+
+## Status
+
+Proposed
+
+## Context
+
+The chatbot now has architectural decisions for the LangGraph conversation
+([0006](0006-conversational-agent-graph.md)), LangGraph thread persistence
+([0007](0007-agent-persistence-threads-and-memory.md)), the FastAPI REST
+boundary ([0008](0008-rest-api-and-fastapi-boundary.md)), and Postgres tables
+for tenants, API keys, ingestion jobs, graph runs, LLM calls, payload retention,
+and feedback ([0009](0009-postgres-sqlalchemy-alembic-schema.md)). Those
+systems record *what happened* internally, but they do not provide the day-to-day
+LLM observability and prompt iteration workflow needed to run an insurance
+chatbot safely.
+
+We need to answer questions like:
+
+- Which graph node failed — `triage`, `retrieve`, `generate`, or `verify`?
+- Which retrieved chunks did the model see before it answered or escalated?
+- Which prompt version produced a bad answer?
+- Are prompt changes improving groundedness or just reducing escalations?
+- Which tenants, domains, or routes are most expensive?
+- Which failures should be fixed by better retrieval, better source content,
+  better prompts, or stricter handoff policy?
+
+Langfuse is a good fit for the observability and prompt/evaluation layer. It
+provides traces, nested observations, generation token/cost tracking, prompt
+version management, scores/user feedback, datasets, experiments, evaluators,
+and annotation workflows. It should not replace the application database:
+Postgres remains the source of truth for API-key authentication, tenant/audit
+records, ingestion jobs, graph run rows, billing/cost ledgers, and retention
+policy enforcement.
+
+Current Langfuse guidance relevant to this project:
+
+- Use the current Langfuse Python v3 SDK.
+- For LangGraph/LangChain, use `from langfuse.langchain import CallbackHandler`
+  and pass the handler in the graph invocation config.
+- Use one trace per user-facing chatbot run.
+- Use stable, low-cardinality trace and observation names.
+- Model calls should appear as generation observations so model, token, latency,
+  and cost information can attach correctly.
+- Conversation grouping should use Langfuse sessions; in this service,
+  LangGraph `thread_id` is the Langfuse `session_id`.
+- Prompt management uses prompt versions and labels (for example `development`,
+  `staging`, `production`, tenant-specific labels, and automatically maintained
+  `latest`). Labels can represent environments, tenants, or experiments.
+- Scores represent explicit feedback and implicit/evaluator signals.
+- Sensitive data should be masked before export, especially in insurance.
+
+## Decision
+
+### Langfuse is the observability plane, not the transactional database
+
+Adopt Langfuse for:
+
+- traces and nested observations;
+- prompt versions, variables, labels, diffs, and trace links;
+- user feedback and evaluator scores;
+- datasets and experiments;
+- annotation queues and systematic error analysis;
+- interactive debugging and dashboards.
+
+Keep Postgres ([0009](0009-postgres-sqlalchemy-alembic-schema.md)) as the
+internal system of record for:
+
+- tenants and API keys;
+- request and point audit records;
+- source files and ingestion jobs;
+- `graph_runs` and `run_feedback`;
+- `llm_calls`, `llm_pricing`, and optional `llm_call_payloads`;
+- tenant-specific retention, erasure, and billing/reporting queries.
+
+The two systems are correlated with stable identifiers:
+
+| Identifier | Origin | Use |
+|---|---|---|
+| `request_id` | FastAPI middleware | Correlate API logs, Postgres audit rows, Langfuse trace metadata. |
+| `tenant_id` | API-key dependency | Tenant filtering/cost attribution; never client-supplied. |
+| `thread_id` | REST path / LangGraph config | LangGraph thread id and Langfuse `session_id`. |
+| `run_id` | `graph_runs.id` | One `POST /v1/threads/{thread_id}/runs` execution. |
+| `llm_call_id` | `llm_calls.id` | Optional correlation from a Langfuse generation to Postgres usage ledger. |
+| `langfuse_trace_id` | Langfuse | Stored on `graph_runs.metadata` initially, or promoted to a typed column later if frequently queried. |
+| `langfuse_observation_id` | Langfuse | Correlated by putting `llm_call_id` in Langfuse observation metadata; add a typed Postgres column later only if needed. |
+
+### SDK and invocation pattern
+
+Use the current Langfuse Python SDK v3 integration for LangGraph/LangChain:
+
+```python
+from langfuse import get_client, propagate_attributes
+from langfuse.langchain import CallbackHandler
+
+langfuse = get_client()
+trace_tags = [environment, "chatbot", "threads:runs"]
+trace_metadata = {
+    "request_id": request_id,
+    "tenant_id": str(tenant_id),
+    "api_key_id": str(api_key_id),
+    "thread_id": thread_id,
+    "run_id": str(run_id),
+    "graph_version": graph_version,
+    "retrieval_config_version": retrieval_config_version,
+    "prompt_label": prompt_label,
+}
+
+with propagate_attributes(
+    trace_name="chat-run",
+    user_id=external_user_id,
+    session_id=thread_id,
+    tags=trace_tags,
+    metadata=trace_metadata,
+):
+    langfuse_handler = CallbackHandler()
+    result = graph.invoke(
+        graph_input,
+        config={
+            "callbacks": [langfuse_handler],
+            "configurable": {
+                "thread_id": thread_id,
+                "tenant_id": str(tenant_id),
+                "user_id": external_user_id,
+            },
+        },
+    )
+```
+
+If the implementation sets Langfuse attributes through LangChain/LangGraph
+`config["metadata"]` instead of or in addition to `propagate_attributes`, use the
+current integration keys `langfuse_user_id`, `langfuse_session_id`, and
+`langfuse_tags` for user, session, and tag propagation.
+
+For non-LangChain/LangGraph work that should appear in the same trace — FastAPI
+request spans, Qdrant retrieval, custom reranking metadata, or explicit audit
+steps — use Langfuse/OpenTelemetry spans around that work rather than relying
+only on automatic LLM callbacks.
+
+Short-lived scripts and tests call `langfuse.flush()` or `langfuse.shutdown()`
+before exit. Long-lived FastAPI workers flush during graceful shutdown.
+
+### Trace shape
+
+Create one Langfuse trace per `POST /v1/threads/{thread_id}/runs` call.
+
+Trace-level fields:
+
+| Field | Value |
+|---|---|
+| name | `chat-run` |
+| input | The latest user message, after request-level redaction. |
+| output | Final assistant answer, clarifying question, or escalation message. |
+| user_id | External `user_id` supplied by the main backend. |
+| session_id | LangGraph `thread_id`. |
+| tags | Environment, feature (`chatbot`), route (`threads:runs`), tenant segment if safe, terminal status. |
+| metadata | `request_id`, `tenant_id`, `api_key_id`, `thread_id`, `run_id`, graph version, retrieval config version, prompt label. |
+
+Observation hierarchy mirrors ADR-0006's graph instead of producing a flat list:
+
+| Observation | Type | Contents |
+|---|---|---|
+| `load-memory` | span | Store lookup latency and memory keys, not sensitive values. |
+| `triage` | generation | Intent classification prompt/output. |
+| `contextualize` | generation | Query rewrite input/output. |
+| `retrieve` | retriever/span | Qdrant collection, vector names, filters, top-k, returned point/chunk IDs, scores, and source metadata. Avoid raw chunk text unless redacted. |
+| `grade` | generation | Relevance verdicts over retrieved chunks. |
+| `clarify` | generation | Grounded clarification question when used. |
+| `generate` | generation | Final answer generation with citation IDs. |
+| `verify` | generation | Groundedness/citation verification result. |
+| `escalate` | span | Escalation reason and handoff context summary. |
+| `write-memory` | span | Memory write keys and outcome, not sensitive facts. |
+
+Observation names stay stable and low-cardinality. Dynamic values such as tenant
+slug, file id, model name, retry number, point id, and user text go in metadata
+or input/output fields, not in observation names.
+
+Every generation should include model name, input tokens, output tokens,
+latency, and cost when available. Langfuse provides the interactive view;
+`llm_calls` in Postgres remains the billing/audit ledger.
+
+### Prompt management and tenant-specific variants
+
+Manage prompts in Langfuse for graph nodes whose behavior will be tuned over
+time:
+
+- `insurance-chatbot/triage`
+- `insurance-chatbot/contextualize`
+- `insurance-chatbot/grade-retrieval`
+- `insurance-chatbot/clarify`
+- `insurance-chatbot/generate-answer`
+- `insurance-chatbot/verify-grounding`
+- `insurance-chatbot/summarize-thread`
+- `insurance-chatbot/extract-memory`
+
+Prompt text, versions, diffs, labels, and rollback live in Langfuse. Tenant
+identity and the policy that chooses which prompt label/name to fetch live in the
+application and, where tenant-specific configuration is needed, Postgres
+`tenants.settings`.
+
+Prompt names are stable and low-cardinality. The initial tenant-customization
+model uses shared prompt names plus tenant-specific labels:
+
+| Label | Use |
+|---|---|
+| `development` | Local/dev experiments. |
+| `staging` | Pre-production validation and experiments. |
+| `production` | Default live traffic prompt when no tenant-specific override exists. |
+| `tenant-{tenant_slug}-{environment}` | Tenant-specific prompt version for one environment, for example `tenant-acme-production`. |
+| `latest` | Automatically points to newest version; useful for discovery, not production runtime. |
+
+Langfuse labels can represent environments, tenants, or experiments. The service
+therefore uses labels to support different live prompts per tenant without
+creating an application-owned prompt-version table.
+
+Tenant-specific prompt resolution is handled in the application with an explicit
+fallback order:
+
+1. derive `tenant_slug` from the authenticated `AuthContext`, never from a
+   request body/query parameter;
+2. derive the environment label from trusted configuration, for example
+   `production` or `staging`;
+3. try the tenant/environment label, for example `tenant-acme-production`;
+4. fall back to the environment label, for example `production`;
+5. fail closed or use a code-packaged emergency prompt only if the configured
+   prompt cannot be loaded.
+
+Indicative resolver shape:
+
+```python
+prompt_name = "insurance-chatbot/generate-answer"
+environment_label = settings.langfuse_prompt_label  # e.g. "production"
+tenant_label = f"tenant-{auth.tenant_slug}-{environment_label}"
+
+prompt = prompt_resolver.get_with_fallback(
+    name=prompt_name,
+    labels=[tenant_label, environment_label],
+)
+compiled_prompt = prompt.compile(
+    language=language,
+    user_message=user_message,
+    conversation_summary=conversation_summary,
+    retrieved_chunks=retrieved_chunks,
+    tenant_policy=tenant_policy,
+    handoff_policy=handoff_policy,
+)
+```
+
+The exact SDK call and not-found error handling should follow the current
+Langfuse Python SDK, but the architectural rule is stable: select the tenant
+label from trusted server-side tenant context, not from caller input.
+
+This keeps prompt content, prompt versions, prompt diffs, and label movement in
+Langfuse while keeping tenant identity and resolver policy in the application.
+If tenant-specific prompts become large and independently owned, an acceptable
+future variant is tenant-namespaced prompt names such as
+`insurance-chatbot/tenants/acme/generate-answer` with ordinary `production` and
+`staging` labels. The initial design prefers shared prompt names plus
+tenant-specific labels because it keeps graph-node prompts easier to compare
+across tenants.
+
+Postgres may store prompt resolver configuration in `tenants.settings`, for
+example an enabled flag, preferred prompt namespace, explicit label override,
+default language, tenant policy reference, or handoff policy reference. Postgres
+must not become the source of truth for prompt text/version history; that
+belongs in Langfuse.
+
+Prompt variables carry dynamic context, for example:
+
+- `{{language}}`
+- `{{user_message}}`
+- `{{conversation_summary}}`
+- `{{retrieved_chunks}}`
+- `{{tenant_policy}}`
+- `{{handoff_policy}}`
+- `{{citation_contract}}`
+- `{{memory_profile}}`
+
+Prompt versions are linked to traces so failures can be traced back to the exact
+prompt version that produced them. Rollback is performed by moving the relevant
+label — `production` for the default prompt, or `tenant-acme-production` for one
+tenant's override — back to a known-good version, not by redeploying code.
+
+Prompt changes are evaluated before promotion using Langfuse datasets and
+experiments. A prompt version should not be promoted to `production` or to a
+tenant-specific production label only because one manual test looked good.
+
+### Privacy, redaction, and retention
+
+Insurance conversations can contain PII, policy details, claim information, and
+financial/health-adjacent facts. Langfuse is therefore sensitive infrastructure.
+
+Rules:
+
+- Mask/redact before export from the application process. Prefer current SDK
+  masking support such as `mask_otel_spans` where applicable, plus deterministic
+  field/path redaction for known sensitive fields.
+- Never put API keys, database URLs, provider secrets, raw auth headers, or
+  unredacted request bodies in trace metadata.
+- Do not trace raw uploaded files or full document text.
+- Retrieval observations should prefer point/chunk IDs, file IDs, scores,
+  titles, and short redacted snippets over full retrieved chunks.
+- Raw LLM input/output retention is governed by ADR-0009's
+  `llm_call_payloads` tenant policy, not by accidental Langfuse trace capture.
+- Langfuse project retention/deletion settings must align with insurance and
+  tenant contractual retention requirements.
+- If self-hosted Langfuse is used later, server-side ingestion masking can add
+  defense in depth, but it does not replace client-side masking because data
+  can still reach pre-masking ingestion infrastructure.
+
+### Feedback and scores
+
+Record explicit and implicit feedback as Langfuse scores while mirroring durable
+feedback to Postgres `run_feedback` when the application needs it.
+
+Explicit feedback:
+
+| Signal | Langfuse score name | Notes |
+|---|---|---|
+| Thumbs up/down | `user-thumbs` | Boolean score, attached to the trace/run. |
+| Optional comment | score comment/metadata | Also stored in `run_feedback.comment` if enabled. |
+| Report bad answer | `user-report` | Useful for annotation queues and support review. |
+
+Implicit feedback:
+
+| Signal | Score/tag | Meaning |
+|---|---|---|
+| Handoff triggered | `handoff-triggered` | The run ended in `status=escalate`. |
+| Regeneration/retry | `user-retry` or metadata | Negative/diagnostic signal. |
+| Citation opened | `citation-opened` | User inspected a cited source. |
+| Answer copied | `answer-copied` | Weak positive signal if frontend exposes it. |
+
+The FastAPI response for a run should include enough trace/run identity for the
+backend or frontend to submit feedback later, but no Langfuse secret key is ever
+sent to a browser. Browser-side feedback, if added, may use only a Langfuse
+public key or, preferably, route feedback through the backend.
+
+### Evaluation and quality workflow
+
+Use Langfuse beyond basic tracing:
+
+1. **Datasets** — maintain representative insurance questions, expected
+   citation behavior, ambiguous questions, out-of-scope cases, and handoff
+   cases.
+2. **Experiments** — compare prompt versions, retrieval parameters, reranking
+   thresholds, model choices, and verification thresholds before deployment.
+3. **LLM-as-judge evaluators** — score groundedness, citation correctness,
+   ambiguity handling, escalation correctness, tone/language, and refusal
+   quality. Judges are aids, not absolute truth.
+4. **Annotation queues** — send low-score traces, user reports, ungrounded
+   answers, and unnecessary escalations to human review.
+5. **Error analysis** — periodically sample traces, open-code failures, cluster
+   failure modes, and decide whether the fix belongs in retrieval, source
+   content, prompt text, graph policy, or human handoff rules.
+6. **Dashboards** — track per-tenant cost, per-node token spend, latency,
+   escalation rate, clarification rate, retrieval-insufficient rate, and
+   verification-failure rate.
+
+### Boundary with ADR-0009 Postgres tables
+
+Langfuse overlaps with parts of ADR-0009, especially LLM-call visibility,
+feedback, and run debugging. This overlap is intentional, but the systems have
+different authority.
+
+| Need | Langfuse | Postgres | Decision |
+|---|---|---|---|
+| Prompt text, versions, diffs, labels, and rollback | Yes | No | Use Langfuse as the prompt source of truth. |
+| Tenant-specific prompt content | Yes | No | Use Langfuse prompt labels or, later, tenant-namespaced prompt names. |
+| Tenant prompt-routing policy | Partly | Yes | Store trusted tenant config in `tenants.settings` or app config; fetch prompt content from Langfuse. |
+| Interactive debugging of one bad answer | Yes | Partly | Use Langfuse trace; correlate to `graph_runs.id`. |
+| Trace/session grouping | Yes | Partly | Use Langfuse `session_id=thread_id`; keep `thread_id` on `graph_runs` for ledger/audit. |
+| LLM token/cost visibility | Yes | Yes | Langfuse for observability; Postgres `llm_calls`/`llm_pricing` for billing and audit. |
+| Durable graph-run ledger | No | Yes | Keep `graph_runs`. Langfuse traces are not the application run table. |
+| User feedback | Yes | Yes | Store scores in Langfuse and durable feedback in `run_feedback` when needed by the app. |
+| Error analysis and annotation queues | Yes | Usually no | Use Langfuse; avoid building an annotation workflow in Postgres initially. |
+| Datasets, experiments, and evaluators | Yes | Usually no | Use Langfuse; copy to Postgres only if a product/compliance need appears. |
+| Tenants and API keys | No | Yes | Keep `tenants` and `api_keys` in Postgres. |
+| API request audit logs | Not enough | Yes | Keep `api_request_logs` in Postgres. |
+| Point/file mutation audit | No | Yes | Keep `point_audit_events`, `source_files`, and `ingestion_jobs` in Postgres. |
+| Controlled raw/redacted payload retention | Partly | Yes | Postgres `llm_call_payloads` plus tenant policy owns retention; Langfuse must be redacted/configured to comply. |
+| Legal retention and tenant erasure policy | Partly | Yes | Postgres owns application policy; Langfuse project retention/deletion settings must align with it. |
+
+Do not remove ADR-0009 tables because Langfuse exists. Langfuse is optimized for
+observability, prompt iteration, feedback analysis, datasets, experiments, and
+annotation workflows. Postgres is optimized for auth, audit, retention, billing,
+and application consistency.
+
+## Consequences
+
+### Positive
+- Every chat run becomes inspectable as a graph-shaped trace rather than a flat
+  log line or a pile of model-call rows.
+- Prompt versions can be changed, compared, rolled back, and linked to the
+  traces they produced without redeploying application code for every prompt
+  edit.
+- Tenant-specific prompt labels let one tenant receive a customized prompt while
+  other tenants continue using the default `production` version.
+- Scores, datasets, experiments, and annotation queues create a path from user
+  feedback to systematic quality improvement.
+- Per-node token/cost/latency data exposes where the graph is expensive or
+  slow: retrieval, generation, verification, or memory.
+- Correlating Langfuse IDs with Postgres IDs preserves both workflows:
+  interactive debugging and durable audit/billing.
+
+### Negative
+- Langfuse adds another service and another set of credentials, retention
+  policies, access controls, and operational dashboards.
+- Prompt management introduces runtime dependency on prompt fetch/caching
+  behavior. The application needs safe fallbacks if Langfuse is temporarily
+  unavailable.
+- Tenant-specific prompt labels add routing complexity. The resolver must avoid
+  caller-controlled labels and must have a clear fallback when a tenant override
+  does not exist.
+- Observability can become a privacy leak if masking is incomplete. Insurance
+  content makes this a high-consequence risk.
+- There is intentional duplication between Langfuse generation usage and
+  Postgres `llm_calls`; implementations must avoid reconciling them as if one
+  were authoritative for the other's purpose.
+- LLM-as-judge evaluators can be wrong. They should guide review and experiments,
+  not replace human analysis for high-risk policy/coverage answers.
+
+## Alternatives Considered
+
+- **Use Postgres tables only for observability**: rejected. `llm_calls` and
+  audit tables are good for billing and compliance queries, but poor for
+  nested trace inspection, prompt-version attribution, annotation queues, and
+  experiment workflows.
+- **Use Langfuse as the only usage/cost store**: rejected. Tenant billing,
+  erasure, audit, and internal consistency need application-owned tables with
+  controlled retention and foreign keys to tenants/API keys.
+- **Use Langfuse instead of ADR-0009 operational tables**: rejected. Langfuse can
+  help with traces, prompt versions, scores, datasets, experiments, and
+  annotation queues, but it should not own API-key authentication, request audit,
+  point/file mutation audit, ingestion jobs, or the internal billing ledger.
+- **Store tenant prompt text and versions in Postgres**: rejected for the initial
+  design. It would duplicate Langfuse prompt-management features and require the
+  project to build its own versioning, diffs, labels, rollback, and trace links.
+  Postgres stores tenant prompt-routing configuration only.
+- **Keep prompts only in code**: rejected for prompts that will be tuned often.
+  Code-only prompts make rollback and prompt/version-to-trace analysis harder.
+  Small static strings and schema definitions may still live in code.
+- **Fetch prompts by `latest` in production**: rejected. `latest` changes every
+  time a new version is created. Production uses a configured label such as
+  `production`; rollback is moving the label.
+- **Trace full retrieved chunks and raw files for easier debugging**: rejected.
+  It would make debugging convenient but creates unnecessary privacy and storage
+  risk. IDs, scores, metadata, and redacted snippets are enough by default.
+- **One Langfuse project per tenant**: rejected for the initial design. A single
+  project per environment with tenant metadata/tags is easier to operate and
+  compare. Revisit per-tenant projects only if contractual isolation requires
+  it.
+- **Skip Langfuse scores until evals are mature**: rejected. Simple user-thumbs
+  and handoff-triggered scores are cheap and immediately useful for filtering
+  traces and building the first annotation queues.