chatbot_v3/docs/adr/0010-langfuse-observability-and-prompt-management.md

0010. Langfuse observability, prompt management, and evaluation workflow

Status

Proposed

Context

The chatbot now has architectural decisions for the LangGraph conversation (0006), LangGraph thread persistence (0007), the FastAPI REST boundary (0008), and Postgres tables for tenants, API keys, ingestion jobs, graph runs, LLM calls, payload retention, and feedback (0009). Those systems record what happened internally, but they do not provide the day-to-day LLM observability and prompt iteration workflow needed to run an insurance chatbot safely.

We need to answer questions like:

• Which graph node failed — triage, retrieve, generate, or verify?
• Which retrieved chunks did the model see before it answered or escalated?
• Which prompt version produced a bad answer?
• Are prompt changes improving groundedness or just reducing escalations?
• Which tenants, domains, or routes are most expensive?
• Which failures should be fixed by better retrieval, better source content, better prompts, or stricter handoff policy?

Langfuse is a good fit for the observability and prompt/evaluation layer. It provides traces, nested observations, generation token/cost tracking, prompt version management, scores/user feedback, datasets, experiments, evaluators, and annotation workflows. It should not replace the application database: Postgres remains the source of truth for API-key authentication, tenant/audit records, ingestion jobs, graph run rows, billing/cost ledgers, and retention policy enforcement.

Current Langfuse guidance relevant to this project:

• Use the current Langfuse Python v3 SDK.
• For LangGraph/LangChain, use from langfuse.langchain import CallbackHandler and pass the handler in the graph invocation config.
• Use one trace per user-facing chatbot run.
• Use stable, low-cardinality trace and observation names.
• Model calls should appear as generation observations so model, token, latency, and cost information can attach correctly.
• Conversation grouping should use Langfuse sessions; in this service, LangGraph thread_id is the Langfuse session_id.
• Prompt management uses prompt versions and labels (for example development, staging, production, tenant-specific labels, and automatically maintained latest). Labels can represent environments, tenants, or experiments.
• Scores represent explicit feedback and implicit/evaluator signals.
• Sensitive data should be masked before export, especially in insurance.

Decision

Langfuse is the observability plane, not the transactional database

Adopt Langfuse for:

• traces and nested observations;
• prompt versions, variables, labels, diffs, and trace links;
• user feedback and evaluator scores;
• datasets and experiments;
• annotation queues and systematic error analysis;
• interactive debugging and dashboards.

Keep Postgres (0009) as the internal system of record for:

• tenants and API keys;
• request and point audit records;
• source files and ingestion jobs;
• graph_runs and run_feedback;
• llm_calls, llm_pricing, and optional llm_call_payloads;
• tenant-specific retention, erasure, and billing/reporting queries.

The two systems are correlated with stable identifiers:

Identifier	Origin	Use
request_id	FastAPI middleware	Correlate API logs, Postgres audit rows, Langfuse trace metadata.
tenant_id	API-key dependency	Tenant filtering/cost attribution; never client-supplied.
thread_id	REST path / LangGraph config	LangGraph thread id and Langfuse session_id.
run_id	graph_runs.id	One POST /v1/threads/{thread_id}/runs execution.
llm_call_id	llm_calls.id	Optional correlation from a Langfuse generation to Postgres usage ledger.
langfuse_trace_id	Langfuse	Stored on graph_runs.metadata initially, or promoted to a typed column later if frequently queried.
langfuse_observation_id	Langfuse	Correlated by putting llm_call_id in Langfuse observation metadata; add a typed Postgres column later only if needed.

SDK and invocation pattern

Use the current Langfuse Python SDK v3 integration for LangGraph/LangChain:

from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler

langfuse = get_client()
trace_tags = [environment, "chatbot", "threads:runs"]
trace_metadata = {
    "request_id": request_id,
    "tenant_id": str(tenant_id),
    "api_key_id": str(api_key_id),
    "thread_id": thread_id,
    "run_id": str(run_id),
    "graph_version": graph_version,
    "retrieval_config_version": retrieval_config_version,
    "prompt_label": prompt_label,
}

with propagate_attributes(
    trace_name="chat-run",
    user_id=external_user_id,
    session_id=thread_id,
    tags=trace_tags,
    metadata=trace_metadata,
):
    langfuse_handler = CallbackHandler()
    result = graph.invoke(
        graph_input,
        config={
            "callbacks": [langfuse_handler],
            "configurable": {
                "thread_id": thread_id,
                "tenant_id": str(tenant_id),
                "user_id": external_user_id,
            },
        },
    )

If the implementation sets Langfuse attributes through LangChain/LangGraph config["metadata"] instead of or in addition to propagate_attributes, use the current integration keys langfuse_user_id, langfuse_session_id, and langfuse_tags for user, session, and tag propagation.

For non-LangChain/LangGraph work that should appear in the same trace — FastAPI request spans, Qdrant retrieval, custom reranking metadata, or explicit audit steps — use Langfuse/OpenTelemetry spans around that work rather than relying only on automatic LLM callbacks.

Short-lived scripts and tests call langfuse.flush() or langfuse.shutdown() before exit. Long-lived FastAPI workers flush during graceful shutdown.

Trace shape

Create one Langfuse trace per POST /v1/threads/{thread_id}/runs call.

Trace-level fields:

Field	Value
name	chat-run
input	The latest user message, after request-level redaction.
output	Final assistant answer, clarifying question, or escalation message.
user_id	External user_id supplied by the main backend.
session_id	LangGraph thread_id.
tags	Environment, feature (chatbot), route (threads:runs), tenant segment if safe, terminal status.
metadata	request_id, tenant_id, api_key_id, thread_id, run_id, graph version, retrieval config version, prompt label.

Observation hierarchy mirrors ADR-0006's graph instead of producing a flat list:

Observation	Type	Contents
load-memory	span	Store lookup latency and memory keys, not sensitive values.
triage	generation	Intent classification prompt/output.
contextualize	generation	Query rewrite input/output.
retrieve	retriever/span	Qdrant collection, vector names, filters, top-k, returned point/chunk IDs, scores, and source metadata. Avoid raw chunk text unless redacted.
grade	generation	Relevance verdicts over retrieved chunks.
clarify	generation	Grounded clarification question when used.
generate	generation	Final answer generation with citation IDs.
verify	generation	Groundedness/citation verification result.
escalate	span	Escalation reason and handoff context summary.
write-memory	span	Memory write keys and outcome, not sensitive facts.

Observation names stay stable and low-cardinality. Dynamic values such as tenant slug, file id, model name, retry number, point id, and user text go in metadata or input/output fields, not in observation names.

Every generation should include model name, input tokens, output tokens, latency, and cost when available. Langfuse provides the interactive view; llm_calls in Postgres remains the billing/audit ledger.

Prompt management and tenant-specific variants

Manage prompts in Langfuse for graph nodes whose behavior will be tuned over time:

• insurance-chatbot/triage
• insurance-chatbot/contextualize
• insurance-chatbot/grade-retrieval
• insurance-chatbot/clarify
• insurance-chatbot/generate-answer
• insurance-chatbot/verify-grounding
• insurance-chatbot/summarize-thread
• insurance-chatbot/extract-memory

Prompt text, versions, diffs, labels, and rollback live in Langfuse. Tenant identity and the policy that chooses which prompt label/name to fetch live in the application and, where tenant-specific configuration is needed, Postgres tenants.settings.

Prompt names are stable and low-cardinality. The initial tenant-customization model uses shared prompt names plus tenant-specific labels:

Label	Use
development	Local/dev experiments.
staging	Pre-production validation and experiments.
production	Default live traffic prompt when no tenant-specific override exists.
tenant-{tenant_slug}-{environment}	Tenant-specific prompt version for one environment, for example tenant-acme-production.
latest	Automatically points to newest version; useful for discovery, not production runtime.

Langfuse labels can represent environments, tenants, or experiments. The service therefore uses labels to support different live prompts per tenant without creating an application-owned prompt-version table.

Tenant-specific prompt resolution is handled in the application with an explicit fallback order:

1. derive tenant_slug from the authenticated AuthContext, never from a request body/query parameter;
2. derive the environment label from trusted configuration, for example production or staging;
3. try the tenant/environment label, for example tenant-acme-production;
4. fall back to the environment label, for example production;
5. fail closed or use a code-packaged emergency prompt only if the configured prompt cannot be loaded.

Indicative resolver shape:

prompt_name = "insurance-chatbot/generate-answer"
environment_label = settings.langfuse_prompt_label  # e.g. "production"
tenant_label = f"tenant-{auth.tenant_slug}-{environment_label}"

prompt = prompt_resolver.get_with_fallback(
    name=prompt_name,
    labels=[tenant_label, environment_label],
)
compiled_prompt = prompt.compile(
    language=language,
    user_message=user_message,
    conversation_summary=conversation_summary,
    retrieved_chunks=retrieved_chunks,
    tenant_policy=tenant_policy,
    handoff_policy=handoff_policy,
)

The exact SDK call and not-found error handling should follow the current Langfuse Python SDK, but the architectural rule is stable: select the tenant label from trusted server-side tenant context, not from caller input.

This keeps prompt content, prompt versions, prompt diffs, and label movement in Langfuse while keeping tenant identity and resolver policy in the application. If tenant-specific prompts become large and independently owned, an acceptable future variant is tenant-namespaced prompt names such as insurance-chatbot/tenants/acme/generate-answer with ordinary production and staging labels. The initial design prefers shared prompt names plus tenant-specific labels because it keeps graph-node prompts easier to compare across tenants.

Postgres may store prompt resolver configuration in tenants.settings, for example an enabled flag, preferred prompt namespace, explicit label override, default language, tenant policy reference, or handoff policy reference. Postgres must not become the source of truth for prompt text/version history; that belongs in Langfuse.

Prompt variables carry dynamic context, for example:

• {{language}}
• {{user_message}}
• {{conversation_summary}}
• {{retrieved_chunks}}
• {{tenant_policy}}
• {{handoff_policy}}
• {{citation_contract}}
• {{memory_profile}}

Prompt versions are linked to traces so failures can be traced back to the exact prompt version that produced them. Rollback is performed by moving the relevant label — production for the default prompt, or tenant-acme-production for one tenant's override — back to a known-good version, not by redeploying code.

Prompt changes are evaluated before promotion using Langfuse datasets and experiments. A prompt version should not be promoted to production or to a tenant-specific production label only because one manual test looked good.

Privacy, redaction, and retention

Insurance conversations can contain PII, policy details, claim information, and financial/health-adjacent facts. Langfuse is therefore sensitive infrastructure.

Rules:

• Mask/redact before export from the application process. Prefer current SDK masking support such as mask_otel_spans where applicable, plus deterministic field/path redaction for known sensitive fields.
• Never put API keys, database URLs, provider secrets, raw auth headers, or unredacted request bodies in trace metadata.
• Do not trace raw uploaded files or full document text.
• Retrieval observations should prefer point/chunk IDs, file IDs, scores, titles, and short redacted snippets over full retrieved chunks.
• Raw LLM input/output retention is governed by ADR-0009's llm_call_payloads tenant policy, not by accidental Langfuse trace capture.
• Langfuse project retention/deletion settings must align with insurance and tenant contractual retention requirements.
• If self-hosted Langfuse is used later, server-side ingestion masking can add defense in depth, but it does not replace client-side masking because data can still reach pre-masking ingestion infrastructure.

Feedback and scores

Record explicit and implicit feedback as Langfuse scores while mirroring durable feedback to Postgres run_feedback when the application needs it.

Explicit feedback:

Signal	Langfuse score name	Notes
Thumbs up/down	user-thumbs	Boolean score, attached to the trace/run.
Optional comment	score comment/metadata	Also stored in run_feedback.comment if enabled.
Report bad answer	user-report	Useful for annotation queues and support review.

Implicit feedback:

Signal	Score/tag	Meaning
Handoff triggered	handoff-triggered	The run ended in status=escalate.
Regeneration/retry	user-retry or metadata	Negative/diagnostic signal.
Citation opened	citation-opened	User inspected a cited source.
Answer copied	answer-copied	Weak positive signal if frontend exposes it.

The FastAPI response for a run should include enough trace/run identity for the backend or frontend to submit feedback later, but no Langfuse secret key is ever sent to a browser. Browser-side feedback, if added, may use only a Langfuse public key or, preferably, route feedback through the backend.

Evaluation and quality workflow

Use Langfuse beyond basic tracing:

1. Datasets — maintain representative insurance questions, expected citation behavior, ambiguous questions, out-of-scope cases, and handoff cases.
2. Experiments — compare prompt versions, retrieval parameters, reranking thresholds, model choices, and verification thresholds before deployment.
3. LLM-as-judge evaluators — score groundedness, citation correctness, ambiguity handling, escalation correctness, tone/language, and refusal quality. Judges are aids, not absolute truth.
4. Annotation queues — send low-score traces, user reports, ungrounded answers, and unnecessary escalations to human review.
5. Error analysis — periodically sample traces, open-code failures, cluster failure modes, and decide whether the fix belongs in retrieval, source content, prompt text, graph policy, or human handoff rules.
6. Dashboards — track per-tenant cost, per-node token spend, latency, escalation rate, clarification rate, retrieval-insufficient rate, and verification-failure rate.

Boundary with ADR-0009 Postgres tables

Langfuse overlaps with parts of ADR-0009, especially LLM-call visibility, feedback, and run debugging. This overlap is intentional, but the systems have different authority.

Need	Langfuse	Postgres	Decision
Prompt text, versions, diffs, labels, and rollback	Yes	No	Use Langfuse as the prompt source of truth.
Tenant-specific prompt content	Yes	No	Use Langfuse prompt labels or, later, tenant-namespaced prompt names.
Tenant prompt-routing policy	Partly	Yes	Store trusted tenant config in tenants.settings or app config; fetch prompt content from Langfuse.
Interactive debugging of one bad answer	Yes	Partly	Use Langfuse trace; correlate to graph_runs.id.
Trace/session grouping	Yes	Partly	Use Langfuse session_id=thread_id; keep thread_id on graph_runs for ledger/audit.
LLM token/cost visibility	Yes	Yes	Langfuse for observability; Postgres llm_calls/llm_pricing for billing and audit.
Durable graph-run ledger	No	Yes	Keep graph_runs. Langfuse traces are not the application run table.
User feedback	Yes	Yes	Store scores in Langfuse and durable feedback in run_feedback when needed by the app.
Error analysis and annotation queues	Yes	Usually no	Use Langfuse; avoid building an annotation workflow in Postgres initially.
Datasets, experiments, and evaluators	Yes	Usually no	Use Langfuse; copy to Postgres only if a product/compliance need appears.
Tenants and API keys	No	Yes	Keep tenants and api_keys in Postgres.
API request audit logs	Not enough	Yes	Keep api_request_logs in Postgres.
Point/file mutation audit	No	Yes	Keep point_audit_events, source_files, and ingestion_jobs in Postgres.
Controlled raw/redacted payload retention	Partly	Yes	Postgres llm_call_payloads plus tenant policy owns retention; Langfuse must be redacted/configured to comply.
Legal retention and tenant erasure policy	Partly	Yes	Postgres owns application policy; Langfuse project retention/deletion settings must align with it.

Do not remove ADR-0009 tables because Langfuse exists. Langfuse is optimized for observability, prompt iteration, feedback analysis, datasets, experiments, and annotation workflows. Postgres is optimized for auth, audit, retention, billing, and application consistency.

Consequences

Positive

• Every chat run becomes inspectable as a graph-shaped trace rather than a flat log line or a pile of model-call rows.
• Prompt versions can be changed, compared, rolled back, and linked to the traces they produced without redeploying application code for every prompt edit.
• Tenant-specific prompt labels let one tenant receive a customized prompt while other tenants continue using the default production version.
• Scores, datasets, experiments, and annotation queues create a path from user feedback to systematic quality improvement.
• Per-node token/cost/latency data exposes where the graph is expensive or slow: retrieval, generation, verification, or memory.
• Correlating Langfuse IDs with Postgres IDs preserves both workflows: interactive debugging and durable audit/billing.

Negative

• Langfuse adds another service and another set of credentials, retention policies, access controls, and operational dashboards.
• Prompt management introduces runtime dependency on prompt fetch/caching behavior. The application needs safe fallbacks if Langfuse is temporarily unavailable.
• Tenant-specific prompt labels add routing complexity. The resolver must avoid caller-controlled labels and must have a clear fallback when a tenant override does not exist.
• Observability can become a privacy leak if masking is incomplete. Insurance content makes this a high-consequence risk.
• There is intentional duplication between Langfuse generation usage and Postgres llm_calls; implementations must avoid reconciling them as if one were authoritative for the other's purpose.
• LLM-as-judge evaluators can be wrong. They should guide review and experiments, not replace human analysis for high-risk policy/coverage answers.

Alternatives Considered

• Use Postgres tables only for observability: rejected. llm_calls and audit tables are good for billing and compliance queries, but poor for nested trace inspection, prompt-version attribution, annotation queues, and experiment workflows.
• Use Langfuse as the only usage/cost store: rejected. Tenant billing, erasure, audit, and internal consistency need application-owned tables with controlled retention and foreign keys to tenants/API keys.
• Use Langfuse instead of ADR-0009 operational tables: rejected. Langfuse can help with traces, prompt versions, scores, datasets, experiments, and annotation queues, but it should not own API-key authentication, request audit, point/file mutation audit, ingestion jobs, or the internal billing ledger.
• Store tenant prompt text and versions in Postgres: rejected for the initial design. It would duplicate Langfuse prompt-management features and require the project to build its own versioning, diffs, labels, rollback, and trace links. Postgres stores tenant prompt-routing configuration only.
• Keep prompts only in code: rejected for prompts that will be tuned often. Code-only prompts make rollback and prompt/version-to-trace analysis harder. Small static strings and schema definitions may still live in code.
• Fetch prompts by latest in production: rejected. latest changes every time a new version is created. Production uses a configured label such as production; rollback is moving the label.
• Trace full retrieved chunks and raw files for easier debugging: rejected. It would make debugging convenient but creates unnecessary privacy and storage risk. IDs, scores, metadata, and redacted snippets are enough by default.
• One Langfuse project per tenant: rejected for the initial design. A single project per environment with tenant metadata/tags is easier to operate and compare. Revisit per-tenant projects only if contractual isolation requires it.
• Skip Langfuse scores until evals are mature: rejected. Simple user-thumbs and handoff-triggered scores are cheap and immediately useful for filtering traces and building the first annotation queues.