# 0010. Langfuse observability, prompt management, and evaluation workflow

## Status

Proposed

## Context

The chatbot now has architectural decisions for the LangGraph conversation
([0006](0006-conversational-agent-graph.md)), LangGraph thread persistence
([0007](0007-agent-persistence-threads-and-memory.md)), the FastAPI REST
boundary ([0008](0008-rest-api-and-fastapi-boundary.md)), and Postgres tables
for tenants, API keys, ingestion jobs, graph runs, LLM calls, payload retention,
and feedback ([0009](0009-postgres-sqlalchemy-alembic-schema.md)). Those
systems record *what happened* internally, but they do not provide the day-to-day
LLM observability and prompt iteration workflow needed to run an insurance
chatbot safely.

We need to answer questions like:

- Which graph node failed — `triage`, `retrieve`, `generate`, or `verify`?
- Which retrieved chunks did the model see before it answered or escalated?
- Which prompt version produced a bad answer?
- Are prompt changes improving groundedness or just reducing escalations?
- Which tenants, domains, or routes are most expensive?
- Which failures should be fixed by better retrieval, better source content,
  better prompts, or stricter handoff policy?

Langfuse is a good fit for the observability and prompt/evaluation layer. It
provides traces, nested observations, generation token/cost tracking, prompt
version management, scores/user feedback, datasets, experiments, evaluators,
and annotation workflows. It should not replace the application database:
Postgres remains the source of truth for API-key authentication, tenant/audit
records, ingestion jobs, graph run rows, billing/cost ledgers, and retention
policy enforcement.

Current Langfuse guidance relevant to this project:

- Use the current Langfuse Python v3 SDK.
- For LangGraph/LangChain, use `from langfuse.langchain import CallbackHandler`
  and pass the handler in the graph invocation config.
- Use one trace per user-facing chatbot run.
- Use stable, low-cardinality trace and observation names.
- Model calls should appear as generation observations so model, token, latency,
  and cost information can attach correctly.
- Conversation grouping should use Langfuse sessions; in this service,
  LangGraph `thread_id` is the Langfuse `session_id`.
- Prompt management uses prompt versions and labels (for example `development`,
  `staging`, `production`, tenant-specific labels, and automatically maintained
  `latest`). Labels can represent environments, tenants, or experiments.
- Scores represent explicit feedback and implicit/evaluator signals.
- Sensitive data should be masked before export, especially in insurance.

## Decision

### Langfuse is the observability plane, not the transactional database

Adopt Langfuse for:

- traces and nested observations;
- prompt versions, variables, labels, diffs, and trace links;
- user feedback and evaluator scores;
- datasets and experiments;
- annotation queues and systematic error analysis;
- interactive debugging and dashboards.

Keep Postgres ([0009](0009-postgres-sqlalchemy-alembic-schema.md)) as the
internal system of record for:

- tenants and API keys;
- request and point audit records;
- source files and ingestion jobs;
- `graph_runs` and `run_feedback`;
- `llm_calls`, `llm_pricing`, and optional `llm_call_payloads`;
- tenant-specific retention, erasure, and billing/reporting queries.

The two systems are correlated with stable identifiers:

| Identifier | Origin | Use |
|---|---|---|
| `request_id` | FastAPI middleware | Correlate API logs, Postgres audit rows, Langfuse trace metadata. |
| `tenant_id` | API-key dependency | Tenant filtering/cost attribution; never client-supplied. |
| `thread_id` | REST path / LangGraph config | LangGraph thread id and Langfuse `session_id`. |
| `run_id` | `graph_runs.id` | One `POST /v1/threads/{thread_id}/runs` execution. |
| `llm_call_id` | `llm_calls.id` | Optional correlation from a Langfuse generation to Postgres usage ledger. |
| `langfuse_trace_id` | Langfuse | Stored on `graph_runs.metadata` initially, or promoted to a typed column later if frequently queried. |
| `langfuse_observation_id` | Langfuse | Correlated by putting `llm_call_id` in Langfuse observation metadata; add a typed Postgres column later only if needed. |

### SDK and invocation pattern

Use the current Langfuse Python SDK v3 integration for LangGraph/LangChain:

```python
from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler

langfuse = get_client()
trace_tags = [environment, "chatbot", "threads:runs"]
trace_metadata = {
    "request_id": request_id,
    "tenant_id": str(tenant_id),
    "api_key_id": str(api_key_id),
    "thread_id": thread_id,
    "run_id": str(run_id),
    "graph_version": graph_version,
    "retrieval_config_version": retrieval_config_version,
    "prompt_label": prompt_label,
}

with propagate_attributes(
    trace_name="chat-run",
    user_id=external_user_id,
    session_id=thread_id,
    tags=trace_tags,
    metadata=trace_metadata,
):
    langfuse_handler = CallbackHandler()
    result = graph.invoke(
        graph_input,
        config={
            "callbacks": [langfuse_handler],
            "configurable": {
                "thread_id": thread_id,
                "tenant_id": str(tenant_id),
                "user_id": external_user_id,
            },
        },
    )
```

If the implementation sets Langfuse attributes through LangChain/LangGraph
`config["metadata"]` instead of or in addition to `propagate_attributes`, use the
current integration keys `langfuse_user_id`, `langfuse_session_id`, and
`langfuse_tags` for user, session, and tag propagation.

For non-LangChain/LangGraph work that should appear in the same trace — FastAPI
request spans, Qdrant retrieval, custom reranking metadata, or explicit audit
steps — use Langfuse/OpenTelemetry spans around that work rather than relying
only on automatic LLM callbacks.

Short-lived scripts and tests call `langfuse.flush()` or `langfuse.shutdown()`
before exit. Long-lived FastAPI workers flush during graceful shutdown.

### Trace shape

Create one Langfuse trace per `POST /v1/threads/{thread_id}/runs` call.

Trace-level fields:

| Field | Value |
|---|---|
| name | `chat-run` |
| input | The latest user message, after request-level redaction. |
| output | Final assistant answer, clarifying question, or escalation message. |
| user_id | External `user_id` supplied by the main backend. |
| session_id | LangGraph `thread_id`. |
| tags | Environment, feature (`chatbot`), route (`threads:runs`), tenant segment if safe, terminal status. |
| metadata | `request_id`, `tenant_id`, `api_key_id`, `thread_id`, `run_id`, graph version, retrieval config version, prompt label. |

Observation hierarchy mirrors ADR-0006's graph instead of producing a flat list:

| Observation | Type | Contents |
|---|---|---|
| `load-memory` | span | Store lookup latency and memory keys, not sensitive values. |
| `triage` | generation | Intent classification prompt/output. |
| `contextualize` | generation | Query rewrite input/output. |
| `retrieve` | retriever/span | Qdrant collection, vector names, filters, top-k, returned point/chunk IDs, scores, and source metadata. Avoid raw chunk text unless redacted. |
| `grade` | generation | Relevance verdicts over retrieved chunks. |
| `clarify` | generation | Grounded clarification question when used. |
| `generate` | generation | Final answer generation with citation IDs. |
| `verify` | generation | Groundedness/citation verification result. |
| `escalate` | span | Escalation reason and handoff context summary. |
| `write-memory` | span | Memory write keys and outcome, not sensitive facts. |

Observation names stay stable and low-cardinality. Dynamic values such as tenant
slug, file id, model name, retry number, point id, and user text go in metadata
or input/output fields, not in observation names.

Every generation should include model name, input tokens, output tokens,
latency, and cost when available. Langfuse provides the interactive view;
`llm_calls` in Postgres remains the billing/audit ledger.

### Prompt management and tenant-specific variants

Manage prompts in Langfuse for graph nodes whose behavior will be tuned over
time:

- `insurance-chatbot/triage`
- `insurance-chatbot/contextualize`
- `insurance-chatbot/grade-retrieval`
- `insurance-chatbot/clarify`
- `insurance-chatbot/generate-answer`
- `insurance-chatbot/verify-grounding`
- `insurance-chatbot/summarize-thread`
- `insurance-chatbot/extract-memory`

Prompt text, versions, diffs, labels, and rollback live in Langfuse. Tenant
identity and the policy that chooses which prompt label/name to fetch live in the
application and, where tenant-specific configuration is needed, Postgres
`tenants.settings`.

Prompt names are stable and low-cardinality. The initial tenant-customization
model uses shared prompt names plus tenant-specific labels:

| Label | Use |
|---|---|
| `development` | Local/dev experiments. |
| `staging` | Pre-production validation and experiments. |
| `production` | Default live traffic prompt when no tenant-specific override exists. |
| `tenant-{tenant_slug}-{environment}` | Tenant-specific prompt version for one environment, for example `tenant-acme-production`. |
| `latest` | Automatically points to newest version; useful for discovery, not production runtime. |

Langfuse labels can represent environments, tenants, or experiments. The service
therefore uses labels to support different live prompts per tenant without
creating an application-owned prompt-version table.

Tenant-specific prompt resolution is handled in the application with an explicit
fallback order:

1. derive `tenant_slug` from the authenticated `AuthContext`, never from a
   request body/query parameter;
2. derive the environment label from trusted configuration, for example
   `production` or `staging`;
3. try the tenant/environment label, for example `tenant-acme-production`;
4. fall back to the environment label, for example `production`;
5. fail closed or use a code-packaged emergency prompt only if the configured
   prompt cannot be loaded.

Indicative resolver shape:

```python
prompt_name = "insurance-chatbot/generate-answer"
environment_label = settings.langfuse_prompt_label  # e.g. "production"
tenant_label = f"tenant-{auth.tenant_slug}-{environment_label}"

prompt = prompt_resolver.get_with_fallback(
    name=prompt_name,
    labels=[tenant_label, environment_label],
)
compiled_prompt = prompt.compile(
    language=language,
    user_message=user_message,
    conversation_summary=conversation_summary,
    retrieved_chunks=retrieved_chunks,
    tenant_policy=tenant_policy,
    handoff_policy=handoff_policy,
)
```

The exact SDK call and not-found error handling should follow the current
Langfuse Python SDK, but the architectural rule is stable: select the tenant
label from trusted server-side tenant context, not from caller input.

This keeps prompt content, prompt versions, prompt diffs, and label movement in
Langfuse while keeping tenant identity and resolver policy in the application.
If tenant-specific prompts become large and independently owned, an acceptable
future variant is tenant-namespaced prompt names such as
`insurance-chatbot/tenants/acme/generate-answer` with ordinary `production` and
`staging` labels. The initial design prefers shared prompt names plus
tenant-specific labels because it keeps graph-node prompts easier to compare
across tenants.

Postgres may store prompt resolver configuration in `tenants.settings`, for
example an enabled flag, preferred prompt namespace, explicit label override,
default language, tenant policy reference, or handoff policy reference. Postgres
must not become the source of truth for prompt text/version history; that
belongs in Langfuse.

Prompt variables carry dynamic context, for example:

- `{{language}}`
- `{{user_message}}`
- `{{conversation_summary}}`
- `{{retrieved_chunks}}`
- `{{tenant_policy}}`
- `{{handoff_policy}}`
- `{{citation_contract}}`
- `{{memory_profile}}`

Prompt versions are linked to traces so failures can be traced back to the exact
prompt version that produced them. Rollback is performed by moving the relevant
label — `production` for the default prompt, or `tenant-acme-production` for one
tenant's override — back to a known-good version, not by redeploying code.

Prompt changes are evaluated before promotion using Langfuse datasets and
experiments. A prompt version should not be promoted to `production` or to a
tenant-specific production label only because one manual test looked good.

### Privacy, redaction, and retention

Insurance conversations can contain PII, policy details, claim information, and
financial/health-adjacent facts. Langfuse is therefore sensitive infrastructure.

Rules:

- Mask/redact before export from the application process. Prefer current SDK
  masking support such as `mask_otel_spans` where applicable, plus deterministic
  field/path redaction for known sensitive fields.
- Never put API keys, database URLs, provider secrets, raw auth headers, or
  unredacted request bodies in trace metadata.
- Do not trace raw uploaded files or full document text.
- Retrieval observations should prefer point/chunk IDs, file IDs, scores,
  titles, and short redacted snippets over full retrieved chunks.
- Raw LLM input/output retention is governed by ADR-0009's
  `llm_call_payloads` tenant policy, not by accidental Langfuse trace capture.
- Langfuse project retention/deletion settings must align with insurance and
  tenant contractual retention requirements.
- If self-hosted Langfuse is used later, server-side ingestion masking can add
  defense in depth, but it does not replace client-side masking because data
  can still reach pre-masking ingestion infrastructure.

### Feedback and scores

Record explicit and implicit feedback as Langfuse scores while mirroring durable
feedback to Postgres `run_feedback` when the application needs it.

Explicit feedback:

| Signal | Langfuse score name | Notes |
|---|---|---|
| Thumbs up/down | `user-thumbs` | Boolean score, attached to the trace/run. |
| Optional comment | score comment/metadata | Also stored in `run_feedback.comment` if enabled. |
| Report bad answer | `user-report` | Useful for annotation queues and support review. |

Implicit feedback:

| Signal | Score/tag | Meaning |
|---|---|---|
| Handoff triggered | `handoff-triggered` | The run ended in `status=escalate`. |
| Regeneration/retry | `user-retry` or metadata | Negative/diagnostic signal. |
| Citation opened | `citation-opened` | User inspected a cited source. |
| Answer copied | `answer-copied` | Weak positive signal if frontend exposes it. |

The FastAPI response for a run should include enough trace/run identity for the
backend or frontend to submit feedback later, but no Langfuse secret key is ever
sent to a browser. Browser-side feedback, if added, may use only a Langfuse
public key or, preferably, route feedback through the backend.

### Evaluation and quality workflow

Use Langfuse beyond basic tracing:

1. **Datasets** — maintain representative insurance questions, expected
   citation behavior, ambiguous questions, out-of-scope cases, and handoff
   cases.
2. **Experiments** — compare prompt versions, retrieval parameters, reranking
   thresholds, model choices, and verification thresholds before deployment.
3. **LLM-as-judge evaluators** — score groundedness, citation correctness,
   ambiguity handling, escalation correctness, tone/language, and refusal
   quality. Judges are aids, not absolute truth.
4. **Annotation queues** — send low-score traces, user reports, ungrounded
   answers, and unnecessary escalations to human review.
5. **Error analysis** — periodically sample traces, open-code failures, cluster
   failure modes, and decide whether the fix belongs in retrieval, source
   content, prompt text, graph policy, or human handoff rules.
6. **Dashboards** — track per-tenant cost, per-node token spend, latency,
   escalation rate, clarification rate, retrieval-insufficient rate, and
   verification-failure rate.

### Boundary with ADR-0009 Postgres tables

Langfuse overlaps with parts of ADR-0009, especially LLM-call visibility,
feedback, and run debugging. This overlap is intentional, but the systems have
different authority.

| Need | Langfuse | Postgres | Decision |
|---|---|---|---|
| Prompt text, versions, diffs, labels, and rollback | Yes | No | Use Langfuse as the prompt source of truth. |
| Tenant-specific prompt content | Yes | No | Use Langfuse prompt labels or, later, tenant-namespaced prompt names. |
| Tenant prompt-routing policy | Partly | Yes | Store trusted tenant config in `tenants.settings` or app config; fetch prompt content from Langfuse. |
| Interactive debugging of one bad answer | Yes | Partly | Use Langfuse trace; correlate to `graph_runs.id`. |
| Trace/session grouping | Yes | Partly | Use Langfuse `session_id=thread_id`; keep `thread_id` on `graph_runs` for ledger/audit. |
| LLM token/cost visibility | Yes | Yes | Langfuse for observability; Postgres `llm_calls`/`llm_pricing` for billing and audit. |
| Durable graph-run ledger | No | Yes | Keep `graph_runs`. Langfuse traces are not the application run table. |
| User feedback | Yes | Yes | Store scores in Langfuse and durable feedback in `run_feedback` when needed by the app. |
| Error analysis and annotation queues | Yes | Usually no | Use Langfuse; avoid building an annotation workflow in Postgres initially. |
| Datasets, experiments, and evaluators | Yes | Usually no | Use Langfuse; copy to Postgres only if a product/compliance need appears. |
| Tenants and API keys | No | Yes | Keep `tenants` and `api_keys` in Postgres. |
| API request audit logs | Not enough | Yes | Keep `api_request_logs` in Postgres. |
| Point/file mutation audit | No | Yes | Keep `point_audit_events`, `source_files`, and `ingestion_jobs` in Postgres. |
| Controlled raw/redacted payload retention | Partly | Yes | Postgres `llm_call_payloads` plus tenant policy owns retention; Langfuse must be redacted/configured to comply. |
| Legal retention and tenant erasure policy | Partly | Yes | Postgres owns application policy; Langfuse project retention/deletion settings must align with it. |

Do not remove ADR-0009 tables because Langfuse exists. Langfuse is optimized for
observability, prompt iteration, feedback analysis, datasets, experiments, and
annotation workflows. Postgres is optimized for auth, audit, retention, billing,
and application consistency.

## Consequences

### Positive
- Every chat run becomes inspectable as a graph-shaped trace rather than a flat
  log line or a pile of model-call rows.
- Prompt versions can be changed, compared, rolled back, and linked to the
  traces they produced without redeploying application code for every prompt
  edit.
- Tenant-specific prompt labels let one tenant receive a customized prompt while
  other tenants continue using the default `production` version.
- Scores, datasets, experiments, and annotation queues create a path from user
  feedback to systematic quality improvement.
- Per-node token/cost/latency data exposes where the graph is expensive or
  slow: retrieval, generation, verification, or memory.
- Correlating Langfuse IDs with Postgres IDs preserves both workflows:
  interactive debugging and durable audit/billing.

### Negative
- Langfuse adds another service and another set of credentials, retention
  policies, access controls, and operational dashboards.
- Prompt management introduces runtime dependency on prompt fetch/caching
  behavior. The application needs safe fallbacks if Langfuse is temporarily
  unavailable.
- Tenant-specific prompt labels add routing complexity. The resolver must avoid
  caller-controlled labels and must have a clear fallback when a tenant override
  does not exist.
- Observability can become a privacy leak if masking is incomplete. Insurance
  content makes this a high-consequence risk.
- There is intentional duplication between Langfuse generation usage and
  Postgres `llm_calls`; implementations must avoid reconciling them as if one
  were authoritative for the other's purpose.
- LLM-as-judge evaluators can be wrong. They should guide review and experiments,
  not replace human analysis for high-risk policy/coverage answers.

## Alternatives Considered

- **Use Postgres tables only for observability**: rejected. `llm_calls` and
  audit tables are good for billing and compliance queries, but poor for
  nested trace inspection, prompt-version attribution, annotation queues, and
  experiment workflows.
- **Use Langfuse as the only usage/cost store**: rejected. Tenant billing,
  erasure, audit, and internal consistency need application-owned tables with
  controlled retention and foreign keys to tenants/API keys.
- **Use Langfuse instead of ADR-0009 operational tables**: rejected. Langfuse can
  help with traces, prompt versions, scores, datasets, experiments, and
  annotation queues, but it should not own API-key authentication, request audit,
  point/file mutation audit, ingestion jobs, or the internal billing ledger.
- **Store tenant prompt text and versions in Postgres**: rejected for the initial
  design. It would duplicate Langfuse prompt-management features and require the
  project to build its own versioning, diffs, labels, rollback, and trace links.
  Postgres stores tenant prompt-routing configuration only.
- **Keep prompts only in code**: rejected for prompts that will be tuned often.
  Code-only prompts make rollback and prompt/version-to-trace analysis harder.
  Small static strings and schema definitions may still live in code.
- **Fetch prompts by `latest` in production**: rejected. `latest` changes every
  time a new version is created. Production uses a configured label such as
  `production`; rollback is moving the label.
- **Trace full retrieved chunks and raw files for easier debugging**: rejected.
  It would make debugging convenient but creates unnecessary privacy and storage
  risk. IDs, scores, metadata, and redacted snippets are enough by default.
- **One Langfuse project per tenant**: rejected for the initial design. A single
  project per environment with tenant metadata/tags is easier to operate and
  compare. Revisit per-tenant projects only if contractual isolation requires
  it.
- **Skip Langfuse scores until evals are mature**: rejected. Simple user-thumbs
  and handoff-triggered scores are cheap and immediately useful for filtering
  traces and building the first annotation queues.