chatbot_v3/docs/adr/0006-conversational-agent-graph.md

0006. Conversational agent graph (LangGraph): grounding, clarification, and handoff signalling

Status

Proposed

Context

ADRs 0001–0005 define how insurance documents get into Qdrant and how the best chunks come back out (0003). Nothing yet defines the conversation on top of that retrieval: what asks the question, what decides the retrieved chunks are good enough, what happens when they aren't, and what happens when the user should be talking to a person instead of a model.

The product requirements that shape this decision:

• Agentic, not a fixed RAG chain. A single "embed → retrieve → stuff into a prompt" pass is not enough. Follow-up questions ("what about for my wife?") are unanswerable without rewriting the query against conversation history, and one retrieval round is often the wrong round.
• Ambiguity is resolved by asking, and the question must be grounded. If the user asks "how much does it cost?", the bot must ask which policy — and the options it offers must come from what retrieval actually returned (e.g. three distinct products in the fused top-k), not from the model's imagination. An ungrounded clarifying question is a hallucination with a question mark on it.
• Handoff is decided by the agent but executed elsewhere. The existing main backend already owns human handoff: when a user clicks "connect me to a human assistant," that backend transfers the whole chat session to a human and the session never returns to the agent. This service's job is to reach the same outcome autonomously — to decide that a conversation should be transferred and say so — not to implement the transfer.
• This is insurance. A confidently wrong answer about what a policy covers is a liability, not an inconvenience. "I don't know, let me get someone" is a correct output, and the graph must be able to produce it.

LangGraph is the right layer here (rather than a LangChain create_agent loop or Deep Agents): the workflow mixes agentic steps (query rewriting, bounded retrieval retries) with deterministic gates (groundedness verification, escalation policy, loop budgets) over durable per-session state.

Decision

Overall shape: deterministic outer graph, bounded agentic core

An explicit StateGraph — not a prebuilt tool-calling agent. The model gets agency where agency helps (rewriting the retrieval query, deciding whether what came back is sufficient) and none where it doesn't (whether to escalate, whether to answer at all, how many times to retry). Escalation policy and loop budgets are graph edges, not model discretion.

Nodes

Node	Kind	Responsibility
load_memory	data	Reads the user's long-term profile from the Store (0007) into state.
triage	LLM (small)	Classifies the turn: chitchat, knowledge (answerable from the corpus), account (needs a tool + identity), handoff_request (user explicitly asked for a person), out_of_scope.
contextualize	LLM (small)	Rewrites the latest turn into a standalone retrieval query using recent history + profile. "what about for my wife?" → "third-party liability coverage for a spouse". Emits the rewrite to state; the original user text is never mutated.
retrieve	action	Calls the ADR-0003 hybrid pipeline (dual dense + sparse → RRF → jina-colbert-v2 rerank), with tenant_id/domain injected server-side, plus ADR-0003 context-window expansion. Returns chunks with scores and payload metadata.
grade	LLM (small) / heuristic	Per-chunk relevance judgement over the reranked top-k. Produces one of three verdicts: sufficient, ambiguous, insufficient.
clarify	LLM	Builds a grounded clarifying question from chunk payload metadata — the distinct domain/file_id/title values present in the retrieved set become the offered options. Ends the turn awaiting the user's next message.
generate	LLM (main)	Answers only from retrieved_chunks, with per-claim citations to chunk_id. Structured output: answer, citations[], answered: bool, confidence.
verify	LLM (small) / NLI	Groundedness check: is every claim in the draft supported by a cited chunk? Cheap guard against the failure mode that matters most here.
escalate	terminal	Produces the handoff signal (below) and ends the run. Does not perform the transfer.
write_memory	data	Extracts durable, non-sensitive user facts and writes them to the Store. Runs after a successful answer.

Edges and control flow

START
  └─ load_memory
        │
      triage
  ┌────────────┬──────────────────┼────────────────┬─────────────┐
chitchat   out_of_scope     handoff_request     account      knowledge
  │             │                 │                │             │
generate     generate         escalate         (tools)     contextualize
(no RAG)     (refusal)            │                │             │
  │             │                 ▼                └────────► retrieve ◄─┐
  └─────────────┴──────────────► END                             │       │
                                                               grade     │
                            ┌───────────────┬───────────────────┴──┐     │
                      sufficient        ambiguous            insufficient │
                            │               │                       │     │
                        generate        clarify ──► END      retries_left?─┘
                            │                                       │  yes
                         verify                                     no
                    ┌───────┴────────┐                              │
                grounded        ungrounded                          │
                    │                │                              │
               write_memory     regen_left? ──no──► escalate ◄──────┘
                    │                │ yes              │
                   END               └──► generate     END

Loop budgets (hard, enforced in state — not by the model)

Budget	Limit	On exhaustion
Retrieval rounds per turn	2	→ escalate
Clarifying questions per topic	2 consecutive	→ escalate
Regenerations after failed verify	1	→ escalate
Total LLM calls per turn	~8	→ escalate

Every budget's exhaustion path leads to a human, never to a guessed answer.

Escalation triggers

escalate is reached when any of these hold — the union is the policy:

1. The user explicitly asks for a person (triage → handoff_request).
2. Retrieval came back insufficient after the retry budget.
3. verify found the draft ungrounded after the regeneration budget.
4. generate returned answered: false or confidence below threshold.
5. Two consecutive clarifying questions failed to converge.
6. The intent is one we deliberately never automate: claim disputes, complaints, cancellations, anything with a payout or legal consequence. This is a policy list, not a model judgement.
7. Detected user frustration (repeat phrasing, explicit dissatisfaction).

Handoff is a signal, not a mechanism

escalate is an ordinary terminal node. It performs no transfer, opens no ticket, and does not use interrupt(). It returns a structured result and the run ends normally:

{
  "status": "escalate",
  "message": "<what the bot says to the user before handing off>",
  "reason": "user_requested | insufficient_retrieval | ungrounded | low_confidence | restricted_intent | budget_exhausted | frustration",
  "handoff_context": {
    "rewritten_query": "...",
    "top_chunks": [{ "chunk_id": "...", "score": 0.0, "title": "..." }],
    "conversation_summary": "...",
    "user_profile": { }
  }
}

The main backend consumes status: "escalate" and performs exactly what it already does when a user clicks the "connect me to a human assistant" button — the agent is simply another thing that can trigger that path. Once the backend transfers the session, it stops issuing runs against that thread (0007).

handoff_context is included even though this service doesn't own the handoff: it is what lets the human assistant start informed instead of reading the whole transcript. The backend may forward it to the agent console or ignore it.

The transfer is one-way. Per the current backend's behaviour, a transferred session never returns to the agent. This graph therefore holds no state about who owns the conversation — there is no bot/human mode flag and no relay path — because after escalation the graph is simply never run again on that thread. See Alternatives Considered for what would change if handback is ever introduced.

Grounding and citation contract

• generate receives only retrieved_chunks as evidence. Model prior knowledge about insurance is not evidence and the system prompt says so.
• Every factual claim carries a chunk_id citation; the frontend can resolve these to source documents via the ADR-0002 CRUD API.
• A claim that cannot be cited is not emitted — the model sets answered: false instead, which routes to escalate.
• Numbers (premiums, limits, deductibles) are quoted verbatim from chunks or computed by a deterministic tool. The model never does arithmetic on money.

Language

The corpus is Persian-first (bm25-fa-norm-stop, nomic-embed-text-v2-moe, ADR-0001/0004). The detected language of the user's turn is carried in state and the answer is produced in that language, while retrieval always runs on the contextualized query in the corpus's language.

Consequences

Positive

• Escalation is a property of the graph, not of a prompt — the agent cannot talk itself out of handing off, which is the correct default for insurance.
• Because handoff is a terminal signal rather than an interrupt, this service drops into the existing backend without changing how that backend transfers sessions today: the agent becomes a second trigger for a path that already exists.
• No interrupt() anywhere means no checkpoint-resume semantics to reason about, and none of the "code before interrupt() re-runs on resume" idempotency hazards.
• Clarifying questions are derived from real retrieved chunk metadata, so the options offered always correspond to content that actually exists.
• verify catches the highest-consequence failure mode (fluent, ungrounded coverage claims) before it reaches a user, at the cost of one small model call.
• Bounded retry loops make worst-case latency and per-turn token cost predictable, which a free-running tool loop does not.

Negative

• Five to eight model calls per turn (triage, contextualize, grade, generate, verify) is materially more latency and cost than single-shot RAG. Several are small-model calls, but the floor is higher.
• More nodes means more prompts to maintain and evaluate; each of triage, grade, and verify is its own quality surface that can regress.
• verify will produce false positives, sending answerable questions to humans and raising handoff volume. The threshold needs tuning against real traffic before it can be trusted.
• Nothing in this service prevents a further run on an already-transferred thread; correctness depends on the main backend not issuing one. If that assumption breaks, the bot will answer over a human assistant.
• The escalation policy list (trigger 6) is hard-coded product knowledge that will drift from what the business actually wants automated.

Alternatives Considered

• LangChain create_agent with retrieve as a tool: rejected as the outer structure. It gives the model discretion over exactly the decisions that must be deterministic here — when to stop retrying, whether to escalate — and offers no natural place for a groundedness gate. A bounded agentic loop inside an explicit graph gets the useful half of this without the liability.
• Deep Agents: rejected — planning, filesystem, and subagent delegation solve problems this workflow doesn't have.
• interrupt() + Command(resume=...) for handoff: rejected. It models a human replying into a paused graph run, which is only worth its complexity if the conversation later returns to the agent. It doesn't here, and the human channel is another system's responsibility.
• A mode: "bot" | "human" latch in graph state (deferred, not rejected): a persisted flag read at the top of every run, routing human-owned threads past the LLM to a relay node that only appends to the transcript. This becomes necessary the moment either (a) handback to the agent is introduced, so the bot must know what the human said, or (b) the backend can no longer guarantee it stops issuing runs after a transfer. Out of scope while transfers are one-way and backend-enforced; revisit via a new ADR if either assumption changes.
• Single-pass RAG with no grading or verification: rejected — it has no mechanism to detect that it should escalate, which is a hard product requirement, and no defence against ungrounded coverage claims.
• Escalation as a tool the model can call: rejected as the only mechanism. Kept as one trigger among several (via triage), but a model that is confidently wrong will not call an escalation tool — precisely the case where escalation matters most.
• Clarify before retrieving: rejected — the requirement is that clarification be grounded in retrieved chunks, which is impossible before the retrieval runs. A cheap first retrieval pass is what makes the clarifying question specific.