Ali Zarinkolah
50f6c33c0c
Why: - Design the LangGraph-based insurance chatbot: retrieval-grounded answering, grounded clarification, and handoff to a human assistant owned by the existing main backend. Changes: - ADR-0006: graph nodes/edges, escalation triggers, and the backend-owned handoff signal contract. - ADR-0007: Postgres checkpointer, thread model (thread_id as the shared identifier), history trimming strategy, and cross-thread user memory via the Store. Impact: - Design-only; no code changes yet.
246 lines
14 KiB
Markdown
246 lines
14 KiB
Markdown
# 0006. Conversational agent graph (LangGraph): grounding, clarification, and handoff signalling
|
||
|
||
## Status
|
||
|
||
Proposed
|
||
|
||
## Context
|
||
|
||
ADRs [0001](0001-ingestion-pipeline-and-collection-schema.md)–[0005](0005-reranking-model-and-sparse-analyzer-selection.md)
|
||
define how insurance documents get into Qdrant and how the best chunks come
|
||
back out ([0003](0003-agent-hybrid-retrieval.md)). Nothing yet defines the
|
||
*conversation* on top of that retrieval: what asks the question, what decides
|
||
the retrieved chunks are good enough, what happens when they aren't, and what
|
||
happens when the user should be talking to a person instead of a model.
|
||
|
||
The product requirements that shape this decision:
|
||
|
||
- **Agentic, not a fixed RAG chain.** A single "embed → retrieve → stuff into
|
||
a prompt" pass is not enough. Follow-up questions ("what about for my
|
||
wife?") are unanswerable without rewriting the query against conversation
|
||
history, and one retrieval round is often the wrong round.
|
||
- **Ambiguity is resolved by asking, and the question must be grounded.** If
|
||
the user asks "how much does it cost?", the bot must ask *which* policy —
|
||
and the options it offers must come from what retrieval actually returned
|
||
(e.g. three distinct products in the fused top-k), not from the model's
|
||
imagination. An ungrounded clarifying question is a hallucination with a
|
||
question mark on it.
|
||
- **Handoff is decided by the agent but executed elsewhere.** The existing
|
||
main backend already owns human handoff: when a user clicks "connect me to
|
||
a human assistant," that backend transfers the whole chat session to a human
|
||
and the session never returns to the agent. This service's job is to reach
|
||
the *same* outcome autonomously — to decide that a conversation should be
|
||
transferred and say so — not to implement the transfer.
|
||
- **This is insurance.** A confidently wrong answer about what a policy covers
|
||
is a liability, not an inconvenience. "I don't know, let me get someone" is
|
||
a *correct* output, and the graph must be able to produce it.
|
||
|
||
LangGraph is the right layer here (rather than a LangChain `create_agent`
|
||
loop or Deep Agents): the workflow mixes agentic steps (query rewriting,
|
||
bounded retrieval retries) with deterministic gates (groundedness
|
||
verification, escalation policy, loop budgets) over durable per-session state.
|
||
|
||
## Decision
|
||
|
||
### Overall shape: deterministic outer graph, bounded agentic core
|
||
|
||
An explicit `StateGraph` — not a prebuilt tool-calling agent. The model gets
|
||
agency where agency helps (rewriting the retrieval query, deciding whether
|
||
what came back is sufficient) and none where it doesn't (whether to escalate,
|
||
whether to answer at all, how many times to retry). Escalation policy and
|
||
loop budgets are graph edges, not model discretion.
|
||
|
||
### Nodes
|
||
|
||
| Node | Kind | Responsibility |
|
||
|---|---|---|
|
||
| `load_memory` | data | Reads the user's long-term profile from the Store ([0007](0007-agent-persistence-threads-and-memory.md)) into state. |
|
||
| `triage` | LLM (small) | Classifies the turn: `chitchat`, `knowledge` (answerable from the corpus), `account` (needs a tool + identity), `handoff_request` (user explicitly asked for a person), `out_of_scope`. |
|
||
| `contextualize` | LLM (small) | Rewrites the latest turn into a standalone retrieval query using recent history + profile. `"what about for my wife?"` → `"third-party liability coverage for a spouse"`. Emits the rewrite to state; the original user text is never mutated. |
|
||
| `retrieve` | action | Calls the ADR-0003 hybrid pipeline (dual dense + sparse → RRF → `jina-colbert-v2` rerank), with `tenant_id`/`domain` injected server-side, plus ADR-0003 context-window expansion. Returns chunks with scores and payload metadata. |
|
||
| `grade` | LLM (small) / heuristic | Per-chunk relevance judgement over the reranked top-k. Produces one of three verdicts: `sufficient`, `ambiguous`, `insufficient`. |
|
||
| `clarify` | LLM | Builds a grounded clarifying question **from chunk payload metadata** — the distinct `domain`/`file_id`/title values present in the retrieved set become the offered options. Ends the turn awaiting the user's next message. |
|
||
| `generate` | LLM (main) | Answers **only** from `retrieved_chunks`, with per-claim citations to `chunk_id`. Structured output: `answer`, `citations[]`, `answered: bool`, `confidence`. |
|
||
| `verify` | LLM (small) / NLI | Groundedness check: is every claim in the draft supported by a cited chunk? Cheap guard against the failure mode that matters most here. |
|
||
| `escalate` | terminal | Produces the handoff signal (below) and ends the run. Does not perform the transfer. |
|
||
| `write_memory` | data | Extracts durable, non-sensitive user facts and writes them to the Store. Runs after a successful answer. |
|
||
|
||
### Edges and control flow
|
||
|
||
```
|
||
START
|
||
└─ load_memory
|
||
│
|
||
triage
|
||
┌────────────┬──────────────────┼────────────────┬─────────────┐
|
||
chitchat out_of_scope handoff_request account knowledge
|
||
│ │ │ │ │
|
||
generate generate escalate (tools) contextualize
|
||
(no RAG) (refusal) │ │ │
|
||
│ │ ▼ └────────► retrieve ◄─┐
|
||
└─────────────┴──────────────► END │ │
|
||
grade │
|
||
┌───────────────┬───────────────────┴──┐ │
|
||
sufficient ambiguous insufficient │
|
||
│ │ │ │
|
||
generate clarify ──► END retries_left?─┘
|
||
│ │ yes
|
||
verify no
|
||
┌───────┴────────┐ │
|
||
grounded ungrounded │
|
||
│ │ │
|
||
write_memory regen_left? ──no──► escalate ◄──────┘
|
||
│ │ yes │
|
||
END └──► generate END
|
||
```
|
||
|
||
### Loop budgets (hard, enforced in state — not by the model)
|
||
|
||
| Budget | Limit | On exhaustion |
|
||
|---|---|---|
|
||
| Retrieval rounds per turn | 2 | → `escalate` |
|
||
| Clarifying questions per topic | 2 consecutive | → `escalate` |
|
||
| Regenerations after failed `verify` | 1 | → `escalate` |
|
||
| Total LLM calls per turn | ~8 | → `escalate` |
|
||
|
||
Every budget's exhaustion path leads to a human, never to a guessed answer.
|
||
|
||
### Escalation triggers
|
||
|
||
`escalate` is reached when **any** of these hold — the union is the policy:
|
||
|
||
1. The user explicitly asks for a person (`triage` → `handoff_request`).
|
||
2. Retrieval came back `insufficient` after the retry budget.
|
||
3. `verify` found the draft ungrounded after the regeneration budget.
|
||
4. `generate` returned `answered: false` or `confidence` below threshold.
|
||
5. Two consecutive clarifying questions failed to converge.
|
||
6. The intent is one we deliberately never automate: claim disputes,
|
||
complaints, cancellations, anything with a payout or legal consequence.
|
||
This is a policy list, not a model judgement.
|
||
7. Detected user frustration (repeat phrasing, explicit dissatisfaction).
|
||
|
||
### Handoff is a signal, not a mechanism
|
||
|
||
`escalate` is an ordinary terminal node. It performs no transfer, opens no
|
||
ticket, and does not use `interrupt()`. It returns a structured result and the
|
||
run ends normally:
|
||
|
||
```json
|
||
{
|
||
"status": "escalate",
|
||
"message": "<what the bot says to the user before handing off>",
|
||
"reason": "user_requested | insufficient_retrieval | ungrounded | low_confidence | restricted_intent | budget_exhausted | frustration",
|
||
"handoff_context": {
|
||
"rewritten_query": "...",
|
||
"top_chunks": [{ "chunk_id": "...", "score": 0.0, "title": "..." }],
|
||
"conversation_summary": "...",
|
||
"user_profile": { }
|
||
}
|
||
}
|
||
```
|
||
|
||
The main backend consumes `status: "escalate"` and performs exactly what it
|
||
already does when a user clicks the "connect me to a human assistant" button —
|
||
the agent is simply another thing that can trigger that path. Once the backend
|
||
transfers the session, it stops issuing runs against that thread
|
||
([0007](0007-agent-persistence-threads-and-memory.md)).
|
||
|
||
`handoff_context` is included even though this service doesn't own the
|
||
handoff: it is what lets the human assistant start informed instead of reading
|
||
the whole transcript. The backend may forward it to the agent console or
|
||
ignore it.
|
||
|
||
**The transfer is one-way.** Per the current backend's behaviour, a
|
||
transferred session never returns to the agent. This graph therefore holds no
|
||
state about who owns the conversation — there is no `bot`/`human` mode flag
|
||
and no relay path — because after escalation the graph is simply never run
|
||
again on that thread. See *Alternatives Considered* for what would
|
||
change if handback is ever introduced.
|
||
|
||
### Grounding and citation contract
|
||
|
||
- `generate` receives **only** `retrieved_chunks` as evidence. Model prior
|
||
knowledge about insurance is not evidence and the system prompt says so.
|
||
- Every factual claim carries a `chunk_id` citation; the frontend can resolve
|
||
these to source documents via the ADR-0002 CRUD API.
|
||
- A claim that cannot be cited is not emitted — the model sets
|
||
`answered: false` instead, which routes to `escalate`.
|
||
- Numbers (premiums, limits, deductibles) are quoted verbatim from chunks or
|
||
computed by a deterministic tool. The model never does arithmetic on money.
|
||
|
||
### Language
|
||
|
||
The corpus is Persian-first (`bm25-fa-norm-stop`, `nomic-embed-text-v2-moe`,
|
||
ADR-0001/0004). The detected language of the user's turn is carried in state
|
||
and the answer is produced in that language, while retrieval always runs on
|
||
the contextualized query in the corpus's language.
|
||
|
||
## Consequences
|
||
|
||
### Positive
|
||
- Escalation is a property of the graph, not of a prompt — the agent cannot
|
||
talk itself out of handing off, which is the correct default for insurance.
|
||
- Because handoff is a terminal signal rather than an interrupt, this service
|
||
drops into the existing backend without changing how that backend transfers
|
||
sessions today: the agent becomes a second trigger for a path that already
|
||
exists.
|
||
- No `interrupt()` anywhere means no checkpoint-resume semantics to reason
|
||
about, and none of the "code before `interrupt()` re-runs on resume"
|
||
idempotency hazards.
|
||
- Clarifying questions are derived from real retrieved chunk metadata, so the
|
||
options offered always correspond to content that actually exists.
|
||
- `verify` catches the highest-consequence failure mode (fluent, ungrounded
|
||
coverage claims) before it reaches a user, at the cost of one small model
|
||
call.
|
||
- Bounded retry loops make worst-case latency and per-turn token cost
|
||
predictable, which a free-running tool loop does not.
|
||
|
||
### Negative
|
||
- Five to eight model calls per turn (triage, contextualize, grade, generate,
|
||
verify) is materially more latency and cost than single-shot RAG. Several
|
||
are small-model calls, but the floor is higher.
|
||
- More nodes means more prompts to maintain and evaluate; each of `triage`,
|
||
`grade`, and `verify` is its own quality surface that can regress.
|
||
- `verify` will produce false positives, sending answerable questions to
|
||
humans and raising handoff volume. The threshold needs tuning against real
|
||
traffic before it can be trusted.
|
||
- Nothing in this service prevents a further run on an already-transferred
|
||
thread; correctness depends on the main backend not issuing one. If that
|
||
assumption breaks, the bot will answer over a human assistant.
|
||
- The escalation policy list (trigger 6) is hard-coded product knowledge that
|
||
will drift from what the business actually wants automated.
|
||
|
||
## Alternatives Considered
|
||
|
||
- **LangChain `create_agent` with `retrieve` as a tool**: rejected as the
|
||
outer structure. It gives the model discretion over exactly the decisions
|
||
that must be deterministic here — when to stop retrying, whether to escalate
|
||
— and offers no natural place for a groundedness gate. A bounded agentic
|
||
loop *inside* an explicit graph gets the useful half of this without the
|
||
liability.
|
||
- **Deep Agents**: rejected — planning, filesystem, and subagent delegation
|
||
solve problems this workflow doesn't have.
|
||
- **`interrupt()` + `Command(resume=...)` for handoff**: rejected. It models a
|
||
human replying *into a paused graph run*, which is only worth its complexity
|
||
if the conversation later returns to the agent. It doesn't here, and the
|
||
human channel is another system's responsibility.
|
||
- **A `mode: "bot" | "human"` latch in graph state** (deferred, not rejected):
|
||
a persisted flag read at the top of every run, routing human-owned threads
|
||
past the LLM to a relay node that only appends to the transcript. This
|
||
becomes necessary the moment either (a) handback to the agent is introduced,
|
||
so the bot must know what the human said, or (b) the backend can no longer
|
||
guarantee it stops issuing runs after a transfer. Out of scope while
|
||
transfers are one-way and backend-enforced; revisit via a new ADR if either
|
||
assumption changes.
|
||
- **Single-pass RAG with no grading or verification**: rejected — it has no
|
||
mechanism to detect that it should escalate, which is a hard product
|
||
requirement, and no defence against ungrounded coverage claims.
|
||
- **Escalation as a tool the model can call**: rejected as the *only*
|
||
mechanism. Kept as one trigger among several (via `triage`), but a model
|
||
that is confidently wrong will not call an escalation tool — precisely the
|
||
case where escalation matters most.
|
||
- **Clarify before retrieving**: rejected — the requirement is that
|
||
clarification be grounded in retrieved chunks, which is impossible before
|
||
the retrieval runs. A cheap first retrieval pass is what makes the
|
||
clarifying question specific.
|