azk/chatbot_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): add conversational graph and persistence ADRs

Browse Source 
Why:
- Design the LangGraph-based insurance chatbot: retrieval-grounded
  answering, grounded clarification, and handoff to a human assistant
  owned by the existing main backend.

Changes:
- ADR-0006: graph nodes/edges, escalation triggers, and the
  backend-owned handoff signal contract.
- ADR-0007: Postgres checkpointer, thread model (thread_id as the
  shared identifier), history trimming strategy, and cross-thread
  user memory via the Store.

Impact:
- Design-only; no code changes yet.
This commit is contained in:
Ali Zarin Kolah
2026-08-03 12:56:49 +03:30
parent e0a7d3ec02
commit 50f6c33c0c
2 changed files with 502 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

245
docs/adr/0006-conversational-agent-graph.md Normal file
View File

@@ -0,0 +1,245 @@
# 0006. Conversational agent graph (LangGraph): grounding, clarification, and handoff signalling
## Status
Proposed
## Context
ADRs [0001](0001-ingestion-pipeline-and-collection-schema.md)–[0005](0005-reranking-model-and-sparse-analyzer-selection.md)
define how insurance documents get into Qdrant and how the best chunks come
back out ([0003](0003-agent-hybrid-retrieval.md)). Nothing yet defines the
*conversation* on top of that retrieval: what asks the question, what decides
the retrieved chunks are good enough, what happens when they aren't, and what
happens when the user should be talking to a person instead of a model.
The product requirements that shape this decision:
- **Agentic, not a fixed RAG chain.** A single "embed → retrieve → stuff into
a prompt" pass is not enough. Follow-up questions ("what about for my
wife?") are unanswerable without rewriting the query against conversation
history, and one retrieval round is often the wrong round.
- **Ambiguity is resolved by asking, and the question must be grounded.** If
the user asks "how much does it cost?", the bot must ask *which* policy —
and the options it offers must come from what retrieval actually returned
(e.g. three distinct products in the fused top-k), not from the model's
imagination. An ungrounded clarifying question is a hallucination with a
question mark on it.
- **Handoff is decided by the agent but executed elsewhere.** The existing
main backend already owns human handoff: when a user clicks "connect me to
a human assistant," that backend transfers the whole chat session to a human
and the session never returns to the agent. This service's job is to reach
the *same* outcome autonomously — to decide that a conversation should be
transferred and say so — not to implement the transfer.
- **This is insurance.** A confidently wrong answer about what a policy covers
is a liability, not an inconvenience. "I don't know, let me get someone" is
a *correct* output, and the graph must be able to produce it.
LangGraph is the right layer here (rather than a LangChain `create_agent`
loop or Deep Agents): the workflow mixes agentic steps (query rewriting,
bounded retrieval retries) with deterministic gates (groundedness
verification, escalation policy, loop budgets) over durable per-session state.
## Decision
### Overall shape: deterministic outer graph, bounded agentic core
An explicit `StateGraph` — not a prebuilt tool-calling agent. The model gets
agency where agency helps (rewriting the retrieval query, deciding whether
what came back is sufficient) and none where it doesn't (whether to escalate,
whether to answer at all, how many times to retry). Escalation policy and
loop budgets are graph edges, not model discretion.
### Nodes
| Node | Kind | Responsibility |
|---|---|---|
| `load_memory` | data | Reads the user's long-term profile from the Store ([0007](0007-agent-persistence-threads-and-memory.md)) into state. |
| `triage` | LLM (small) | Classifies the turn: `chitchat`, `knowledge` (answerable from the corpus), `account` (needs a tool + identity), `handoff_request` (user explicitly asked for a person), `out_of_scope`. |
| `contextualize` | LLM (small) | Rewrites the latest turn into a standalone retrieval query using recent history + profile. `"what about for my wife?"` → `"third-party liability coverage for a spouse"`. Emits the rewrite to state; the original user text is never mutated. |
| `retrieve` | action | Calls the ADR-0003 hybrid pipeline (dual dense + sparse → RRF → `jina-colbert-v2` rerank), with `tenant_id`/`domain` injected server-side, plus ADR-0003 context-window expansion. Returns chunks with scores and payload metadata. |
| `grade` | LLM (small) / heuristic | Per-chunk relevance judgement over the reranked top-k. Produces one of three verdicts: `sufficient`, `ambiguous`, `insufficient`. |
| `clarify` | LLM | Builds a grounded clarifying question **from chunk payload metadata** — the distinct `domain`/`file_id`/title values present in the retrieved set become the offered options. Ends the turn awaiting the user's next message. |
| `generate` | LLM (main) | Answers **only** from `retrieved_chunks`, with per-claim citations to `chunk_id`. Structured output: `answer`, `citations[]`, `answered: bool`, `confidence`. |
| `verify` | LLM (small) / NLI | Groundedness check: is every claim in the draft supported by a cited chunk? Cheap guard against the failure mode that matters most here. |
| `escalate` | terminal | Produces the handoff signal (below) and ends the run. Does not perform the transfer. |
| `write_memory` | data | Extracts durable, non-sensitive user facts and writes them to the Store. Runs after a successful answer. |
### Edges and control flow
```
START
└─ load_memory
│
triage
┌────────────┬──────────────────┼────────────────┬─────────────┐
chitchat out_of_scope handoff_request account knowledge
│ │ │ │ │
generate generate escalate (tools) contextualize
(no RAG) (refusal) │ │ │
│ │ ▼ └────────► retrieve ◄─┐
└─────────────┴──────────────► END │ │
grade │
┌───────────────┬───────────────────┴──┐ │
sufficient ambiguous insufficient │
│ │ │ │
generate clarify ──► END retries_left?─┘
│ │ yes
verify no
┌───────┴────────┐ │
grounded ungrounded │
│ │ │
write_memory regen_left? ──no──► escalate ◄──────┘
│ │ yes │
END └──► generate END
```
### Loop budgets (hard, enforced in state — not by the model)
| Budget | Limit | On exhaustion |
|---|---|---|
| Retrieval rounds per turn | 2 | → `escalate` |
| Clarifying questions per topic | 2 consecutive | → `escalate` |
| Regenerations after failed `verify` | 1 | → `escalate` |
| Total LLM calls per turn | ~8 | → `escalate` |
Every budget's exhaustion path leads to a human, never to a guessed answer.
### Escalation triggers
`escalate` is reached when **any** of these hold — the union is the policy:
1. The user explicitly asks for a person (`triage` → `handoff_request`).
2. Retrieval came back `insufficient` after the retry budget.
3. `verify` found the draft ungrounded after the regeneration budget.
4. `generate` returned `answered: false` or `confidence` below threshold.
5. Two consecutive clarifying questions failed to converge.
6. The intent is one we deliberately never automate: claim disputes,
complaints, cancellations, anything with a payout or legal consequence.
This is a policy list, not a model judgement.
7. Detected user frustration (repeat phrasing, explicit dissatisfaction).
### Handoff is a signal, not a mechanism
`escalate` is an ordinary terminal node. It performs no transfer, opens no
ticket, and does not use `interrupt()`. It returns a structured result and the
run ends normally:
```json
{
"status": "escalate",
"message": "<what the bot says to the user before handing off>",
"reason": "user_requested | insufficient_retrieval | ungrounded | low_confidence | restricted_intent | budget_exhausted | frustration",
"handoff_context": {
"rewritten_query": "...",
"top_chunks": [{ "chunk_id": "...", "score": 0.0, "title": "..." }],
"conversation_summary": "...",
"user_profile": { }
}
}
```
The main backend consumes `status: "escalate"` and performs exactly what it
already does when a user clicks the "connect me to a human assistant" button —
the agent is simply another thing that can trigger that path. Once the backend
transfers the session, it stops issuing runs against that thread
([0007](0007-agent-persistence-threads-and-memory.md)).
`handoff_context` is included even though this service doesn't own the
handoff: it is what lets the human assistant start informed instead of reading
the whole transcript. The backend may forward it to the agent console or
ignore it.
**The transfer is one-way.** Per the current backend's behaviour, a
transferred session never returns to the agent. This graph therefore holds no
state about who owns the conversation — there is no `bot`/`human` mode flag
and no relay path — because after escalation the graph is simply never run
again on that thread. See *Alternatives Considered* for what would
change if handback is ever introduced.
### Grounding and citation contract
- `generate` receives **only** `retrieved_chunks` as evidence. Model prior
knowledge about insurance is not evidence and the system prompt says so.
- Every factual claim carries a `chunk_id` citation; the frontend can resolve
these to source documents via the ADR-0002 CRUD API.
- A claim that cannot be cited is not emitted — the model sets
`answered: false` instead, which routes to `escalate`.
- Numbers (premiums, limits, deductibles) are quoted verbatim from chunks or
computed by a deterministic tool. The model never does arithmetic on money.
### Language
The corpus is Persian-first (`bm25-fa-norm-stop`, `nomic-embed-text-v2-moe`,
ADR-0001/0004). The detected language of the user's turn is carried in state
and the answer is produced in that language, while retrieval always runs on
the contextualized query in the corpus's language.
## Consequences
### Positive
- Escalation is a property of the graph, not of a prompt — the agent cannot
talk itself out of handing off, which is the correct default for insurance.
- Because handoff is a terminal signal rather than an interrupt, this service
drops into the existing backend without changing how that backend transfers
sessions today: the agent becomes a second trigger for a path that already
exists.
- No `interrupt()` anywhere means no checkpoint-resume semantics to reason
about, and none of the "code before `interrupt()` re-runs on resume"
idempotency hazards.
- Clarifying questions are derived from real retrieved chunk metadata, so the
options offered always correspond to content that actually exists.
- `verify` catches the highest-consequence failure mode (fluent, ungrounded
coverage claims) before it reaches a user, at the cost of one small model
call.
- Bounded retry loops make worst-case latency and per-turn token cost
predictable, which a free-running tool loop does not.
### Negative
- Five to eight model calls per turn (triage, contextualize, grade, generate,
verify) is materially more latency and cost than single-shot RAG. Several
are small-model calls, but the floor is higher.
- More nodes means more prompts to maintain and evaluate; each of `triage`,
`grade`, and `verify` is its own quality surface that can regress.
- `verify` will produce false positives, sending answerable questions to
humans and raising handoff volume. The threshold needs tuning against real
traffic before it can be trusted.
- Nothing in this service prevents a further run on an already-transferred
thread; correctness depends on the main backend not issuing one. If that
assumption breaks, the bot will answer over a human assistant.
- The escalation policy list (trigger 6) is hard-coded product knowledge that
will drift from what the business actually wants automated.
## Alternatives Considered
- **LangChain `create_agent` with `retrieve` as a tool**: rejected as the
outer structure. It gives the model discretion over exactly the decisions
that must be deterministic here — when to stop retrying, whether to escalate
— and offers no natural place for a groundedness gate. A bounded agentic
loop *inside* an explicit graph gets the useful half of this without the
liability.
- **Deep Agents**: rejected — planning, filesystem, and subagent delegation
solve problems this workflow doesn't have.
- **`interrupt()` + `Command(resume=...)` for handoff**: rejected. It models a
human replying *into a paused graph run*, which is only worth its complexity
if the conversation later returns to the agent. It doesn't here, and the
human channel is another system's responsibility.
- **A `mode: "bot" | "human"` latch in graph state** (deferred, not rejected):
a persisted flag read at the top of every run, routing human-owned threads
past the LLM to a relay node that only appends to the transcript. This
becomes necessary the moment either (a) handback to the agent is introduced,
so the bot must know what the human said, or (b) the backend can no longer
guarantee it stops issuing runs after a transfer. Out of scope while
transfers are one-way and backend-enforced; revisit via a new ADR if either
assumption changes.
- **Single-pass RAG with no grading or verification**: rejected — it has no
mechanism to detect that it should escalate, which is a hard product
requirement, and no defence against ungrounded coverage claims.
- **Escalation as a tool the model can call**: rejected as the *only*
mechanism. Kept as one trigger among several (via `triage`), but a model
that is confidently wrong will not call an escalation tool — precisely the
case where escalation matters most.
- **Clarify before retrieving**: rejected — the requirement is that
clarification be grounded in retrieved chunks, which is impossible before
the retrieval runs. A cheap first retrieval pass is what makes the
clarifying question specific.