docs(agent): add conversational graph and persistence ADRs
Why: - Design the LangGraph-based insurance chatbot: retrieval-grounded answering, grounded clarification, and handoff to a human assistant owned by the existing main backend. Changes: - ADR-0006: graph nodes/edges, escalation triggers, and the backend-owned handoff signal contract. - ADR-0007: Postgres checkpointer, thread model (thread_id as the shared identifier), history trimming strategy, and cross-thread user memory via the Store. Impact: - Design-only; no code changes yet.
This commit is contained in:
245
docs/adr/0006-conversational-agent-graph.md
Normal file
245
docs/adr/0006-conversational-agent-graph.md
Normal file
@@ -0,0 +1,245 @@
|
|||||||
|
# 0006. Conversational agent graph (LangGraph): grounding, clarification, and handoff signalling
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposed
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
ADRs [0001](0001-ingestion-pipeline-and-collection-schema.md)–[0005](0005-reranking-model-and-sparse-analyzer-selection.md)
|
||||||
|
define how insurance documents get into Qdrant and how the best chunks come
|
||||||
|
back out ([0003](0003-agent-hybrid-retrieval.md)). Nothing yet defines the
|
||||||
|
*conversation* on top of that retrieval: what asks the question, what decides
|
||||||
|
the retrieved chunks are good enough, what happens when they aren't, and what
|
||||||
|
happens when the user should be talking to a person instead of a model.
|
||||||
|
|
||||||
|
The product requirements that shape this decision:
|
||||||
|
|
||||||
|
- **Agentic, not a fixed RAG chain.** A single "embed → retrieve → stuff into
|
||||||
|
a prompt" pass is not enough. Follow-up questions ("what about for my
|
||||||
|
wife?") are unanswerable without rewriting the query against conversation
|
||||||
|
history, and one retrieval round is often the wrong round.
|
||||||
|
- **Ambiguity is resolved by asking, and the question must be grounded.** If
|
||||||
|
the user asks "how much does it cost?", the bot must ask *which* policy —
|
||||||
|
and the options it offers must come from what retrieval actually returned
|
||||||
|
(e.g. three distinct products in the fused top-k), not from the model's
|
||||||
|
imagination. An ungrounded clarifying question is a hallucination with a
|
||||||
|
question mark on it.
|
||||||
|
- **Handoff is decided by the agent but executed elsewhere.** The existing
|
||||||
|
main backend already owns human handoff: when a user clicks "connect me to
|
||||||
|
a human assistant," that backend transfers the whole chat session to a human
|
||||||
|
and the session never returns to the agent. This service's job is to reach
|
||||||
|
the *same* outcome autonomously — to decide that a conversation should be
|
||||||
|
transferred and say so — not to implement the transfer.
|
||||||
|
- **This is insurance.** A confidently wrong answer about what a policy covers
|
||||||
|
is a liability, not an inconvenience. "I don't know, let me get someone" is
|
||||||
|
a *correct* output, and the graph must be able to produce it.
|
||||||
|
|
||||||
|
LangGraph is the right layer here (rather than a LangChain `create_agent`
|
||||||
|
loop or Deep Agents): the workflow mixes agentic steps (query rewriting,
|
||||||
|
bounded retrieval retries) with deterministic gates (groundedness
|
||||||
|
verification, escalation policy, loop budgets) over durable per-session state.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### Overall shape: deterministic outer graph, bounded agentic core
|
||||||
|
|
||||||
|
An explicit `StateGraph` — not a prebuilt tool-calling agent. The model gets
|
||||||
|
agency where agency helps (rewriting the retrieval query, deciding whether
|
||||||
|
what came back is sufficient) and none where it doesn't (whether to escalate,
|
||||||
|
whether to answer at all, how many times to retry). Escalation policy and
|
||||||
|
loop budgets are graph edges, not model discretion.
|
||||||
|
|
||||||
|
### Nodes
|
||||||
|
|
||||||
|
| Node | Kind | Responsibility |
|
||||||
|
|---|---|---|
|
||||||
|
| `load_memory` | data | Reads the user's long-term profile from the Store ([0007](0007-agent-persistence-threads-and-memory.md)) into state. |
|
||||||
|
| `triage` | LLM (small) | Classifies the turn: `chitchat`, `knowledge` (answerable from the corpus), `account` (needs a tool + identity), `handoff_request` (user explicitly asked for a person), `out_of_scope`. |
|
||||||
|
| `contextualize` | LLM (small) | Rewrites the latest turn into a standalone retrieval query using recent history + profile. `"what about for my wife?"` → `"third-party liability coverage for a spouse"`. Emits the rewrite to state; the original user text is never mutated. |
|
||||||
|
| `retrieve` | action | Calls the ADR-0003 hybrid pipeline (dual dense + sparse → RRF → `jina-colbert-v2` rerank), with `tenant_id`/`domain` injected server-side, plus ADR-0003 context-window expansion. Returns chunks with scores and payload metadata. |
|
||||||
|
| `grade` | LLM (small) / heuristic | Per-chunk relevance judgement over the reranked top-k. Produces one of three verdicts: `sufficient`, `ambiguous`, `insufficient`. |
|
||||||
|
| `clarify` | LLM | Builds a grounded clarifying question **from chunk payload metadata** — the distinct `domain`/`file_id`/title values present in the retrieved set become the offered options. Ends the turn awaiting the user's next message. |
|
||||||
|
| `generate` | LLM (main) | Answers **only** from `retrieved_chunks`, with per-claim citations to `chunk_id`. Structured output: `answer`, `citations[]`, `answered: bool`, `confidence`. |
|
||||||
|
| `verify` | LLM (small) / NLI | Groundedness check: is every claim in the draft supported by a cited chunk? Cheap guard against the failure mode that matters most here. |
|
||||||
|
| `escalate` | terminal | Produces the handoff signal (below) and ends the run. Does not perform the transfer. |
|
||||||
|
| `write_memory` | data | Extracts durable, non-sensitive user facts and writes them to the Store. Runs after a successful answer. |
|
||||||
|
|
||||||
|
### Edges and control flow
|
||||||
|
|
||||||
|
```
|
||||||
|
START
|
||||||
|
└─ load_memory
|
||||||
|
│
|
||||||
|
triage
|
||||||
|
┌────────────┬──────────────────┼────────────────┬─────────────┐
|
||||||
|
chitchat out_of_scope handoff_request account knowledge
|
||||||
|
│ │ │ │ │
|
||||||
|
generate generate escalate (tools) contextualize
|
||||||
|
(no RAG) (refusal) │ │ │
|
||||||
|
│ │ ▼ └────────► retrieve ◄─┐
|
||||||
|
└─────────────┴──────────────► END │ │
|
||||||
|
grade │
|
||||||
|
┌───────────────┬───────────────────┴──┐ │
|
||||||
|
sufficient ambiguous insufficient │
|
||||||
|
│ │ │ │
|
||||||
|
generate clarify ──► END retries_left?─┘
|
||||||
|
│ │ yes
|
||||||
|
verify no
|
||||||
|
┌───────┴────────┐ │
|
||||||
|
grounded ungrounded │
|
||||||
|
│ │ │
|
||||||
|
write_memory regen_left? ──no──► escalate ◄──────┘
|
||||||
|
│ │ yes │
|
||||||
|
END └──► generate END
|
||||||
|
```
|
||||||
|
|
||||||
|
### Loop budgets (hard, enforced in state — not by the model)
|
||||||
|
|
||||||
|
| Budget | Limit | On exhaustion |
|
||||||
|
|---|---|---|
|
||||||
|
| Retrieval rounds per turn | 2 | → `escalate` |
|
||||||
|
| Clarifying questions per topic | 2 consecutive | → `escalate` |
|
||||||
|
| Regenerations after failed `verify` | 1 | → `escalate` |
|
||||||
|
| Total LLM calls per turn | ~8 | → `escalate` |
|
||||||
|
|
||||||
|
Every budget's exhaustion path leads to a human, never to a guessed answer.
|
||||||
|
|
||||||
|
### Escalation triggers
|
||||||
|
|
||||||
|
`escalate` is reached when **any** of these hold — the union is the policy:
|
||||||
|
|
||||||
|
1. The user explicitly asks for a person (`triage` → `handoff_request`).
|
||||||
|
2. Retrieval came back `insufficient` after the retry budget.
|
||||||
|
3. `verify` found the draft ungrounded after the regeneration budget.
|
||||||
|
4. `generate` returned `answered: false` or `confidence` below threshold.
|
||||||
|
5. Two consecutive clarifying questions failed to converge.
|
||||||
|
6. The intent is one we deliberately never automate: claim disputes,
|
||||||
|
complaints, cancellations, anything with a payout or legal consequence.
|
||||||
|
This is a policy list, not a model judgement.
|
||||||
|
7. Detected user frustration (repeat phrasing, explicit dissatisfaction).
|
||||||
|
|
||||||
|
### Handoff is a signal, not a mechanism
|
||||||
|
|
||||||
|
`escalate` is an ordinary terminal node. It performs no transfer, opens no
|
||||||
|
ticket, and does not use `interrupt()`. It returns a structured result and the
|
||||||
|
run ends normally:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "escalate",
|
||||||
|
"message": "<what the bot says to the user before handing off>",
|
||||||
|
"reason": "user_requested | insufficient_retrieval | ungrounded | low_confidence | restricted_intent | budget_exhausted | frustration",
|
||||||
|
"handoff_context": {
|
||||||
|
"rewritten_query": "...",
|
||||||
|
"top_chunks": [{ "chunk_id": "...", "score": 0.0, "title": "..." }],
|
||||||
|
"conversation_summary": "...",
|
||||||
|
"user_profile": { }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The main backend consumes `status: "escalate"` and performs exactly what it
|
||||||
|
already does when a user clicks the "connect me to a human assistant" button —
|
||||||
|
the agent is simply another thing that can trigger that path. Once the backend
|
||||||
|
transfers the session, it stops issuing runs against that thread
|
||||||
|
([0007](0007-agent-persistence-threads-and-memory.md)).
|
||||||
|
|
||||||
|
`handoff_context` is included even though this service doesn't own the
|
||||||
|
handoff: it is what lets the human assistant start informed instead of reading
|
||||||
|
the whole transcript. The backend may forward it to the agent console or
|
||||||
|
ignore it.
|
||||||
|
|
||||||
|
**The transfer is one-way.** Per the current backend's behaviour, a
|
||||||
|
transferred session never returns to the agent. This graph therefore holds no
|
||||||
|
state about who owns the conversation — there is no `bot`/`human` mode flag
|
||||||
|
and no relay path — because after escalation the graph is simply never run
|
||||||
|
again on that thread. See *Alternatives Considered* for what would
|
||||||
|
change if handback is ever introduced.
|
||||||
|
|
||||||
|
### Grounding and citation contract
|
||||||
|
|
||||||
|
- `generate` receives **only** `retrieved_chunks` as evidence. Model prior
|
||||||
|
knowledge about insurance is not evidence and the system prompt says so.
|
||||||
|
- Every factual claim carries a `chunk_id` citation; the frontend can resolve
|
||||||
|
these to source documents via the ADR-0002 CRUD API.
|
||||||
|
- A claim that cannot be cited is not emitted — the model sets
|
||||||
|
`answered: false` instead, which routes to `escalate`.
|
||||||
|
- Numbers (premiums, limits, deductibles) are quoted verbatim from chunks or
|
||||||
|
computed by a deterministic tool. The model never does arithmetic on money.
|
||||||
|
|
||||||
|
### Language
|
||||||
|
|
||||||
|
The corpus is Persian-first (`bm25-fa-norm-stop`, `nomic-embed-text-v2-moe`,
|
||||||
|
ADR-0001/0004). The detected language of the user's turn is carried in state
|
||||||
|
and the answer is produced in that language, while retrieval always runs on
|
||||||
|
the contextualized query in the corpus's language.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Positive
|
||||||
|
- Escalation is a property of the graph, not of a prompt — the agent cannot
|
||||||
|
talk itself out of handing off, which is the correct default for insurance.
|
||||||
|
- Because handoff is a terminal signal rather than an interrupt, this service
|
||||||
|
drops into the existing backend without changing how that backend transfers
|
||||||
|
sessions today: the agent becomes a second trigger for a path that already
|
||||||
|
exists.
|
||||||
|
- No `interrupt()` anywhere means no checkpoint-resume semantics to reason
|
||||||
|
about, and none of the "code before `interrupt()` re-runs on resume"
|
||||||
|
idempotency hazards.
|
||||||
|
- Clarifying questions are derived from real retrieved chunk metadata, so the
|
||||||
|
options offered always correspond to content that actually exists.
|
||||||
|
- `verify` catches the highest-consequence failure mode (fluent, ungrounded
|
||||||
|
coverage claims) before it reaches a user, at the cost of one small model
|
||||||
|
call.
|
||||||
|
- Bounded retry loops make worst-case latency and per-turn token cost
|
||||||
|
predictable, which a free-running tool loop does not.
|
||||||
|
|
||||||
|
### Negative
|
||||||
|
- Five to eight model calls per turn (triage, contextualize, grade, generate,
|
||||||
|
verify) is materially more latency and cost than single-shot RAG. Several
|
||||||
|
are small-model calls, but the floor is higher.
|
||||||
|
- More nodes means more prompts to maintain and evaluate; each of `triage`,
|
||||||
|
`grade`, and `verify` is its own quality surface that can regress.
|
||||||
|
- `verify` will produce false positives, sending answerable questions to
|
||||||
|
humans and raising handoff volume. The threshold needs tuning against real
|
||||||
|
traffic before it can be trusted.
|
||||||
|
- Nothing in this service prevents a further run on an already-transferred
|
||||||
|
thread; correctness depends on the main backend not issuing one. If that
|
||||||
|
assumption breaks, the bot will answer over a human assistant.
|
||||||
|
- The escalation policy list (trigger 6) is hard-coded product knowledge that
|
||||||
|
will drift from what the business actually wants automated.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
- **LangChain `create_agent` with `retrieve` as a tool**: rejected as the
|
||||||
|
outer structure. It gives the model discretion over exactly the decisions
|
||||||
|
that must be deterministic here — when to stop retrying, whether to escalate
|
||||||
|
— and offers no natural place for a groundedness gate. A bounded agentic
|
||||||
|
loop *inside* an explicit graph gets the useful half of this without the
|
||||||
|
liability.
|
||||||
|
- **Deep Agents**: rejected — planning, filesystem, and subagent delegation
|
||||||
|
solve problems this workflow doesn't have.
|
||||||
|
- **`interrupt()` + `Command(resume=...)` for handoff**: rejected. It models a
|
||||||
|
human replying *into a paused graph run*, which is only worth its complexity
|
||||||
|
if the conversation later returns to the agent. It doesn't here, and the
|
||||||
|
human channel is another system's responsibility.
|
||||||
|
- **A `mode: "bot" | "human"` latch in graph state** (deferred, not rejected):
|
||||||
|
a persisted flag read at the top of every run, routing human-owned threads
|
||||||
|
past the LLM to a relay node that only appends to the transcript. This
|
||||||
|
becomes necessary the moment either (a) handback to the agent is introduced,
|
||||||
|
so the bot must know what the human said, or (b) the backend can no longer
|
||||||
|
guarantee it stops issuing runs after a transfer. Out of scope while
|
||||||
|
transfers are one-way and backend-enforced; revisit via a new ADR if either
|
||||||
|
assumption changes.
|
||||||
|
- **Single-pass RAG with no grading or verification**: rejected — it has no
|
||||||
|
mechanism to detect that it should escalate, which is a hard product
|
||||||
|
requirement, and no defence against ungrounded coverage claims.
|
||||||
|
- **Escalation as a tool the model can call**: rejected as the *only*
|
||||||
|
mechanism. Kept as one trigger among several (via `triage`), but a model
|
||||||
|
that is confidently wrong will not call an escalation tool — precisely the
|
||||||
|
case where escalation matters most.
|
||||||
|
- **Clarify before retrieving**: rejected — the requirement is that
|
||||||
|
clarification be grounded in retrieved chunks, which is impossible before
|
||||||
|
the retrieval runs. A cheap first retrieval pass is what makes the
|
||||||
|
clarifying question specific.
|
||||||
257
docs/adr/0007-agent-persistence-threads-and-memory.md
Normal file
257
docs/adr/0007-agent-persistence-threads-and-memory.md
Normal file
@@ -0,0 +1,257 @@
|
|||||||
|
# 0007. Agent persistence, thread model, history trimming, and cross-thread memory
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposed
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
[ADR-0006](0006-conversational-agent-graph.md) defines the conversational
|
||||||
|
graph but deliberately leaves its persistence to this ADR. Four separate
|
||||||
|
concerns need deciding, and they are easy to conflate:
|
||||||
|
|
||||||
|
1. **Durable conversation state.** Each turn must see the previous turns, and
|
||||||
|
that history must survive a process restart and be visible to every
|
||||||
|
replica. (Note that ADR-0006 uses no `interrupt()`, so persistence is
|
||||||
|
needed for *memory*, not for resuming paused runs.)
|
||||||
|
2. **Conversation identity.** The main backend calls this service with the
|
||||||
|
identifier of one chat session; something must map that to a LangGraph
|
||||||
|
thread, and it must be clear which system owns the session records and
|
||||||
|
which vocabulary this service's API speaks.
|
||||||
|
3. **History size.** The frontend displays at most 20 messages. It is not
|
||||||
|
obvious whether that number should also govern what the model sees, what
|
||||||
|
Postgres stores, or neither. Getting this wrong either blows up context
|
||||||
|
cost or silently amputates conversations.
|
||||||
|
4. **`user_id`.** The backend supplies a `user_id` that is stable across all
|
||||||
|
of a user's threads. It currently has no defined role.
|
||||||
|
|
||||||
|
The retrieval path already carries its own tenancy boundary
|
||||||
|
([0002](0002-chunk-crud-and-search-api.md)): `tenant_id` is injected
|
||||||
|
server-side from auth and never accepted from an end client. Anything decided
|
||||||
|
here must preserve that.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### Checkpointer: `AsyncPostgresSaver`, one pool, compiled once
|
||||||
|
|
||||||
|
Postgres via `langgraph-checkpoint-postgres` (`AsyncPostgresSaver`), because
|
||||||
|
the API is async FastAPI. Concretely:
|
||||||
|
|
||||||
|
- A single `psycopg_pool.AsyncConnectionPool` (`autocommit=True`,
|
||||||
|
`row_factory=dict_row`) opened in the FastAPI lifespan, shared by the
|
||||||
|
checkpointer and the Store.
|
||||||
|
- The graph is compiled **once** at startup and reused; it is not rebuilt
|
||||||
|
per request.
|
||||||
|
- `.setup()` runs as a deployment/migration step, **not** at application
|
||||||
|
startup — it issues DDL, and every replica racing to create tables on boot
|
||||||
|
is a deployment hazard.
|
||||||
|
|
||||||
|
`InMemorySaver` remains acceptable in tests only.
|
||||||
|
|
||||||
|
### Thread model: the thread is the only conversation record this service keeps
|
||||||
|
|
||||||
|
The identifier the main backend uses for a chat session **is** the LangGraph
|
||||||
|
`thread_id`. No indirection, no mapping table.
|
||||||
|
|
||||||
|
This service's API therefore speaks `thread_id` — in paths, request bodies,
|
||||||
|
and responses alike — rather than introducing a second noun for the same
|
||||||
|
entity. The backend passes its session identifier as the `thread_id` and that
|
||||||
|
is the whole of the mapping. Internal vocabulary matching the runtime keeps
|
||||||
|
the resource model honest: a thread here really is a LangGraph thread, with
|
||||||
|
LangGraph's semantics (a durable checkpoint sequence), not a product concept
|
||||||
|
that merely resembles one.
|
||||||
|
|
||||||
|
The main backend already owns users, sessions, and the human-assistant
|
||||||
|
transfer (ADR-0006). Duplicating that as a thread-metadata table here would be
|
||||||
|
a second set of records to keep in sync for no benefit, so this service stores
|
||||||
|
**no conversation metadata of its own** — no owner column, no title, no
|
||||||
|
status. Listing a user's conversations, showing titles, and knowing whether a
|
||||||
|
conversation was transferred to a human are all the backend's concerns. The
|
||||||
|
checkpoint sequence is the only record this service keeps.
|
||||||
|
|
||||||
|
The consequence is a **trust boundary**: `thread_id`, `user_id`, and
|
||||||
|
`tenant_id` arrive as trusted inputs on every call, and this service performs
|
||||||
|
no ownership check on them. That is only safe if the service is reachable
|
||||||
|
solely by the main backend — private network or service-to-service auth, never
|
||||||
|
exposed to browsers. Without that, a `thread_id` becomes a bearer token for
|
||||||
|
someone else's conversation. This is the single most important operational
|
||||||
|
constraint in this ADR.
|
||||||
|
|
||||||
|
### History: persist everything, trim only at the model call
|
||||||
|
|
||||||
|
The three "history sizes" are decoupled, and each gets its own answer:
|
||||||
|
|
||||||
|
| Layer | Size | Rationale |
|
||||||
|
|---|---|---|
|
||||||
|
| Postgres checkpoints | **Unbounded** (subject to retention) | Audit, handoff context, and eval datasets all need the full transcript. Insurance conversations are records. |
|
||||||
|
| Model context | **Trimmed per call** | Cost and latency, and Persian tokenizes expensively. |
|
||||||
|
| Frontend | **20 messages** | A display/pagination concern only. It must not drive either of the above. |
|
||||||
|
|
||||||
|
Trimming happens in a **pre-model hook** — `trim_messages` on the way into
|
||||||
|
`generate`, keeping the system prompt plus the most recent turns under a token
|
||||||
|
budget, and always starting the trimmed window on a `HumanMessage` with no
|
||||||
|
orphaned tool call/result pairs. It is applied to the messages *passed to the
|
||||||
|
model*; it does not mutate persisted state. Nothing is deleted.
|
||||||
|
|
||||||
|
Two additions on top of plain trimming:
|
||||||
|
|
||||||
|
- **Rolling summary.** Once a thread exceeds a threshold (~20 turns), a
|
||||||
|
summarization step folds the dropped older turns into a `summary` field in
|
||||||
|
state, which is prepended to the trimmed window. This is what stops a long
|
||||||
|
conversation from losing its own beginning — pure trimming does. It also
|
||||||
|
supplies `handoff_context.conversation_summary` in ADR-0006's escalation
|
||||||
|
payload for free.
|
||||||
|
- **Retrieved chunks are not messages.** Chunk text from ADR-0003 lands in a
|
||||||
|
dedicated `retrieved_chunks` state field scoped to the current turn, not in
|
||||||
|
the message list. Persisting full reranked chunk text as tool messages on
|
||||||
|
every turn is what actually blows up a RAG thread's state size; only the
|
||||||
|
`chunk_id` citations persist in the transcript.
|
||||||
|
|
||||||
|
The frontend's 20 messages are served by a paginated read endpoint over
|
||||||
|
checkpoint state, not by capping what is stored.
|
||||||
|
|
||||||
|
### `user_id`: long-term cross-thread memory, handoff context, and operations
|
||||||
|
|
||||||
|
`user_id` is passed per request by the main backend and does three jobs:
|
||||||
|
|
||||||
|
1. **Cross-thread memory (the main one).** An `AsyncPostgresStore`, same
|
||||||
|
pool, namespaced `(tenant_id, "user", user_id, "profile")` and
|
||||||
|
`(… , "facts")`. ADR-0006's `load_memory` reads it at the top of every
|
||||||
|
turn; `write_memory` writes after a successful answer. What belongs there:
|
||||||
|
language and formality preference, which products the user holds or has
|
||||||
|
asked about, household context they volunteered, recurring topics. What
|
||||||
|
does **not**: national ID, payment details, health information, or anything
|
||||||
|
derived from a single turn that shouldn't outlive it. Memory writes are
|
||||||
|
extraction-based and bounded, not "append every message."
|
||||||
|
2. **Handoff context** — the escalation payload in ADR-0006 carries the
|
||||||
|
profile so the human assistant starts informed rather than from zero.
|
||||||
|
3. **Operations** — per-user rate limiting and cohort segmentation in
|
||||||
|
analytics and evals.
|
||||||
|
|
||||||
|
`tenant_id` leads every Store namespace, for the same reason ADR-0002 gives:
|
||||||
|
a shared store partitioned only by `user_id` leaks across tenants the moment
|
||||||
|
user IDs are tenant-scoped rather than globally unique.
|
||||||
|
|
||||||
|
Cross-thread memory is the *only* reason this service needs `user_id` at all;
|
||||||
|
if long-term memory is dropped, `thread_id` and `tenant_id` would suffice.
|
||||||
|
|
||||||
|
### FastAPI surface
|
||||||
|
|
||||||
|
Resources are named after the LangGraph concepts they actually are, matching
|
||||||
|
LangGraph Platform's own `/threads/{thread_id}/runs` shape. One user message is
|
||||||
|
one graph execution, so it is a **run**, not a bespoke noun.
|
||||||
|
|
||||||
|
| Endpoint | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `POST /v1/threads/{thread_id}/runs` | Execute one graph run for the user's message. Body carries the message; `user_id`, `tenant_id`, and locale come from the service-to-service context. SSE stream (`stream_mode="messages"` for tokens, `"custom"` for retrieval progress). Terminal status: `answered` \| `clarifying` \| `escalate`. |
|
||||||
|
| `GET /v1/threads/{thread_id}/messages?limit=20&before=…` | Paginated transcript from checkpoint state, for the frontend's 20-message window. |
|
||||||
|
| `DELETE /v1/threads/{thread_id}` | Purge a thread's checkpoints (user-initiated "clear chat", or erasure). |
|
||||||
|
| `POST /v1/threads/{thread_id}/runs/{run_id}/feedback` | Thumbs up/down + optional text, for the eval dataset. |
|
||||||
|
| `GET /v1/users/{user_id}/memory` / `DELETE …` | Inspect and reset what the Store remembers about a user. Not thread-scoped — memory deliberately outlives any one thread. |
|
||||||
|
|
||||||
|
The thread is not created by an explicit endpoint: the first `POST …/runs`
|
||||||
|
against an unseen `thread_id` creates the checkpoint sequence implicitly, which
|
||||||
|
is what the checkpointer does anyway. Adding a `POST /v1/threads` would be a
|
||||||
|
row this service has already decided not to store.
|
||||||
|
|
||||||
|
There are **no handoff endpoints**. Escalation is a terminal status on the run
|
||||||
|
response (ADR-0006); the main backend acts on it and then stops issuing runs
|
||||||
|
against that `thread_id`.
|
||||||
|
|
||||||
|
**One in-flight run per thread.** Concurrent runs on the same `thread_id` race
|
||||||
|
on checkpoint writes. The run endpoint takes a Postgres advisory lock keyed on
|
||||||
|
`thread_id` and returns `409 Conflict` rather than queueing — a user
|
||||||
|
double-sending is a UI problem, not something to serialize silently.
|
||||||
|
|
||||||
|
`user_id`, `tenant_id`, and locale are passed per invocation in
|
||||||
|
`config["configurable"]`, not stored in graph state, so a change in the
|
||||||
|
caller's context is never overridden by a stale checkpoint.
|
||||||
|
|
||||||
|
### Retention
|
||||||
|
|
||||||
|
Checkpoints are not kept forever. A scheduled job deletes threads whose most
|
||||||
|
recent checkpoint is older than the retention window (to be set with the
|
||||||
|
business — insurance record-keeping obligations likely dominate this choice),
|
||||||
|
and the erasure endpoints above provide the per-user path that removes both
|
||||||
|
the thread checkpoints and the Store namespaces.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Positive
|
||||||
|
- One Postgres holds checkpoints and long-term memory — one backup, one
|
||||||
|
connection pool, one transactional story — and no conversation records are
|
||||||
|
duplicated between this service and the main backend.
|
||||||
|
- The API names the runtime's own concepts (`threads`, `runs`) rather than
|
||||||
|
inventing parallel ones, so there is no translation layer between the URL,
|
||||||
|
the checkpointer, and the LangGraph SDK — and a future move onto LangGraph
|
||||||
|
Platform would not be a rename.
|
||||||
|
- Decoupling the three history sizes means the frontend's 20-message window
|
||||||
|
can change without touching model cost or the durability guarantee.
|
||||||
|
- Keeping chunk text out of the message list is what keeps a long RAG thread's
|
||||||
|
checkpoint size flat, and it is far cheaper to get right now than to
|
||||||
|
retrofit.
|
||||||
|
- Checkpoint history gives support a way to replay exactly what the agent saw
|
||||||
|
when a user disputes an answer.
|
||||||
|
- With no `interrupt()` in the design, the checkpointer serves one purpose
|
||||||
|
only (conversation memory), which keeps the persistence story small.
|
||||||
|
|
||||||
|
### Negative
|
||||||
|
- **Authorization is entirely delegated.** This service will happily read or
|
||||||
|
append to any `thread_id` it is given. If it is ever exposed beyond the
|
||||||
|
main backend, that is a direct data-leak path.
|
||||||
|
- The API and the main backend use different words for the same identifier
|
||||||
|
(`thread_id` here, session elsewhere). The mapping is trivial and stated
|
||||||
|
once, but it is one more thing an integrator has to know.
|
||||||
|
- Unbounded checkpoint growth is a real storage cost, and it is deferred to a
|
||||||
|
retention policy that is not yet decided.
|
||||||
|
- The rolling summary is lossy by construction — a fact stated 40 turns ago
|
||||||
|
and dropped from the summary is gone from the model's view even though it
|
||||||
|
is still in Postgres.
|
||||||
|
- Advisory-lock-and-409 pushes retry handling onto the caller.
|
||||||
|
- Long-term memory is a new correctness surface: a wrong fact written to the
|
||||||
|
Store silently contaminates every future session until something removes it.
|
||||||
|
Hence the inspect/reset endpoints, which the backend should surface to users.
|
||||||
|
- Nothing here records that a conversation was transferred to a human, so this
|
||||||
|
service cannot distinguish an abandoned thread from a transferred one in
|
||||||
|
its own analytics; that join has to happen on the backend's data.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
- **`InMemorySaver` / `SqliteSaver`**: rejected — history must survive
|
||||||
|
restarts and be shared across replicas.
|
||||||
|
- **An application-owned thread-metadata table in this service** (user, title,
|
||||||
|
status, timestamps): rejected now that the main backend owns sessions and
|
||||||
|
the human transfer. It would be a second source of truth to keep in sync.
|
||||||
|
It becomes worth revisiting only if this service ever needs to answer
|
||||||
|
ownership or listing queries itself.
|
||||||
|
- **Naming the API resource `session` rather than `thread`** (e.g.
|
||||||
|
`/v1/sessions/{session_id}/turns`, mirroring the main backend's vocabulary):
|
||||||
|
rejected. This is an internal service-to-service API, not a public one, so
|
||||||
|
there is no audience it needs to shield from the runtime's concepts — and
|
||||||
|
the entity genuinely *is* a LangGraph thread with a checkpoint sequence, not
|
||||||
|
a session record, since this service stores no session record at all.
|
||||||
|
"Thread" is also ordinary conversation vocabulary (email, chat), so it
|
||||||
|
carries little framework lock-in as a name. The main backend keeps saying
|
||||||
|
"session"; the translation happens at its call site, in one place.
|
||||||
|
- **A separate `messages` table as the source of truth, with LangGraph state
|
||||||
|
rebuilt per request**: rejected — two transcripts to keep in sync, and it
|
||||||
|
discards the checkpoint history that makes replay and audit possible.
|
||||||
|
- **Hard-capping stored history at 20 messages to match the frontend**:
|
||||||
|
rejected — it conflates a display window with a retention policy, destroys
|
||||||
|
audit and eval data, and would hand a human assistant a truncated transcript
|
||||||
|
at exactly the moment they need the whole thing.
|
||||||
|
- **`RemoveMessage` / `REMOVE_ALL_MESSAGES` to prune persisted state**:
|
||||||
|
rejected as an automatic policy — it is destructive trimming where
|
||||||
|
non-destructive trimming costs nothing extra. It remains the right tool
|
||||||
|
behind the explicit `DELETE /v1/threads/{thread_id}` action.
|
||||||
|
- **Summarization instead of trimming**: rejected as a replacement, adopted as
|
||||||
|
a supplement. Summarizing every turn is an extra model call for short
|
||||||
|
conversations that don't need one; trimming is free and handles the common
|
||||||
|
case.
|
||||||
|
- **Redis for checkpoints**: rejected — the Store and the checkpoints want the
|
||||||
|
same transactional store, and durability matters more here than checkpoint
|
||||||
|
write latency.
|
||||||
|
- **A single long-lived thread per user instead of one per conversation**:
|
||||||
|
rejected — it would make one escalation apply to all of a user's future
|
||||||
|
conversations, and it removes any natural retention or "clear chat"
|
||||||
|
boundary. Cross-thread continuity is what the Store is for.
|
||||||
Reference in New Issue
Block a user