Measuring Real Resolution: A Deterministic Post-Chat Survey for Amazon Connect Chatbots
Creating a post-chat survey in Amazon Connect
The main problem I was facing: deflection is not resolution
If you run a chatbot in Amazon Connect, the metric you can get for free is containment (also called deflection): the share of conversations that ended without a human agent. It is a useful cost signal, but it answers the wrong question. "No human was needed" is not the same as "the customer got what they needed." A conversation can be contained because the bot solved the problem, or because the customer gave up. Containment alone cannot tell those apart.
To report on genuine resolution, you need to ask the customer directly and capture the answer reliably. This article describes a pattern for doing exactly that: a short, deterministic post-chat survey wired into the contact flow, with the results flowing automatically into daily reporting.
Why "deterministic" — and why not let the LLM ask
A natural first instinct is to let the generative AI agent (Amazon Q in Connect / a Lex bot orchestration) ask the survey question itself as part of wrapping up. We tried that first, and it did not hold up: across multiple prompt revisions, the model would sometimes skip the question and simply close the conversation, leaving no recorded answer. An intermittently-asked survey produces biased data — you cannot tell whether a missing answer means "declined" or "the model forgot to ask." That undermines the entire point of measuring.
The fix is to move the question out of the model's discretion and into the contact flow, where it fires deterministically every time a conversation reaches completion. The model decides when the conversation is done; the flow decides what happens next, and asking the survey is not optional.
Architecture overview
The design has three independent layers. The key property is that each layer keys off stable identifiers, not display names, so renaming or reorganizing flows never breaks reporting.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Customer chat/SMS
│
▼
[ Contact flow ]
• Bot conversation (Lex / Q-in-Connect)
• Compare on the "tool"/intent session attribute:
├─ Escalate branch → set escalation attributes → transfer to a queue (NO survey)
└─ Complete branch → SURVEY GRAFT:
set survey_offered = true
→ ask "Were you able to get the help you needed today?"
→ capture Yes / No via a dedicated yes/no Lex bot
→ set survey_response = yes | no | declined
→ goodbye → disconnect
│
▼
[ Contact records in S3 ] (survey_offered / survey_response ride along as contact attributes)
│
▼
[ Metrics service ] reads contact records, identifies bot contacts,
extracts survey attributes, aggregates per day (cached)
│
▼
[ Daily report / email ] renders a "Bot Satisfaction" section when data existsLayer 1 — the survey graft in the contact flow
The bot conversation ends by evaluating a session attribute that signals intent — typically something the model sets to indicate "escalate to a human" versus "conversation complete." A
Compare block branches on it:- Escalate branch: writes escalation context (reason, summary, sentiment, contact identity) to contact attributes and transfers to a queue. This branch intentionally gets no survey — you only survey conversations the bot resolved itself.
- Complete branch (the graft): this is where the survey lives.
UpdateContactAttributes→ setsurvey_offered = "true".MessageParticipant→ ask a single yes/no question.ConnectParticipantWithLexBot→ a small, dedicated yes/no Lex bot classifies the reply intoSurveyYes/SurveyNo.- Branch on the result and
UpdateContactAttributes→ setsurvey_responsetoyes,no, ordeclined. - Fall through to the existing goodbye message and disconnect.
Two design details that matter:
- Use a separate, purpose-built yes/no Lex bot for the survey rather than reusing the main conversational bot. It has one job (classify an affirmative/negative reply), which keeps it reliable, and it coexists with the main bot as a second
ConnectParticipantWithLexBotcall after the primary conversation. - Distinguish "declined" from "no answer." If the customer gives an explicit non-yes/no reply, record
declined. If they simply abandon at the survey question,survey_responseis never written — you derive "offered, no answer" downstream asoffered = truewithresponseabsent. Collapsing these two would distort your response rate.
Layer 2 — bot-contact identification (the part that survives refactoring)
Downstream reporting must decide which contact records represent bot conversations. The robust way is to key off the generative-AI assistant identifier carried on the contact record (the Q-in-Connect / Wisdom session association), plus the channel. Concretely, a contact counts as a bot contact when it is a chat/SMS contact AND its session association references your assistant's ID.
Do not identify bot contacts by the contact flow's name or even its flow ID. Names change, flows get reorganized, and flow IDs can multiply across environments. The assistant association is stable across all of that. This single decision is what lets you rename or restructure flows freely without touching any reporting code.
Layer 3 — survey extraction and aggregation
The metrics service reads contact records for a day and, for each bot contact, extracts two attributes by fixed name:
survey_offered→ boolean (true/false)survey_response→ normalized toyes,no,declined, or (when offered but absent)no_answer
It then aggregates per day:
- Offered = contacts with
survey_offered = true - Responded = yes + no (customers who gave a usable answer)
- Satisfaction rate = yes ÷ (yes + no) — computed only over responders
- Response rate = responded ÷ offered
Reporting satisfaction over responders and always showing the response rate beside it keeps the number honest: declines and no-answers lower the response rate without inflating or deflating satisfaction.
Reporting output
The daily report renders a "Bot Satisfaction (post-chat survey)" section with: surveys offered, answered (yes/no), declined, offered-no-answer, response rate, and satisfaction rate. The section is gated purely on data — if a day had zero surveys offered, it is omitted entirely, so the report degrades gracefully before the survey is live and on zero-traffic days. No feature flag or code change is needed to "turn on" the section; it appears automatically once real completions start writing the attributes.
Caching gotcha to plan for
Per-day metrics are typically cached (compute once, reuse). A common freshness rule is: the current day's cache has a short TTL, past days are treated as immutable. That rule has one sharp edge on the day you launch the survey: any day already cached before the survey went live holds zero survey counts, and because it is now a "past day," it will not recompute on its own.
Two clean ways to handle the launch day:
- Invalidate that one day's cache object so the next request recomputes it from the raw contact records (which now include survey data). Verify the exact cache key first; deleting it forces a full recompute of that day's metrics — which is harmless because it re-derives from source, just more completely.
- Or confirm the day hasn't been cached yet. If your cache is written a day in arrears (e.g. an overnight job computes yesterday), the launch day simply gets computed fresh after it closes and naturally includes the survey. Check which pattern you have before assuming you need to invalidate anything.
Every day after launch is unaffected — they compute fresh with the survey already live.
Validation checklist
Before trusting the numbers, validate each outcome with live test chats and confirm the contact attributes landed. The fastest authoritative check reads attributes directly from the Connect API for a known contact ID (no waiting on the S3 export):
- Yes →
survey_offered = true,survey_response = yes - No →
survey_offered = true,survey_response = no - Unclear reply →
survey_offered = true,survey_response = declined - Abandon at survey →
survey_offered = true,survey_responseabsent (reads as "no answer") - Escalation (control) → escalation attributes present, no survey attributes — this is the critical safety check that the survey never runs on the escalate path
Confirm the escalation control especially: an escalated contact must carry zero survey attributes.
Takeaways
- Put the survey in the flow, not the model. Determinism is what makes the resulting data trustworthy.
- Only survey self-resolved conversations. Escalations and abandons are correctly excluded, which also means early data is sparse until real bot-completed conversations accumulate.
- Identify bot contacts by the assistant association, never by flow name or ID. This decouples reporting from flow lifecycle changes entirely.
- Report satisfaction over responders and always show response rate. It is the honest framing, and it is a large step up from having no resolution signal at all.
- Plan for the cache on launch day. Know whether your per-day cache needs a one-time invalidation for the cutover date.
Honest caveats to communicate to stakeholders
- Satisfaction is self-reported by the subset of customers who choose to answer; frustrated customers may leave, so always pair it with response rate.
- It measures reported resolution, not independently verified resolution. It is the best direct signal available, not ground truth.
All in all, this took about a week to build, test, and document and around 20 minutes to graft into my existing Connect flow and test in production.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article