UX Case Study: Rolling Out LLM-Powered Smart Replies in a B2B Support App Without Sacrificing Control

AI · 7 min read

SupportFlow introduced an LLM-powered draft suggestion feature to help agents respond faster to inbound tickets. The core design challenge for product and design was balancing efficiency gains with the need for agent oversight, legal auditability, and domain accuracy. The team adopted a human-in-the-loop model where the LLM generated multiple candidate replies with provenance metadata, while agents remained responsible for edits and final send.

Design iterations focused on placement, affordances, and explainability. The UI offered three ranked suggestions, an inline source panel linking to knowledge base snippets used by the model, and an edit-before-send flow that preserved change history. The team implemented guardrails including intent confirmation, a high-confidence threshold for suggestions, and automated flags for regulatory language requiring legal review.

In a controlled rollout across 120 agents, median reply time fell by 36 percent and first-response SLA compliance improved from 88 to 94 percent. A small set of errors (0.8 percent of suggestions) required rollback due to domain inaccuracy; these cases led to tighter retraining and a new feedback loop from agents into the model dataset. The case demonstrates how ergonomics and governance can unlock LLM gains without eroding agent agency.