A/B testing chatbot tone: design lessons from a support AI pilot

AI · 3 min read

The team launched parallel flows that varied only in prompt engineering and microcopy: a terse formal voice focused on steps, an empathetic voice that mirrored customer emotion and apologized, and an expert voice that used confident, technical phrasing. Designers tracked single-session resolution, escalation rate to human agents, and post-interaction satisfaction.

Quantitative results favored a hybrid empathetic-expert tone: it reduced escalations by 21% and improved satisfaction scores by 14% compared with the formal voice. Qualitative feedback showed users trusted a bot more when it acknowledged their frustration and then offered a clear, stepwise solution. The formal voice was efficient for low-friction troubleshooting but felt robotic in higher-emotion contexts.

The product decision was to ship the hybrid tone as default while allowing enterprise clients to toggle presets. The experiment underscored that conversational design is not purely a language problem — it’s a product choice that affects operational costs and brand perception.