AI-Powered Microcopy: A/B Testing 120 Variants for a Ride-hailing App

AI · 5 min read

Transitly’s product team hypothesized that subtle changes in microcopy could sway last-mile conversion. They leveraged a constrained LLM to generate short variants for CTAs, error messages, and ETA tooltips, then ran a factorial A/B test covering tone (friendly vs. professional) and length (short vs. explanatory) across key touchpoints.

Rather than wholesale automation, the team curated model outputs and paired each variant with a hypothesis and metric. They prioritized control and safety: all copy passed a brand voice checklist and accessibility review. The test matrix included 120 variants across three funnel steps to identify interactions between location in flow and copy style.

Results showed the biggest lift when short, confident CTAs were paired with slightly more explanatory tooltips on price changes—overall booking completion climbed 9%. Notably, overly playful tone harmed trust in surge-price scenarios, while concise professionalism underperformed for first-time users in urban contexts where context mattered.

The experiment illustrates a balanced product-design approach to AI: use generative models to scale ideation, but pair outputs with rigorous testing and human curation. The team shipped a small microcopy library and deployment rules so future changes could be A/B tested quickly without design churn.