A/B Testing with Generative AI: How QuillCut Used Synthetic Users to Rapidly Iterate Copy

AI · 7 min read

QuillCut faced a classic growth problem: they needed to iterate hundreds of microcopy variants across onboarding and billing flows but lacked scaleable access to real user feedback. The team experimented with using generative AI to synthesize likely user responses and preferences, creating 'synthetic cohorts' aligned with observed demographics and behavior patterns.

Designers used the synthetic feedback to winnow down candidates, then ran reduced live A/B tests on the top contenders. This two-stage approach cut the number of live experiments by 60% and shortened iteration cycles from weeks to days. However, the article notes significant caveats: generative models reflected training-data biases, tended to over-prioritize certain phrasings, and occasionally hallucinated unrealistic user motivations.

To mitigate risk, QuillCut added human-in-the-loop validation, enforced demographic stratification in the model prompts, and treated synthetic feedback as a directional signal rather than a replacement for real-world testing. The article outlines a practical workflow for integrating synthetic users into product design systems and highlights governance steps product teams should take when using AI to inform UX decisions.