A/B Testing AI Assistants in Onboarding - What We Learned at a YC Startup

AI · 4 min read

The startup wanted to increase activation by introducing an AI assistant that could answer onboarding questions and offer shortcuts. Designers were skeptical: conversational UIs can be engaging, but they sometimes add a layer of indirection for straightforward tasks. The hypothesis was that the assistant would reduce friction for complex configurations.

They ran an A/B test with three cohorts: control, assistant, and inline micro-tutorials. Results were surprising: the assistant improved completion only for complex, multi-step setups by 14%, but reduced completion for basic tasks by 9% because users paused to 'ask' rather than follow a quick checklist. The micro-tutorial group had consistent small gains across the board.

The team concluded that AI assistants are best used as a supplemental layer activated for complexity signals, not a default onboarding channel. The practical product change was to surface micro-tutorials initially and present the assistant as an opt-in helper when the system detected hesitation or repeated failure patterns.