A/B Test Retrospective: Social App Experiment That Improved Community Retention

Tech · 7 min read

Discovery analytics showed that new users often failed to find active, relevant communities and would churn after a few passive scrolls. The team hypothesized that better surfacing of active conversations and personalized group previews would increase engagement. They developed three variations: activity-first sorting, curated snippets highlighting conversation tone, and a personalized 'match score' based on early interactions.

Tests ran across 60,000 users and were augmented with qualitative sessions to understand perception versus behavior. The activity-first variant yielded the strongest lift in immediate engagement, while curated snippets improved time-to-first-post. The combined redesign integrated the best elements and added onboarding nudges that suggested three groups tailored to each new user, with one-click join and starter prompts.

The final outcome improved 30-day retention in recommended groups by 17% and increased new-post creation by 22%. Beyond numbers, the retrospective emphasized the value of iterative testing with rapid prototypes and the need to measure both behavioral signals and sentiment to avoid superficial gains that don't foster long-term belonging.