A/B testing AI personalization vs rules: the Shoply experiment that surprised product teams

AI · 6 min read

Shoply ran a randomized experiment across their homepage and product pages to compare two approaches: an AI-driven recommender that optimized for predicted conversion and a rules-based engine that enforced diversity and brand constraints. The model provided more immediately relevant items, while the rules engine ensured curated exposure to key categories.

Initial results favored the AI: click-through rates on recommendations increased 18% and short-term conversion rose 11%. However, retention analyses at 30 and 90 days told a different story — users exposed to the rules-based recommendations had 7% higher repeat purchase rates and explored a wider range of product categories. Qualitative feedback suggested the algorithmic recommendations felt homogeneous over time.

The product team responded by designing a hybrid UX: AI recommendations filled the top slot for immediate relevance, while a visible 'Browse More' carousel used business rules to surface diverse or promotional content. They also added an explainer tooltip about why an item was recommended to increase transparency and trust.

Shoply's finding: optimizing solely for short-term lift can reduce long-term engagement. The final design decision balanced personalization and curated discovery through an interface that made trade-offs explicit to users and stakeholders.