Testing Invisible AI: UX Research for Contextual Recommender in Shoply

AI · 5 min read

Shoply rolled out a contextual recommender that surfaced product suggestions based on browsing context and session signals. Early metrics were promising for click-through rate, but follow-up interviews revealed users felt recommendations were 'creepy' when no explanation was provided. The UX team ran mixed-method studies—diary studies, intercept surveys, and moderated sessions—to understand trust thresholds and mental models around invisible AI.

Design responses favored low-friction transparency: subtle microcopy labeled elements like 'Recommended based on products you viewed' and an affordance to see 'Why this recommendation' with a concise rationale. The team also added a lightweight toggle to control personalization and a feedback mechanism to improve model relevance. The recommender's UI used clear affordances to signal sponsored content versus algorithmic picks.

Results showed that adding a one-line explanation and the ability to give thumbs-up/down increased sustained engagement by 14% and reduced opt-outs by 38%. The study illustrated that invisible AI performs better when users feel they understand the inputs and can correct the system, turning opaque personalization into a collaborative feature rather than a mysterious intrusion.