AI Personalization vs. Predictability: Design Tradeoffs in a Fintech App

AI · 6 min read

The product team built a feed that used a hybrid model: collaborative filtering signals plus an LLM layer to generate contextual summaries and content prioritization. Early offline tests showed improvement in click-through predictions, but live experiments revealed mixed reactions: some users loved tailored tips, others found the feed unpredictable or opaque.

Designers introduced transparency affordances—clear labels like "Recommended for you" and a small "Why this" tap target that surfaced model signals and recent interaction triggers. They also added a simple "Reduce similar posts" control and a toggle to switch to chronological view. The team ran a staged A/B test with 40,000 users and tracked engagement, perceived usefulness via post-interaction micro‑surveys, and retention.

Results showed a 9% uplift in feed engagement for users who kept personalization on, but a 5% churn among power users who preferred predictable ordering. The pragmatic outcome was to expose control, lean into explainability, and limit personalization intensity during onboarding. The case illustrates necessary UX guardrails when AI features change product expectations.