Choosing Between Synthetic and Real Data for Early AI UX: A Startup's Trade-offs

AI · 6 min read

When the startup built its first onboarding assistant, they faced a common trade-off: use synthetic data to iterate quickly or invest in real user data collection that would be slower and costlier. The product team prioritized speed and privacy early on, so initial prototypes trained on synthetic conversation logs generated by domain experts. The synthetic route accelerated iteration but produced edge-case behavior that looked plausible to engineers and alarming to pilot users.

Usability testing revealed two distinct UX problems: confident-sounding hallucinations when the model extrapolated beyond plausible user intents, and brittle phrasing sensitivity where phrasings unseen in synthetic logs caused the assistant to fail silently. The team paired designers, data scientists, and lawyers to map where synthetic data harmed perceived trust versus areas where it was acceptable. They defined a taxonomy of high-stakes intents that would require real example coverage and lower-stakes intents that could continue using synthetic augmentation.

The final decision was a hybrid pipeline: synthetic data for breadth and cold-start intents, targeted collection of real examples for the high-stakes taxonomy, and a UI pattern that surfaced confidence and source (e.g., “suggested based on patterns” vs “based on your account history”). After two months, task completion rose 22% and reported trust in the assistant increased by 18 percentage points in pilot cohorts. The case highlights that data strategy is a UX decision: choosing what users should see and trust informs both the model training plan and the interface language.