Designing an AI Assistant’s Uncertainty UI: A Case Study from NoteAI

AI · 6 min read

When NoteAI shipped a general-purpose note summarizer, some summaries were confidently wrong. Users reacted by over-trusting or abandoning the assistant entirely. The design and ML teams collaborated to treat uncertainty as a first-class interaction problem rather than a backend-only metric.

We implemented three UI patterns: confidence badges (low/medium/high with short rationales), traceable source highlights showing which paragraphs fed the summary, and a lightweight correction flow that converted user edits into immediate feedback for the model. On the ML side we calibrated prediction probabilities with additional validation data and trained the UI to surface explanations only when confidence dipped below a tuned threshold.

In a 4-week rollout with 9,200 users, trust signals reduced incorrect-acceptance by 38% and lowered support escalations related to hallucinations by 27%. Qualitative feedback showed users appreciated transparency and the ability to correct summaries inline. The takeaway: honest, contextual uncertainty UI combined with calibrated models improves long-term adoption.