Designing for AI Uncertainty: Before/After Redesign of Copilot Chat Interface at NoteFlow

AI · 7 min read

NoteFlow’s original copilot chat looked and felt like a human conversation: a single stream of messages with confident answers. As the product scaled, users reported instances of hallucinations and misinterpretations that led to incorrect notes and action items. The product team decided the UI must communicate AI uncertainty upfront and offer controls to tune responses.

The redesign added three core elements: confidence badges (low/medium/high) attached to model responses, an expandable evidence panel showing source documents or query snippets, and a “Regenerate with stricter constraints” toggle that limited the model to cite sources and use shorter answers. The team also introduced inline edit affordances so users could correct outputs without starting over.

In a six-week pilot with 2,400 users, task completion for knowledge-heavy notes rose by 13%, and reports of critical hallucinations dropped by 62%. Qualitative feedback indicated users trusted the assistant more because they could inspect sources and easily re-run generation with stricter parameters. The confidence badges became a coaching signal: users learned to treat low-confidence answers as starting points, not final outputs.

The team documented product patterns for other AI surfaces: never present AI output as unquestionable fact, show provenance where possible, and provide simple remediations (regenerate, edit, or escalate). The redesign required engineering effort to attach metadata to responses, but NoteFlow estimates the trust gains will reduce costly downstream corrections.