DeepMind debuts SketchGen to turn hand-drawn UIs into functioning prototypes

AI · 5 min read

SketchGen accepts photos of sketches or whiteboard captures and returns a structured output that includes component hierarchies, probable states and suggested transitions. DeepMind showcased examples where rough marker sketches were turned into interactive web prototypes with minimal cleanup.

The model combines vision and program synthesis modules to infer layout semantics and output code snippets for common frameworks. DeepMind highlighted privacy features: processing can run on-premises for enterprise customers and the model provides confidence scores for each inferred element to help triage manual corrections.

Alongside the model release, DeepMind is publishing a dataset and evaluation suite for sketch-to-prototype conversion to encourage transparent benchmarking across the field. The company invites academic and studio partners to iterate on safety, accessibility and localization for different UI patterns.