SketchGPT: on-device model converts hand-drawn UI sketches into interactive prototypes

AI · 5 min read

The core of SketchGPT is a compact transformer trained on pairs of sketch images and markup for common UI patterns. Running on modern phones and laptops, it recognizes components, assigns semantic roles, and generates tappable hotspots and navigation flows without sending images to the cloud.

Designers can sketch on paper or use a stylus; the app captures photos and returns a live prototype within seconds. Outputs include a vectorized layout, component suggestions, and a simple state machine for screen transitions that you can refine in-app or export to standard prototyping tools.

Because it runs locally, SketchGPT is positioned for sensitive projects where privacy matters. Limitations include lower fidelity on very abstract sketches and occasional misclassification of unconventional controls, but users appreciated how quickly it turns whiteboard scribbles into clickable demos.