Google open-sources 'SketchFormer' for vector-to-vector UI transformations

AI · 4 min read

SketchFormer takes raster or vector sketches and outputs structured SVG-like components with named elements like buttons, inputs, and containers. The model pairs detection with layout inference, producing grouped components and suggested constraints to jumpstart prototyping.

Google released pre-trained checkpoints and a utility library to integrate SketchFormer with common design tools and code generators. The model includes heuristics to preserve user intent, such as distinguishing annotation text from labels and respecting approximate alignment in sketched grids.

Design teams experimenting with SketchFormer reported significant reductions in initial setup time; however, they still needed to refine spacing and accessibility attributes. Google is encouraging community contributions to broaden the model’s dataset and improve support for non-Western UI idioms.