HelixAI open-sources Whisper-3D: audio-to-3D gesture model for AR design

AI · 4 min read

Whisper-3D is a lightweight model that takes speech or audio events and produces 3D motion curves for virtual objects and characters. Designed for AR and interactive design, the model outputs kinematic-friendly trajectories and timing gestures aligned to prosody.

The open-source release includes a dataset, training scripts, and a runtime optimized for mobile GPUs. HelixAI envisions use cases like voice-guided UI elements, gesture-driven narratives in AR storytelling, and quick prototyping for interaction designers.

Community contributors have already shared plugins that map Whisper-3D output to animation rigs in popular authoring tools. The team warns that audio-to-gesture mapping remains context-sensitive, so designers should treat outputs as strong starting points rather than final animations.