Open-source AI model delivers real-time, personalized audio descriptions for web images

AI · 5 min read

An open-source project led by accessibility researchers and contributors from multiple organizations has released a realtime-capable model for generating personalized audio descriptions of images. Rather than producing a single, generic caption, the model can be tuned to a listener’s preferred level of detail, domain vocabulary (e.g., fashion vs. medical), and language register, enabling more useful and context-aware descriptions.

The architecture runs on-device for latency-sensitive flows and supports a privacy-preserving fallback that synthesizes descriptions on a trusted server when more compute is needed. Developers can plug the model into existing image components in design systems via a small API: supply user preferences and the image metadata, and receive a caption optimized for that profile.

Designers and accessibility teams report that the personalization reduces the frequency of “irrelevant” descriptions and increases task completion for discovery and shopping scenarios. The project also ships a test harness for validating description fidelity and a set of heuristics for when to offer longer-form narratives versus concise labels. Adoption is growing among CMS plugins and a few experimental newsrooms, with contributors prioritizing improvements for complex scenes and charts.