Meta adds Scene Understanding API for AR effects on mobile devices

Tech · 4 min read

The Scene Understanding API is optimized for mobile cameras and includes fast plane and surface detection, per-pixel semantic segmentation for floors and walls, and occlusion mask outputs to anchor virtual objects convincingly.

Meta baked in power optimizations and a burst capture mode that reduces CPU load by offloading transient tasks to a camera ISP when available. The API supports Unity and native mobile SDKs, with bindings for popular AR engines.

Privacy is front and center: Meta's docs recommend processing segmentation locally and only sending anonymized metadata to servers. The company also provided tutorials on blending virtual content with real-world lighting using the new ambient sampling helpers.