Microsoft and Nvidia partner on ONNX 2.0 with model shards for local inference in creative apps

Tech · 6 min read

Microsoft and Nvidia unveiled ONNX 2.0, an update to the interoperability format that introduces model shards: lightweight, composable blocks of model logic that can be executed locally on GPUs while heavier parts run in the cloud. The approach is aimed at apps like video editors and design suites that need instantaneous preview and protect user data.

Model shards can be mixed and matched, allowing developers to run a local sketching predictor for quick iterations while deferring compute-heavy final renders to managed endpoints. Nvidia showcased performance benchmarks where local shards reduced perceived latency by up to 70 percent in prototype editing tasks.

The partners also released developer tooling for converting large models into shardable ONNX graphs and published best practices for encryption, provenance, and privacy modes to ensure enterprises can safely distribute shards to endpoints with varying trust levels.