llama.cpp 3.0 speeds inference and adds Apple M3 / Apple Neural Engine optimizations

Tech · 4 min read

llama.cpp 3.0 introduces optimized kernels for Apple's M3 architecture and an NN acceleration path that leverages the Apple Neural Engine, delivering lower latency and reduced energy usage on compatible devices. The release also includes new quantization strategies that improve inference speed with minimal accuracy loss.

The project added better memory management for long-context workloads, enabling more stable on-device sessions for creative tools that keep iterative context (e.g., multi-turn prototyping assistants). Distributed inference helpers allow offloading heavier layers to remote accelerators when needed.

Maintainers stressed that community contributions were key to the release, including performance patches from independent developers and profiling data from teams integrating llama.cpp into design tooling. The release notes recommend tests per model and device to select the best quantization and kernel settings for production use.