CoreML Adds 4x Overhead to Neural Engine
Reverse-engineering of Apple's M4 Neural Engine (ANE) reveals that while the hardware is "ridiculously efficient," the CoreML software layer adds a 2-4x overhead for small operations. This suggests the ANE's full potential is currently bottlenecked by software, with a new CoreAI framework expected at WWDC to address it.
The Apple M4's Neural Engine is rated for 38 trillion operations per second (TOPS), a 60x increase over the first A11 Bionic ANE. However, reverse-engineering analysis reveals its true peak performance is 19 TFLOPS at FP16 precision, achieving a remarkable 6.6 TFLOPS/W at a peak power of just 2.8 watts. This efficiency is roughly 80 times greater per FLOP than an NVIDIA A100 datacenter GPU, highlighting a design philosophy prioritizing power efficiency for on-device tasks over raw throughput. The current CoreML framework, which has been the primary interface for developers since 2017, acts as a high-level abstraction layer. While this simplifies integration, it also obscures the ANE's true capabilities and introduces significant overhead. The software is responsible for compiling and optimizing models, but for smaller, rapid operations, this abstraction layer is the source of the 2-4x performance bottleneck, preventing direct, low-level access to the hardware. This software limitation is a critical cross-functional challenge, as the hardware's potential is gated by the software stack. Apple's historical strength lies in this tight hardware-software co-design, from the A-series and M-series chips to frameworks like Metal and CoreML. The upcoming CoreAI framework, expected with iOS 27, is anticipated to be a more modern, flexible system built for generative AI and large language models, aiming to better exploit the underlying silicon. The move from "Machine Learning" to "AI" in the framework's naming is significant, signaling a strategic shift to align with the broader industry narrative. CoreAI is expected to provide developers with more direct access to on-device foundation models and better integration with third-party models, a key feature for expanding AI capabilities in apps. This transition mirrors Apple's broader AI overhaul, including a more capable Siri and deeper integration of "Apple Intelligence" across its operating systems. On-device AI processing is also a cornerstone of Apple's supply chain and manufacturing strategy. The company leverages machine learning for predictive demand forecasting, inventory optimization, and automated warehousing systems. By designing its own silicon, Apple creates a vertically integrated ecosystem where hardware capabilities, like the Neural Engine, can be directly leveraged for operational efficiencies, from production lines to global logistics.
