Apple Silicon: bandwidth beats peak compute
A recent hands‑on video argues Apple Silicon on‑device ML performance is often limited more by memory bandwidth than by peak accelerator compute. The piece recommends profiling memory movement, minimizing CPU/GPU/neural copies and designing inference pipelines with memory pressure in mind (youtube.com).
Machine-learning chips can do math fast, but Apple Silicon often slows down when models wait on data to arrive from memory, not when they run out of arithmetic. (youtube.com) A recent hands-on video, “Apple Silicon Performance Explained: Bandwidth vs Compute,” walks through the roofline model, a standard way to test whether a workload is limited by memory traffic or raw compute. Its chapter list shows separate sections for “Memory bound kernel,” “Compute bound kernel,” a demo, and a comparison of memory bandwidth across Apple Silicon generations. (youtube.com) On Apple hardware, unified memory means the central processing unit and graphics processing unit share one memory pool instead of keeping separate copies. Apple’s MLX framework says arrays live in shared memory and can run on the central processing unit or graphics processing unit “without transferring data.” (ml-explore.github.io) That setup changes how local artificial-intelligence apps should be built. Apple’s Metal Performance Shaders Graph is designed to execute compute graphs across the graphics processing unit, central processing unit, and Neural Engine, and Apple’s Metal tools include profilers aimed at finding performance bottlenecks. (developer.apple.com, developer.apple.com) Apple’s own recent chip launches have leaned on bandwidth alongside accelerator gains. In October 2025, Apple said M5 raised unified memory bandwidth to 153 gigabytes per second, nearly 30 percent above M4, while also adding a 16-core Neural Engine and “Neural Accelerator” blocks in each graphics core. (apple.com) Apple Machine Learning Research has made the same software argument in public. A November 2025 post on running large language models with MLX said operations can run on the central processing unit or graphics processing unit “without needing to move memory around,” tying performance to the chip’s unified memory design. (machinelearning.apple.com) MLX’s documentation shows why that matters in practice. In one example measured on an M1 Max, splitting a matrix multiply to the graphics processing unit and a long sequence of small exponent operations to the central processing unit cut runtime to about 1.4 milliseconds from 2.8 milliseconds on the graphics processing unit alone. (ml-explore.github.io) The practical takeaway is narrower than “the Neural Engine does not matter.” The video argues developers should first measure memory movement, reduce copies between the central processing unit, graphics processing unit, and Neural Engine, and design inference pipelines around memory pressure before chasing headline tera-operations figures. (youtube.com) That leaves Apple Silicon with a familiar local-artificial-intelligence tradeoff: the chips keep getting faster at matrix math, but many on-device models still rise or fall on how efficiently software feeds those units from shared memory. (apple.com, ml-explore.github.io)
