Vociply speeds image inference on Graviton

Image inference on CPUs sounds boring next to all the GPU and AI-chip noise. But for a lot of startups, this is still where the bill lands — millions of small classification jobs, running all day, on plain cloud instances. That is the setup behind Arm’s new Vociply case study from April 29. The interesting part is not just that AWS Graviton got cheaper. It is that the biggest speedup came from finding the wrong bottleneck and fixing that first. ### What actually changed? Vociply was running a TensorFlow Lite image-classification pipeline on an AWS Graviton t4g instance and getting only 2.21 images per second. After profiling and rewriting part of the pipeline, throughput climbed to 3.11 images per second — about 40% faster — while estimated compute cost fell 29%. Arm framed the result as a real production-style optimization, not a synthetic benchmark. ### Where was the slowdown? Not in the model, which is the whole point. Arm’s profiling showed 65% of CPU time was stuck in Python loops during image normalization and preprocessing. TensorFlow Lite inference itself used only 22% of CPU time. So the expensive guess — “the model is slow on Arm” — turned out to be wrong. ### What fixed it? Vociply replaced the nested Python preprocessing loops with vectorized NumPy operations. That cut preprocessing’s share of total execution time from 65% to 28%. Basically, the CPU was spending less time trudging through interpreter overhead and more time doing useful math in optimized array code. The model stayed the same. The stack around it got smarter. ### Why does Arm Performix matter here? Arm Performix is Arm’s profiling toolkit for Arm-based cloud platforms like AWS Graviton. It is built to show function-level hotspots and microarchitecture behavior so developers can see where cycles are really going. In this case, that meant one profiling run exposed the bad assumption quickly, instead of sending the team into weeks of blind model tweaking. ### Why not just move to Graviton and stop there? Because migration and optimization are different jobs. AWS has been pushing Graviton for inference for years, with support in SageMaker and Arm64-ready containers for TensorFlow, PyTorch, XGBoost, and scikit-learn. AWS has also shown Graviton3 can cut inference costs by up to 50% versus comparable x86 instances in some workloads. at on the table. ### Is this just a one-off benchmark? Not really, though it is still a vendor-backed case study, so keep that frame in mind. The pattern matches a broader shift — Graviton is no longer just a “port your code and save some money” story. Arm says more than 50% of new AWS CPU capacity added in recent years is on Graviton, which means the ecosystem is now big enough that tooling and tuning matter more than before. ### Why does this matter for startups? Because image inference costs often scale in annoyingly small increments. Nobody notices at 100 requests. Everybody notices at 10 million. If a startup can get a 29% cost cut from software work on existing CPU instances, that is a very different decision from jumping straight to specialized inference chips or re-architecting the whole serving stack. AWS does have a bigger move with its own tooling path. ### So what is the real takeaway? The lesson is simple — on Arm CPUs, the model is not always the problem. Sometimes the money leak is the glue code around it. Vociply’s result matters because it turns Graviton optimization from a vague cloud-migration promise into a concrete playbook: profile first, find the hotspot, and fix the boring part that is quietly eating most of the CPU.