Google Gemma 4 speeds phone inference

Phone AI has a boring problem that turns into a huge product problem — waiting. Even when a model is small enough to run on-device, it still tends to generate text one token at a time, which means lots of tiny pauses and lots of memory traffic. This week Google laid out how Gemma 4 attacks that bottleneck with multi-token prediction, or MTP, and speculative decoding. The pitch is simple: get up to 3x faster inference on local hardware, including phones, without changing the final model’s answers. ### What did Google actually ship? Google didn’t announce a brand-new model family this week. Gemma 4 itself launched on April 2, 2026, but Google has now published the implementation details and developer docs for the speed trick built into every Gemma 4 model — E2B, E4B, 31B, and 26B A4B. Those models include a dedicated draft model that works alongside the main model during generation. (blog.google) ### Why are phones slow in the first place? The bottleneck usually is not pure compute. It’s memory bandwidth. A model has to keep pulling weights and activations through relatively constrained mobile memory systems, and that happens again and again for each generated token. So even if the chip is capable, the model spends too much time waiting on data movement. That is why “just use a smaller model” only helps so much. (blog.google) ### So what is multi-token prediction? Normally, an autoregressive model predicts one next token, then repeats. MTP adds a drafter that predicts several future tokens in a row. Think of it as a fast assistant sketching the next few words while the main model checks the work. If the main model agrees, it can accept the whole drafted chunk in one pass — and even emit an extra token beyond that. That collapses several slow steps into one. (blog.google) ### Why does speculative decoding matter? Speculative decoding is the mechanism that makes the trick safe. The small drafter proposes tokens, but the larger target model still verifies them. So the speedup does not come from lowering the bar on quality. It comes from reducing how often the expensive model has to do full sequential generation. Google’s docs frame this as significantly faster inference with no quality loss, which is the key reason developers will care. (blog.google) ### Why is this a bigger deal on phones? On a server with lots of high-bandwidth memory, token-by-token generation is painful but manageable. On a phone, every wasted pass hurts more. Faster local decoding means assistants feel less laggy, voice and camera workflows can respond in tighter loops, and developers can keep more interactions on-device instead of bouncing to the cloud for responsiveness. Google is explicitly positioning Gemma 4 for agentic workflows on mobile and edge devices. (ai.google.dev) ### Does this change app design? Basically, yes. If local inference gets meaningfully faster, teams can revisit where they split work between device and server. Some features that used to require cloud calls for acceptable latency may now fit on-device, which helps with privacy, offline use, and cost control. The catch is that model size, RAM limits, thermal constraints, and battery still matter — this is a speedup, not magic. (developers.googleblog.com) ### What should developers watch next? The real question is how close the “up to 3x” number gets to everyday app behavior. That will depend on hardware, prompt shape, sequence length, and how often the drafter’s guesses get accepted. But the direction is clear — model architecture is starting to matter as much as raw parameter count for mobile AI. Faster local generation is no longer just a chip story. It’s becoming a model-design story too. (developers.googleblog.com) ### Bottom line? Google’s Gemma 4 update is really about making local AI feel usable, not just possible. If your app lives or dies on response time, shaving whole decoding steps matters more than another benchmark point. (blog.google)