On‑device LLMs go mainstream

For years, running a language model on a phone was a party trick. You could do it if you were willing to wrestle with quantized weights, terminal commands, and a device that got hot in your hand. What changed is not the basic idea. It is the packaging. A recent hands-on tutorial about running a local LLM on a phone lands in a world where Apple, Google, and open-source toolchains have already turned on-device inference into a supported developer path, not a research detour (youtube.com, developer.apple.com, developer.android.com). That shift matters because mainstream software does not win on possibility alone. It wins on friction. Apple now exposes the on-device model behind Apple Intelligence through its Foundation Models framework, with APIs for text generation, structured output, and tool calling inside apps on iPhone, iPad, Mac, and Vision Pro (developer.apple.com). Google has taken a parallel route on Android, putting Gemini Nano behind AICore and ML Kit GenAI APIs so developers can build local summarization, rewriting, image description, and speech transcription without shipping their own full inference stack (developer.android.com). Once the platform vendors do that work, “local LLM” stops sounding exotic. The open-source world has moved in the same direction. The llama.cpp project now documents Android builds through Termux and points developers toward more polished mobile experiences built on the same core runtime (github.com). MLC LLM has spent the last year turning mobile deployment into a cross-platform compiler problem, with one engine that targets iOS and Android GPUs as first-class citizens rather than afterthoughts (github.com). Google’s Gemma 3n developer guide makes the normalization explicit by listing Hugging Face Transformers, llama.cpp, Google AI Edge, Ollama, and MLX as ordinary parts of the on-device workflow (developers.googleblog.com). Once local inference becomes normal, the contest moves down a layer. The hard question is no longer whether a phone can produce tokens. The hard question is whether it can do that fast enough, cool enough, and cheaply enough to feel invisible. Google says Gemma 3n’s mobile-first design lets its E2B and E4B variants run with memory footprints comparable to 2B and 4B models, operating with as little as 2GB and 3GB of memory (developers.googleblog.com). Qualcomm’s own pitch for on-device generative AI is even more blunt: performance depends on splitting work across CPU, GPU, and NPU to improve thermal efficiency and battery life, because raw model quality means little if the device throttles after a minute (qualcomm.com). That is why benchmarks are changing too. A new benchmark called Mobile-MMLU does not just score answer quality. It tracks latency, energy consumption, and memory usage in realistic mobile tasks, which is exactly what product teams have to care about once these models leave the lab and enter apps people use on the train or in a grocery store (huggingface.co). The center of gravity is moving from model bragging rights to systems engineering. Developer tooling becomes the new leverage in that world. Apple is offering guided generation and adapter training around its system model, which means the company wants developers to shape behavior without touching the full model itself (developer.apple.com, developer.apple.com). Google is doing the same from the Android side with AICore, ML Kit, and AI Edge, where model updates, safety layers, and hardware access are wrapped in services developers do not have to build themselves (developer.android.com, developer.android.com). The tutorial format matters because it teaches developers to think of local inference as something you install, test, profile, and ship. On Android, the official llama.cpp docs now reduce the first step to three words that would have sounded absurd two years ago: install Termux, build, run (github.com).