Edge AI: on-device gets real

Edge AI used to mean a narrow class of tricks. A phone could spot a face in a camera frame. A speaker could wake on a keyword. A factory sensor could flag an anomaly. The hard part of AI still lived somewhere else, in a data center, behind an API. That line is starting to break. Developers are now building against on-device stacks that can run whole inference pipelines locally, with Google’s MediaPipe and the TensorFlow Lite line now folded into the newer LiteRT runtime as the most visible example. Google describes MediaPipe as a cross-platform set of reusable AI pipelines, and LiteRT as its high-performance on-device runtime built for low latency and privacy on billions of devices. (ai.google.dev) That shift matters because the old cloud pattern solved one problem by creating three more. Sending every camera frame, voice snippet, or user prompt to a server adds delay. It burns battery and bandwidth. It also turns routine product behavior into a privacy question. On-device inference changes the trade. The model sits on the phone, browser, embedded board, or robot, and the app can answer immediately without a round trip. Google’s current AI Edge material is explicit about the goal: chain multiple models, run accelerated pipelines on GPU and NPU hardware, and avoid blocking on the CPU. (ai.google.dev) What makes this moment different is that the models are no longer limited to tiny classifiers. Google’s MediaPipe LLM Inference tooling now supports running large language models completely on device for tasks like text generation, retrieval, and summarization. Google has also been moving its edge stack toward more open and specialized runtimes, including LiteRT-LM, while showcasing on-device retrieval-augmented generation and function calling in its own demo gallery. That is a big step up from the earlier era of edge ML, when most developers were wiring together detectors and embeddings rather than local assistants that can plan and act. (ai.google.dev) Once a local model can call tools, the edge stops being just a place to run inference and starts looking like a place to run software agents. Google’s AI Edge APIs repository now includes an on-device function-calling SDK and a separate on-device RAG SDK, both aimed at local agent-style workflows. The function-calling layer lets a model emit structured requests to external tools. The RAG layer gives it a way to search local documents and app data without shipping that context to the cloud. Those are the basic parts of an agent loop. They are also the parts that used to be easiest to justify only on a server. (github.com) That is why the smaller updates around edge tooling matter. They look minor if you treat edge AI as a deployment target. They look decisive if you treat it as a new runtime. Open source projects are already framing the problem this way: local-first agent runtimes for offline devices, tool use over local app data, memory stored on device, optional cloud fallback, and coordination across multiple edge nodes. Some of that language is aspirational. Some of it is rough prototype work. But the direction is clear. Developers are no longer asking only how to compress a model enough to fit on a device. They are asking how to let that device reason, retrieve, call functions, and coordinate with nearby systems while staying local by default. (github.com) The surprise is not that edge hardware got faster. That has been true for years. The surprise is that the software stack has finally caught up enough to make local AI feel like a product surface instead of a research demo. MediaPipe offers prebuilt tasks and full pipelines. LiteRT pushes hardware acceleration more directly through its newer CompiledModel API. Google’s own gallery is now using on-device RAG and on-device function calling as showcase features, which is another way of saying the demos have moved past “look, it runs” and into “look, it can do something.” (developers.googleblog.com)