Google Launches Edge AI Runtime 'LiteRT'
Google has made its Distributed Cloud Edge generally available, enabling the deployment of AI agents closer to user devices. A key component is LiteRT, a new edge AI runtime set to replace TensorFlow Lite. LiteRT offers high-performance machine learning and generative AI on edge platforms, which is critical for developing low-latency and privacy-focused applications.
- LiteRT serves as the successor to TensorFlow Lite, expanding support beyond TensorFlow to include models from PyTorch, JAX, and Keras, reflecting a broader multi-framework vision for on-device AI. Existing applications using TensorFlow Lite will not be affected by the rebranding, but future updates and performance enhancements will be exclusive to LiteRT. - A key feature of LiteRT is its ability to abstract away the complexity of accessing Neural Processing Units (NPUs) from different chip manufacturers like MediaTek. This allows developers to leverage specialized AI hardware for more efficient, low-power execution of complex generative AI models on a wide range of edge devices. - Performance benchmarks show significant gains over previous on-device runtimes, with LiteRT delivering up to 1.4x faster GPU performance than the legacy TFLite GPU delegate. For generative AI models like Gemma, LiteRT has demonstrated 3x faster CPU performance and 7x faster GPU decoding speed compared to alternatives. - Google Distributed Cloud (GDC) Edge, the platform on which LiteRT operates, became generally available in March 2022. It provides fully managed hardware and software, including rack-based servers and smaller appliances, to run AI and 5G workloads at edge locations like retail stores and factory floors. - The move toward powerful edge runtimes like LiteRT aligns with a broader venture capital trend of investing in AI applications that run closer to the user. This focus on edge computing is driven by the need for lower latency, enhanced data privacy, and reduced costs associated with cloud-based inference for real-time applications. - For developers, Google provides an Agent Development Kit (ADK) to build and deploy custom AI agents on a managed, scalable environment called Vertex AI Agent Engine. This supports a code-first approach for creating agents that can orchestrate complex, multi-step workflows. - The architecture of modern edge AI systems increasingly relies on multi-agent collaboration, where specialized agents handle different tasks and coordinate responses. This approach, combined with on-device learning, enables more sophisticated and adaptive autonomous behavior in distributed systems. - The development of agentic AI for edge devices is a significant area of startup innovation, with companies like Stanhope AI raising millions in seed funding to create AI that can learn and adapt in dynamic physical environments, moving beyond the limitations of large language models.
