Sergey Levine on Robotic Foundation Models
In a recent talk, robotics expert Sergey Levine detailed how Robotic Foundation Models (RFMs) enable robots to generalize across tasks and sensor types with minimal fine-tuning. This transfer learning approach allows a single model to process vision, depth, and touch data, leading to more robust and flexible robot behavior in unstructured environments.
The push for Robotic Foundation Models mirrors the evolution of Large Language Models like GPT, shifting the paradigm from single-task, bespoke systems to general-purpose intelligence. This transition aims to democratize robotics development, allowing engineers to build complex applications on a pre-trained base rather than starting from scratch for every new function. The key challenge, unlike for LLMs that leverage vast internet text, is the scarcity of large-scale, diverse physical interaction data. Sergey Levine, a professor at UC Berkeley and co-founder of Physical Intelligence, is a central figure in this domain. His lab's work, including models like RT-2 developed with Google DeepMind, demonstrated how vision-language models could be adapted for end-to-end robotic control, directly outputting robot actions from visual and text inputs. His new company, Physical Intelligence, recently open-sourced its general-purpose model, π0 (pi-zero). Major tech companies are heavily invested in this race. Google's DeepMind has developed models like RT-1 and Gato, while NVIDIA is creating foundational models like GR00T to power a new wave of humanoid robots. This push is fueled by the need for more adaptable automation in sectors like logistics and manufacturing, where companies like Covariant are already deploying foundation models in warehouse settings. These models are evolving from Vision-Language Models (VLMs) into Vision-Language-Action (VLA) models. A VLA translates a high-level command, such as "pick up the apple," by processing real-time video and outputting the specific motor controls to execute the task. This integration of semantic understanding with physical action is what allows the models to generalize across different objects and environments. For students targeting this field, a strong foundation in Python and C++ is essential, alongside experience with machine learning frameworks like TensorFlow and PyTorch. Expertise in the Robot Operating System (ROS), linear algebra, and control theory provides the necessary background to build and control the physical systems that these advanced AI models operate. The development of RFMs is a critical enabler for the next generation of humanoid robots from companies like Tesla (Optimus) and Figure AI (Figure 01). These general-purpose robots rely on foundation models to perform a wide range of tasks in unpredictable human environments, moving beyond the structured confines of traditional industrial automation. A significant hurdle is bridging the "sim-to-real" gap, ensuring models trained on synthetic data perform reliably in the real world. Researchers are tackling this through techniques like data augmentation and improved simulators, as well as massive real-world data collection efforts like the Open X-Embodiment dataset, which aggregates data from dozens of different robots. Sergey Levine predicts that a "self-improvement flywheel" for general-purpose robots is imminent, with a median estimate of 2030 for robots to be capable of autonomously running a household. This rapid advancement hinges on scaling both the data collection and the hardware required to train and deploy these increasingly sophisticated foundation models.
