New RL Paper Tackles Action Representation
A new paper on action representations in deep reinforcement learning has been accepted to ICLR2026. The research from the Learning Systems and Robotics Lab analyzes how different action choices in SO(3) space impact RL stability, offering insights for building more robust AI tutors that handle complex, multi-step student interactions.
The way a reinforcement learning agent perceives and can act upon its environment is defined by its action space. For a robot arm, this might be joint positions or velocities, while for a wheeled robot, it could be wheel velocity. The choice of which actions are possible and how they are represented significantly impacts the agent's learning performance. This new research dives into the complexities of action representations for rotations in 3D space, known as SO(3). Representing these rotations is non-trivial because there isn't a perfect, one-to-one mapping that is always stable. Common methods like Euler angles have "gimbal lock" singularities, while quaternions can be ambiguous. The study systematically tested different SO(3) action representations with standard RL algorithms like PPO and SAC. The findings indicate that the geometric properties of the chosen representation heavily influence the agent's exploration and the overall stability of the learning process. This choice alone can be the difference between a policy that converges quickly and one that fails. For an AI reading tutor, the "action space" isn't physical but is equally complex, involving decisions on which phonics exercise to present, what level of text complexity to introduce, or when to offer encouragement. Each of these "actions" exists in a high-dimensional space of pedagogical choices, where one choice influences the next, shaping a student's learning trajectory over time. This is especially true for multi-step interactions with young learners, where the sequence and combination of content are crucial. Just as an improper action representation can destabilize a robot, a poorly defined action space for a tutor could lead to suboptimal learning paths, perhaps by repeatedly offering content that is too easy or too difficult. The paper's core insight—that the underlying geometry of the action space is critical for stable learning—suggests a new lens for edtech. By carefully designing the representation of pedagogical "actions," an AI tutor could more effectively navigate the vast space of possible teaching strategies to personalize the learning experience for each child. This research highlights the importance of how an RL agent's choices are framed. For an AI tutor, this could mean representing its next move not as a discrete content choice, but as a point in a continuous space of pedagogical dimensions like 'difficulty,' 'scaffolding,' and 'engagement.' Ultimately, this work underscores a key challenge in real-world reinforcement learning: sample inefficiency and safety. Just as a robot in the physical world needs to learn without causing damage, an AI tutor for children must adapt without causing frustration or disengagement, making the stability of the learning agent paramount.
