Abstract:Reinforcement learning (RL) is a popular data-driven method that has demonstrated great success in robotics. Previous works usually focus on learning an end-to-end (direct) policy to directly output joint torques. While the direct policy seems convenient, the resultant performance may not meet our expectations. To improve its performance, more sophisticated reward functions or more structured policies can be utilized. This paper focuses on the latter because the structured policy is more intuitive and can inherit insights from previous model-based controllers. It is unsurprising that the structure, such as a better choice of the action space and constraints of motion trajectory, may benefit the training process and the final performance of the policy at the cost of generality, but the quantitative effect is still unclear. To analyze the effect of the structure quantitatively, this paper investigates three policies with different levels of structure in learning quadruped locomotion: a direct policy, a structured policy, and a highly structured policy. The structured policy is trained to learn a task-space impedance controller and the highly structured policy learns a controller tailored for trot running, which we adopt from previous work. To evaluate trained policies, we design a simulation experiment to track different desired velocities under force disturbances. Simulation results show that structured policy and highly structured policy require 1/3 and 3/4 fewer training steps than the direct policy to achieve a similar level of cumulative reward, and seem more robust and efficient than the direct policy. We highlight that the structure embedded in the policies significantly affects the overall performance of learning a complicated task when complex dynamics are involved, such as legged locomotion.

Learning Setup Policies: Reliable Transition Between Locomotion Behaviours

Learning When to Switch: Composing Controllers to Traverse a Sequence of Terrain Artifacts

Learning Bipedal Walking On Planned Footsteps For Humanoid Robots

Guided Curriculum Learning for Walking Over Complex Terrain

Dynamic terrain traversal skills using reinforcement learning

Learning Agile Locomotion on Risky Terrains

Training Transition Policies via Distribution Matching for Complex Tasks

Dynamic Bipedal Maneuvers through Sim-to-Real Reinforcement Learning

Terrain-Aware Quadrupedal Locomotion via Reinforcement Learning

Robust and Versatile Bipedal Jumping Control through Reinforcement Learning

Walking with Terrain Reconstruction: Learning to Traverse Risky Sparse Footholds

Learning Task Space Actions for Bipedal Locomotion

Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains

A Multiobjective Collaborative Deep Reinforcement Learning Algorithm for Jumping Optimization of Bipedal Robot

How does the structure embedded in learning policy affect learning quadruped locomotion?

RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control

Dexterous Legged Locomotion in Confined 3D Spaces with Reinforcement Learning

Learning Jumping Skills from Human with a Fast Reinforcement Learning Framework

DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning

Reuse Non-Terrain Policies for Learning Terrain-Adaptive Humanoid Locomotion Skills

DTC: Deep Tracking Control