Abstract:Recently reinforcement learning (RL) has emerged as a promising approach for quadrupedal locomotion, which can save the manual effort in conventional approaches such as designing skill-specific controllers. However, due to the complex nonlinear dynamics in quadrupedal robots and reward sparsity, it is still difficult for RL to learn effective gaits from scratch, especially in challenging tasks such as walking over the balance beam. To alleviate such difficulty, we propose a novel RL-based approach that contains an evolutionary foot trajectory generator. Unlike prior methods that use a fixed trajectory generator, the generator continually optimizes the shape of the output trajectory for the given task, providing diversified motion priors to guide the policy learning. The policy is trained with reinforcement learning to output residual control signals that fit different gaits. We then optimize the trajectory generator and policy network alternatively to stabilize the training and share the exploratory data to improve sample efficiency. As a result, our approach can solve a range of challenging tasks in simulation by learning from scratch, including walking on a balance beam and crawling through the cave. To further verify the effectiveness of our approach, we deploy the controller learned in the simulation on a 12-DoF quadrupedal robot, and it can successfully traverse challenging scenarios with efficient gaits. We provide a video to show the learned gaits in different tasks in YouTube.11[Online]. Available: youtube.com/watch?vhgBLR09MEOw, and code is available in Github: github.com/PaddlePaddle/PaddleRobotics [Online]. Available: youtube.com/watch?vhgBLR09MEOw, and code is available in Github: github.com/PaddlePaddle/PaddleRobotics

ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills

Learning Dynamic Bipedal Walking Across Stepping Stones

Guided Curriculum Learning for Walking Over Complex Terrain

Learning 3D Bipedal Walking with Planned Footsteps and Fourier Series Periodic Gait Planning

Learning Bipedal Walking On Planned Footsteps For Humanoid Robots

Adaptive Step Duration for Precise Foot Placement: Achieving Robust Bipedal Locomotion on Terrains with Restricted Footholds

FootstepNet: an Efficient Actor-Critic Method for Fast On-line Bipedal Footstep Planning and Forecasting

Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments

Experience-Learning Inspired Two-Step Reward Method for Efficient Legged Locomotion Learning Towards Natural and Robust Gaits

Learning Agile Locomotion on Risky Terrains

Robust Walking and Sim-to-Real Optimization for Quadruped Robots via Reinforcement Learning

Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild

From Human Walking to Bipedal Robot Locomotion: Reflex Inspired Compensation on Planned and Unplanned Downsteps

DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning

Dynamic terrain traversal skills using reinforcement learning

Adaptive Gait Acquisition through Learning Dynamic Stimulus Instinct of Bipedal Robot

Reinforcement Learning for Blind Stair Climbing with Legged and Wheeled-Legged Robots

Steppability-informed Quadrupedal Contact Planning through Deep Visual Search Heuristics

Follow My Step: A Framework for Biped Robots to Imitate Human Walking

Reinforcement Learning With Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion

Motion planning for humanoid robots in complex environments based on stance sequence and ZMP reference