Abstract:This paper addresses the challenge of terrain-adaptive dynamic locomotion in humanoid robots, a problem traditionally tackled by optimization-based methods or reinforcement learning (RL). Optimization-based methods, such as model-predictive control, excel in finding optimal reaction forces and achieving agile locomotion, especially in quadruped, but struggle with the nonlinear hybrid dynamics of legged systems and the real-time computation of step location, timing, and reaction forces. Conversely, RL-based methods show promise in navigating dynamic and rough terrains but are limited by their extensive data requirements. We introduce a novel locomotion architecture that integrates a neural network policy, trained through RL in simplified environments, with a state-of-the-art motion controller combining model-predictive control (MPC) and whole-body impulse control (WBIC). The policy efficiently learns high-level locomotion strategies, such as gait selection and step positioning, without the need for full dynamics simulations. This control architecture enables humanoid robots to dynamically navigate discrete terrains, making strategic locomotion decisions (e.g., walking, jumping, and leaping) based on ground height maps. Our results demonstrate that this integrated control architecture achieves dynamic locomotion with significantly fewer training samples than conventional RL-based methods and can be transferred to different humanoid platforms without additional training. The control architecture has been extensively tested in dynamic simulations, accomplishing terrain height-based dynamic locomotion for three different robots.

Biped walking on rough terfrain using reinforcement learning

Learning biped locomotion based on Q-learning and neural networks

Learning Gait-conditioned Bipedal Locomotion with Motor Adaptation

Learning Quadrupedal Locomotion on Tough Terrain Using an Asymmetric Terrain Feature Mining Network

Terrain-Aware Quadrupedal Locomotion via Reinforcement Learning

Walking with Terrain Reconstruction: Learning to Traverse Risky Sparse Footholds

Terrain Adaptive Walking of Biped Neuromuscular Virtual Human Using Deep Reinforcement Learning

Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild

A Heuristics-Based Reinforcement Learning Method to Control Bipedal Robots

A Biped Robot Learning to Walk like Human by Reinforcement Learning.

A Multi-Agent Reinforcement Learning Method for Omnidirectional Walking of Bipedal Robots

Learning 3D Bipedal Walking with Planned Footsteps and Fourier Series Periodic Gait Planning

Robust Walking and Sim-to-Real Optimization for Quadruped Robots via Reinforcement Learning

Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains

Learning Bipedal Walking for Humanoid Robots in Challenging Environments with Obstacle Avoidance

Online Learning of Uneven Terrain for Humanoid Bipedal Walking

Modeling And Fuzzy Q-Learning Control Of Biped Walking

Learning Dynamic Bipedal Walking Across Stepping Stones

Learning Bipedal Walking for Humanoids with Current Feedback

Bipedal walking pattern generation and control for humanoid robot with bivariate stability margin optimization

M-A3C: A Mean-Asynchronous Advantage Actor-Critic Reinforcement Learning Method for Real-Time Gait Planning of Biped Robot.