Abstract:Stable gait generation is a crucial problem for legged robot locomotion as this impacts other critical performance factors such as, e.g. mobility over an uneven terrain and power consumption. Gait generation stability results from the efficient control of the interaction between the legged robot's body and the environment where it moves. Here, we study how this can be achieved by a combination of model-predictive and predictive reinforcement learning controllers. Model-predictive control (MPC) is a well-established method that does not utilize any online learning (except for some adaptive variations) as it provides a convenient interface for state constraints management. Reinforcement learning (RL), in contrast, relies on adaptation based on pure experience. In its bare-bone variants, RL is not always suitable for robots due to their high complexity and expensive simulation/experimentation. In this work, we combine both control methods to address the quadrupedal robot stable gate generation problem. The hybrid approach that we develop and apply uses a cost roll-out algorithm with a tail cost in the form of a Q-function modeled by a neural network; this allows to alleviate the computational complexity, which grows exponentially with the prediction horizon in a purely MPC approach. We demonstrate that our RL gait controller achieves stable locomotion at short horizons, where a nominal MP controller fails. Further, our controller is capable of live operation, meaning that it does not require previous training. Our results suggest that the hybridization of MPC with RL, as presented here, is beneficial to achieve a good balance between online control capabilities and computational complexity.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the problem of stable gait generation for quadruped robots. Specifically, the authors have studied how to achieve this goal by combining Model - Predictive Control (MPC) and Predictive Reinforcement Learning (PRL) controllers. Stable gait generation is crucial for the movement of quadruped robots because it affects other key performance factors, such as maneuverability on uneven terrain and energy consumption.
### Background and Motivation
- **Complexity and Challenges**: The main challenge in the motion control of quadruped robots lies in their complex mechanical structures and numerous degrees of freedom, which makes the development of a control system capable of efficient movement and operation in a dynamically changing environment an important task in the field of quadruped robot research.
- **Limitations of Existing Methods**:
- **MPC**: Although MPC can effectively handle state - constraint management, its computational complexity increases exponentially with the growth of the prediction time horizon, and it may not reach the optimal solution within a short prediction time range.
- **RL**: RL depends on experience for adaptation, but in robot control, due to the high complexity of the robot itself and the expensive simulation/experiment costs, traditional RL methods are not always applicable.
### Solutions
- **Hybrid Method**: This paper proposes a hybrid method that combines MPC and RL, using the cost roll - out algorithm and a Q - function modeled by a neural network as the tail cost, thereby reducing the computational complexity of the pure MPC method.
- **Specific Contributions**:
- **Stable Gait Generation**: In a short prediction time range, the proposed RL gait controller can achieve stable movement, while the traditional MPC controller may fail in this case.
- **Real - Time Operation Capability**: This controller has real - time operation capability and does not require pre - training.
- **Balancing Control Capability and Computational Complexity**: By combining MPC with RL, this method achieves a good balance between online control capability and computational complexity.
### Experimental Verification
- **Experimental Setup**: The experiment was based on the A1 Unitree quadruped robot, using three frameworks: rcognita, ROS, and Quad - SDK.
- **Results**: The experimental results show that in a short prediction time range, Roll - Out Q - Learning (RQL) is significantly superior to the traditional MPC method in terms of cumulative running cost, especially in terms of attitude error (especially roll - angle error). And in a long prediction time range, the performance of the two methods tends to be the same.
### Conclusions
- **Future Prospects**: Although the current experiment is based on a simple Q - function model, the authors believe that a nonlinear Q - function model will further improve the performance of the system, especially in a long prediction time range.
In conclusion, this paper proposes a new hybrid control method by combining MPC and RL, which effectively solves the problem of stable gait generation for quadruped robots in complex environments.