DTC: Deep Tracking Control

Fabian Jenelten,Junzhe He,Farbod Farshidian,Marco Hutter

DOI: https://doi.org/10.1126/scirobotics.adh5401

IF: 25

2024-01-18

Science Robotics

Abstract:Legged locomotion is a complex control problem that requires both accuracy and robustness to cope with real-world challenges. Legged systems have traditionally been controlled using trajectory optimization with inverse dynamics. Such hierarchical model-based methods are appealing because of intuitive cost function tuning, accurate planning, generalization, and, most importantly, the insightful understanding gained from more than one decade of extensive research. However, model mismatch and violation of assumptions are common sources of faulty operation. Simulation-based reinforcement learning, on the other hand, results in locomotion policies with unprecedented robustness and recovery skills. Yet, all learning algorithms struggle with sparse rewards emerging from environments where valid footholds are rare, such as gaps or stepping stones. In this work, we propose a hybrid control architecture that combines the advantages of both worlds to simultaneously achieve greater robustness, foot-placement accuracy, and terrain generalization. Our approach uses a model-based planner to roll out a reference motion during training. A deep neural network policy is trained in simulation, aiming to track the optimized footholds. We evaluated the accuracy of our locomotion pipeline on sparse terrains, where pure data-driven methods are prone to fail. Furthermore, we demonstrate superior robustness in the presence of slippery or deformable ground when compared with model-based counterparts. Last, we show that our proposed tracking controller generalizes across different trajectory optimization methods not seen during training. In conclusion, our work unites the predictive capabilities and optimality guarantees of online planning with the inherent robustness attributed to offline learning.

robotics

What problem does this paper attempt to address?

The paper aims to address the motion control problem of legged robots in complex environments. Specifically, the researchers attempt to combine model-based methods (such as trajectory optimization) and model-free methods (such as reinforcement learning) to overcome the limitations that exist when each is used alone. 1. **Problems with model-based methods**: Although model-based methods (such as trajectory optimization) have advantages in designing robot actions, such as intuitive cost function adjustment, precise planning, and a broad research foundation, model mismatches and violations of assumptions often lead to operational failures. 2. **Problems with model-free methods**: Model-free methods (such as reinforcement learning) can generate motion strategies with unprecedented robustness and recovery capabilities, but it is difficult to train effective strategies in sparse reward environments (e.g., gaps or stepping stone terrains). To address the above issues, the research proposes a hybrid control architecture that combines model-based trajectory optimization with learning-based control strategies, thereby achieving greater robustness and terrain generalization while maintaining high precision. Specifically, a model-based planner is used during training to generate reference motions, and a deep neural network policy is trained to track these optimized foothold positions. This approach not only improves accuracy on sparse terrains but also demonstrates stronger robustness when facing slippery or deformable surfaces. Additionally, the proposed tracking controller can generalize to different trajectory optimization methods not seen during training. In summary, this work combines the predictive capability and optimality guarantees of online planning with the inherent robustness of offline learning.

DTC: Deep Tracking Control

DTC: Deep Tracking Control

Learning Accurate and Robust Velocity Tracking for Quadrupedal Robots

Perceptive Locomotion through Nonlinear Model Predictive Control

Dexterous Legged Locomotion in Confined 3D Spaces with Reinforcement Learning

Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion

Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains

RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion

A Real-Time Planning and Control Framework for Robust and Dynamic Quadrupedal Locomotion

CPG-Based Hierarchical Locomotion Control for Modular Quadrupedal Robots Using Deep Reinforcement Learning.

Terrain-Aware Quadrupedal Locomotion via Reinforcement Learning

RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control

DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies

A non-periodic planning and control framework of dynamic legged locomotion

Agile and versatile bipedal robot tracking control through reinforcement learning

Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response

Robust-Locomotion-by-Logic: Perturbation-Resilient Bipedal Locomotion via Signal Temporal Logic Guided Model Predictive Control

Multi-Layered Safety for Legged Robots via Control Barrier Functions and Model Predictive Control

CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion

LORM: a Novel Reinforcement Learning Framework for Biped Gait Control

Learning-based Hierarchical Control: Emulating the Central Nervous System for Bio-Inspired Legged Robot Locomotion