Abstract:The learning inefficiency of reinforcement learning (RL) from scratch hinders its practical application towards continuous robotic tracking control, especially for high-dimensional robots. This work proposes a data-informed residual reinforcement learning (DR-RL) based robotic tracking control scheme applicable to robots with high dimensionality. The proposed DR-RL methodology outperforms common RL methods regarding sample efficiency and scalability. Specifically, we first decouple the original robot into low-dimensional robotic subsystems; and further utilize one-step backward (OSBK) data to construct incremental subsystems that are equivalent model-free representations of the above decoupled robotic subsystems. The formulated incremental subsystems allow for parallel learning to relieve computation load and offer us mathematical descriptions of robotic movements for conducting theoretical analysis. Then, we apply DR-RL to learn the tracking control policy, a combination of incremental base policy and incremental residual policy, under a parallel learning architecture. The incremental residual policy uses the guidance from the incremental base policy as the learning initialization and further learns from interactions with environments to endow the tracking control policy with adaptability towards dynamically changing environments. Our proposed DR-RL based tracking control scheme is developed with rigorous theoretical analysis of system stability and weight convergence. The effectiveness of our proposed method is validated numerically on a 7-DoF KUKA iiwa robot manipulator and experimentally on a 3-DoF robot manipulator that would fail for other counterpart RL methods.

Reinforcement learning for optimal tracking of large-scale systems with multitime scales

Operational Optimal Tracking Control for Industrial Multirate Systems Subject to Unknown Disturbances

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Reinforcement Learning for Input Constrained Sub-optimal Tracking Control in Discrete-time Two-time-scale Systems

Data-Efficient Off-Policy Learning for Distributed Optimal Tracking Control of HMAS with Unidentified Exosystem Dynamics.

Optimal Tracking Control of Nonlinear Multiagent Systems Using Internal Reinforce Q-Learning

Optimal Asymptotic Tracking Control for Nonzero-Sum Differential Game Systems with Unknown Drift Dynamics via Integral Reinforcement Learning

Iterative Learning Controllers for Discrete-Time Large-Scale Systems to Track Trajectories with Distinct Magnitudes.

Decentralized Tracking Optimization Control for Partially Unknown Fuzzy Interconnected Systems via Reinforcement Learning Method

Constrained Reinforcement Learning-Based Closed-Loop Reference Model for Optimal Tracking Control of Unknown Continuous-Time Systems

Model-based reinforcement learning for infinite-horizon approximate optimal tracking

NN Adaptive Optimal Tracking Control for a Class of Uncertain Nonstrict Feedback Nonlinear Systems

Robust Output Regulation and Reinforcement Learning-Based Output Tracking Design for Unknown Linear Discrete-Time Systems

Decentralized Optimal Tracking Control of Large-Scale Nonlinear Systems

Dynamic Event-Triggered Reinforcement Learning Control of Stochastic Nonlinear Systems

Optimal Robust Online Tracking Control for Space Manipulator in Task Space Using Off-Policy Reinforcement Learning

Learning-based optimal control of linear time-varying systems over large time intervals

Data Informed Residual Reinforcement Learning for High-Dimensional Robotic Tracking Control

Iterative Learning Tracking for Multisensor Systems: A Weighted Optimization Approach

Reduced-dimensional reinforcement learning control using singular perturbation approximations

Optimal Tracking Control Problem for Linear State and Control Time-Delay Systems