Abstract:Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinforcement Learning Optimal Control, which operates on two levels of abstraction and utilises a reduced number of controllers to solve nonlinear systems with unknown dynamics in continuous state and action spaces. Our approach is inspired by research at two levels of abstraction: first, at the level of limb coordination human behaviour is explained by linear optimal feedback control theory. Second, in cognitive tasks involving learning symbolic level action selection, humans learn such problems using model-free and model-based reinforcement learning algorithms. We propose that combining these two levels of abstraction leads to a fast global solution of nonlinear control problems using reduced number of controllers. Our framework learns the local task dynamics from naive experience and forms locally optimal infinite horizon Linear Quadratic Regulators which produce continuous low-level control. A top-level reinforcement learner uses the controllers as actions and learns how to best combine them in state space while maximising a long-term reward. A single optimal control objective function drives high-level symbolic learning by providing training signals on desirability of each selected controller. We show that a small number of locally optimal linear controllers are able to solve global nonlinear control problems with unknown dynamics when combined with a reinforcement learner in this hierarchical framework. Our algorithm competes in terms of computational cost and solution quality with sophisticated control algorithms and we illustrate this with solutions to benchmark problems.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Direct Optimization Based Compensation Adaptive Robust Control of Nonlinear Systems with State and Input Constraints

Integral reinforcement learning‐based optimal tracking control for uncertain nonlinear systems under input constraint and specified performance constraints

Optimization-based Adaptive Neural Sliding Mode Control for Nonlinear Systems with Fast and Accurate Response under State and Input Constraints

A hybrid model-based optimal control method for nonlinear systems using simultaneous dynamic optimization strategies

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

Adaptive Neural Network Optimal Backstepping Control of Strict Feedback Nonlinear Systems Via Reinforcement Learning

Reinforcement Learning for Adaptive Optimal Control of Unknown Continuous-Time Nonlinear Systems with Input Constraints.

Adaptive Optimal Tracking Controls of Unknown Multi-Input Systems Based on Nonzero-Sum Game Theory

Adaptive Identifier-Critic-Based Optimal Tracking Control for Nonlinear Systems with Experimental Validation

Adaptive Optimized Backstepping Control-Based RL Algorithm for Stochastic Nonlinear Systems with State Constraints and Its Application.

Model-Free H∞ Optimal Tracking Control of Constrained Nonlinear Systems Via an Iterative Adaptive Learning Algorithm

Reinforcement learning-based optimal adaptive fuzzy control for nonlinear multi-agent systems with prescribed performance

Neural network-based adaptive reinforcement learning for optimized backstepping tracking control of nonlinear systems with input delay

Robust Tracking Control with Reinforcement Learning for Nonlinear‐constrained Systems

Neural Network-Based Model-Free Adaptive Near-Optimal Tracking Control for a Class of Nonlinear Systems

Adaptive Optimal Tracking Control of Unknown Nonlinear Systems Using System Augmentation.

Event-Triggered H∞ Tracking Control of Nonlinear Systems Via Reinforcement Learning Method.

Generalised Policy Learning Based Adaptive Optimal Tracking Control of Nonlinear Servomechanisms

Fast and Optimal Adaptive Tracking Control: A Novel Meta-Reinforcement Learning via Conditional Generative Adversarial Net