Abstract:Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinforcement Learning Optimal Control, which operates on two levels of abstraction and utilises a reduced number of controllers to solve nonlinear systems with unknown dynamics in continuous state and action spaces. Our approach is inspired by research at two levels of abstraction: first, at the level of limb coordination human behaviour is explained by linear optimal feedback control theory. Second, in cognitive tasks involving learning symbolic level action selection, humans learn such problems using model-free and model-based reinforcement learning algorithms. We propose that combining these two levels of abstraction leads to a fast global solution of nonlinear control problems using reduced number of controllers. Our framework learns the local task dynamics from naive experience and forms locally optimal infinite horizon Linear Quadratic Regulators which produce continuous low-level control. A top-level reinforcement learner uses the controllers as actions and learns how to best combine them in state space while maximising a long-term reward. A single optimal control objective function drives high-level symbolic learning by providing training signals on desirability of each selected controller. We show that a small number of locally optimal linear controllers are able to solve global nonlinear control problems with unknown dynamics when combined with a reinforcement learner in this hierarchical framework. Our algorithm competes in terms of computational cost and solution quality with sophisticated control algorithms and we illustrate this with solutions to benchmark problems.

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Online Reinforcement Learning Neural Network Controller Design for Nanomanipulation

Learning-Based Neural Dynamic Surface Predictive Control for MMC

Reinforcement Learning for Adaptive Optimal Control of Unknown Continuous-Time Nonlinear Systems with Input Constraints.

Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Online adaptive learning of optimal control solutions using integral reinforcement learning

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

Online adaptive data-driven control for unknown nonlinear systems with constrained-input

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Hierarchical Optimal Synchronization for Linear Systems Via Reinforcement Learning: A Stackelberg–Nash Game Perspective

Hierarchical Control of Multi-Agent Systems using Online Reinforcement Learning

Adaptive Neural Network Optimal Backstepping Control of Strict Feedback Nonlinear Systems Via Reinforcement Learning

Optimal Control for Continuous-Time Unknown Nonlinear Affine Systems: A $\Mathcal{q}$-Learning Approach