Abstract:Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinforcement Learning Optimal Control, which operates on two levels of abstraction and utilises a reduced number of controllers to solve nonlinear systems with unknown dynamics in continuous state and action spaces. Our approach is inspired by research at two levels of abstraction: first, at the level of limb coordination human behaviour is explained by linear optimal feedback control theory. Second, in cognitive tasks involving learning symbolic level action selection, humans learn such problems using model-free and model-based reinforcement learning algorithms. We propose that combining these two levels of abstraction leads to a fast global solution of nonlinear control problems using reduced number of controllers. Our framework learns the local task dynamics from naive experience and forms locally optimal infinite horizon Linear Quadratic Regulators which produce continuous low-level control. A top-level reinforcement learner uses the controllers as actions and learns how to best combine them in state space while maximising a long-term reward. A single optimal control objective function drives high-level symbolic learning by providing training signals on desirability of each selected controller. We show that a small number of locally optimal linear controllers are able to solve global nonlinear control problems with unknown dynamics when combined with a reinforcement learner in this hierarchical framework. Our algorithm competes in terms of computational cost and solution quality with sophisticated control algorithms and we illustrate this with solutions to benchmark problems.

Hierarchical Optimal Synchronization for Linear Systems Via Reinforcement Learning: A Stackelberg–Nash Game Perspective

Multiplayer Stackelberg-Nash Game for Nonlinear System via Value Iteration-Based Integral Reinforcement Learning

Two-Player Stackelberg Game for Linear System Via Value Iteration Algorithm

Data-Based Optimal Synchronization of Heterogeneous Multiagent Systems in Graphical Games Via Reinforcement Learning.

Adaptive Optimal Output-Feedback Consensus Tracking Control of Nonlinear Multiagent Systems Using Two-Player Stackelberg Game

Optimal Synchronization Control of Multiagent Systems with Input Saturation Via Off-Policy Reinforcement Learning.

Q-learning Solution for Optimal Consensus Control of Discrete-Time Multiagent Systems Using Reinforcement Learning

Online Synchronous Iterative Algorithm for Optimal Control of Stochastic Affine Nonlinear Systems

Optimal Leader-Following Consensus Control of Multi-Agent Systems: A Neural Network Based Graphical Game Approach

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

Optimal Synchronization Control of Heterogeneous Asymmetric Input-Constrained Unknown Nonlinear MASs via Reinforcement Learning

Data-Based Optimal Control of Multiagent Systems: A Reinforcement Learning Design Approach

Dual Performance Stackelberg Game for Unknown Nonlinear System with Saturating Input Constraint

Optimal Control and Filtering for Hierarchical Decision Problems with $H_{\infty }$ Constraint based on Stackelberg Strategy

Mixed H2/H∞ Control with Event-Triggered Mechanism for Nonlinear Stochastic Systems with Closed-Loop Stackelberg Games

Pareto-optimal Synchronization Control of Nonlinear Multi-Agent Systems Via Integral Reinforcement Learning

Observer-Based Human-in-the-Loop Optimal Output Cluster Synchronization Control for Multiagent Systems: A Model-Free Reinforcement Learning Method

A Model-free Framework for Nonlinear Optimal Control Based on Solving HJB Equation

Game-Based Backstepping Design for Strict-Feedback Nonlinear Multi-Agent Systems Based on Reinforcement Learning

Optimal Bipartite Consensus Control for Heterogeneous Unknown Multi-Agent Systems Via Reinforcement Learning

Hybrid Reinforcement Learning for Optimal Control of Non-Linear Switching System