Abstract:Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinforcement Learning Optimal Control, which operates on two levels of abstraction and utilises a reduced number of controllers to solve nonlinear systems with unknown dynamics in continuous state and action spaces. Our approach is inspired by research at two levels of abstraction: first, at the level of limb coordination human behaviour is explained by linear optimal feedback control theory. Second, in cognitive tasks involving learning symbolic level action selection, humans learn such problems using model-free and model-based reinforcement learning algorithms. We propose that combining these two levels of abstraction leads to a fast global solution of nonlinear control problems using reduced number of controllers. Our framework learns the local task dynamics from naive experience and forms locally optimal infinite horizon Linear Quadratic Regulators which produce continuous low-level control. A top-level reinforcement learner uses the controllers as actions and learns how to best combine them in state space while maximising a long-term reward. A single optimal control objective function drives high-level symbolic learning by providing training signals on desirability of each selected controller. We show that a small number of locally optimal linear controllers are able to solve global nonlinear control problems with unknown dynamics when combined with a reinforcement learner in this hierarchical framework. Our algorithm competes in terms of computational cost and solution quality with sophisticated control algorithms and we illustrate this with solutions to benchmark problems.

A successive approximation method in functional spaces for hierarchical optimal control problems and its application to learning

Further extensions on the successive approximation method for hierarchical optimal control problems and its application to learning

Nested Sparse Successive Galerkin Approximation for Nonlinear Optimal Control Problems

Iterative Learning Controllers for Discrete-Time Large-Scale Systems to Track Trajectories with Distinct Magnitudes.

Application of an adaptive model hierarchy to parametrized optimal control problems

Optimal Control and Filtering for Hierarchical Decision Problems with $H_{\infty }$ Constraint based on Stackelberg Strategy

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats

On the hierarchical optimal control of a chain of distributed systems

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

Adaptive optimal control of continuous-time nonlinear affine systems via hybrid iteration

Model-Free H∞ Optimal Tracking Control of Constrained Nonlinear Systems Via an Iterative Adaptive Learning Algorithm

Sparse successive approximation for nonlinear H2 and H∞ optimal control problems under residual errors

Trajectory Tracking within a Hierarchical Primitive-Based Learning Approach

Data-Driven Networked Optimal Iterative Learning Control for Discrete Linear Time-Varying Systems with One-Operation Bernoulli-Type Communication Delays

A Multilevel Approach for Stochastic Nonlinear Optimal Control

Solving high-dimensional Hamilton-Jacobi-Bellman equation with functional hierarchical tensor

Convergence Analysis in Sense of Lebesgue- P Norm of Decentralized Non-Repetitive Iterative Learning Control for Linear Large-Scale Systems

A comparison study of supervised learning techniques for the approximation of high dimensional functions and feedback control

Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control.

Multi-objective optimal control for a class of unknown nonlinear systems based on finite-approximation-error ADP algorithm