Abstract:Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinforcement Learning Optimal Control, which operates on two levels of abstraction and utilises a reduced number of controllers to solve nonlinear systems with unknown dynamics in continuous state and action spaces. Our approach is inspired by research at two levels of abstraction: first, at the level of limb coordination human behaviour is explained by linear optimal feedback control theory. Second, in cognitive tasks involving learning symbolic level action selection, humans learn such problems using model-free and model-based reinforcement learning algorithms. We propose that combining these two levels of abstraction leads to a fast global solution of nonlinear control problems using reduced number of controllers. Our framework learns the local task dynamics from naive experience and forms locally optimal infinite horizon Linear Quadratic Regulators which produce continuous low-level control. A top-level reinforcement learner uses the controllers as actions and learns how to best combine them in state space while maximising a long-term reward. A single optimal control objective function drives high-level symbolic learning by providing training signals on desirability of each selected controller. We show that a small number of locally optimal linear controllers are able to solve global nonlinear control problems with unknown dynamics when combined with a reinforcement learner in this hierarchical framework. Our algorithm competes in terms of computational cost and solution quality with sophisticated control algorithms and we illustrate this with solutions to benchmark problems.

Temporally Layered Architecture for Adaptive, Distributed and Continuous Control

Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architectures

Context-triggered Abstraction-based Control Design

Specification-guided temporal logic control for stochastic systems: a multi-layered approach

Towards a Theory of Control Architecture: A quantitative framework for layered multi-rate control

A Quantitative Framework for Layered Multirate Control: Toward a Theory of Control Architecture

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications

Augmented Lagrangian Methods as Layered Control Architectures

Control of Multi-Agent Systems with Finite Time Control Barrier Certificates and Temporal Logic

Temporal Difference Learning for Model Predictive Control

Unified Multi-Rate Control: from Low Level Actuation to High Level Planning

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL

Designing Temporal Controls

Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropout

Deep Reinforcement Learning Based Networked Control with Network Delays for Signal Temporal Logic Specifications

Intelligent temporal control

Context-Dependent Structure Control for Adaptive Behavior Selection

A Contract Theory for Layered Control Architectures

Hierarchical Meta-learning-based Adaptive Controller