Abstract:In this paper we propose a framework towards achieving two intertwined objectives: (i) equipping reinforcement learning with active exploration and deliberate information gathering, such that it regulates state and parameter uncertainties resulting from modeling mismatches and noisy sensory; and (ii) overcoming the huge computational cost of stochastic optimal control. We approach both objectives by using reinforcement learning to attain the stochastic optimal control law. On one hand, we avoid the curse of dimensionality prohibiting the direct solution of the stochastic dynamic programming equation. On the other hand, the resulting stochastic control inspired reinforcement learning agent admits the behavior of a dual control, namely, caution and probing, that is, regulating the state estimate together with its estimation quality. Unlike exploration and exploitation, caution and probing are employed automatically by the controller in real-time, even after the learning process is concluded. We use the proposed approach on a numerical example of a model that belongs to an emerging class in system identification. We show how, for the dimensionality of the stochastic version of this model, Dynamic Programming is prohibitive, Model Predictive Control requires an expensive nonlinear optimization, and a Linear Quadratic Regulator with the certainty equivalence assumption leads to poor performance and filter divergence, all contrasting our approach which is shown to be both: computationally convenient, stabilizing and of an acceptable performance.

Learning-based optimal control of linear time-varying systems over large time intervals

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

Learning Optimal Control Policy for Unknown Discrete-Time Systems

New Methods for Optimal Operational Control of Industrial Processes Using Reinforcement Learning on Two Time Scales

Optimization-Based Learning Control for Nonlinear Time-Varying Systems

Model-based reinforcement learning for infinite-horizon approximate optimal tracking

Reinforcement Learning Based Optimal Control of Linear Singularly Perturbed Systems

A Learning Controller for Periodic Disturbance Rejection of Uncertain Nonholonomic Systems

Optimal Control for Linear Time-Varying Systems with Multiple Time-Delays

Two‐loop reinforcement learning algorithm for finite‐horizon optimal control of continuous‐time affine nonlinear systems

Reinforcement Learning for Adaptive Optimal Control of Unknown Continuous-Time Nonlinear Systems with Input Constraints.

Model-Based Cross-Scale Reinforcement Learning Optimal Control

Learning-based Optimal Control for Linear Systems with Model Uncertainties

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Reinforcement Learning for Finite-Horizon H∞ Tracking Control of Unknown Discrete Linear Time-Varying System

Adaptive Optimal Control for Linear Discrete Time-Varying Systems

Learning-based Optimal Boundary Control for Parabolic Distributed Parameter System with Actuator Dynamics

Reinforcement learning-based optimal control of uncertain nonlinear systems