Abstract:This paper presents a model-free reinforcement learning (RL) algorithm to solve the risk-averse optimal control (RAOC) problem for discrete-time nonlinear systems. While successful RL algorithms have been presented to learn optimal control solutions under epistemic uncertainties (i.e., lack of knowledge of system dynamics), they do so by optimizing the expected utility of outcomes, which ignores the variance of cost under aleatory uncertainties (i.e., randomness). Performance-critical systems, however, must not only optimize the expected performance, but also reduce its variance to avoid performance fluctuation during RL's course of operation. To solve the RAOC problem, this paper presents the following three variants of RL algorithms and analyze their advantages and preferences for different situations/systems: 1) a one-shot static convex program -based RL, 2) an iterative value iteration (VI) algorithm that solves a linear programming (LP) optimization at each iteration, and 3) an iterative policy iteration (PI) algorithm that solves a convex optimization at each iteration and guarantees the stability of the consecutive control policies. Convergence of the exact optimization problems, which are infinite-dimensional in all three cases, to the optimal risk-averse value function is shown. To turn these optimization problems into standard optimization problems with finite decision variables and constraints, function approximation for value estimations as well as constraint sampling are leveraged. Data-driven implementations of these algorithms are provided based on Q-function which enables learning the optimal value without any knowledge of the system dynamics. The performance of the approximated solutions is also verified through a weighted sup-norm bound and the Lyapunov bound. A simulation example is provided to verify the effectiveness of the presented approach.

Optimal Control of Two-Dimensional Roesser Model: Solution Based on Reinforcement Learning

Reinforcement Learning-Based $\mathcal{h}_{\infty }$ Control of 2-D Markov Jump Roesser Systems with Optimal Disturbance Attenuation

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

A Reinforcement Learning Method for LQR Control Problem

Model-Free Cooperative Optimal Output Regulation for Linear Discrete-Time Multi-Agent Systems Using Reinforcement Learning

Sublinear Regret for a Class of Continuous-Time Linear--Quadratic Reinforcement Learning Problems

Model-free design of stochastic LQR controller from a primal–dual optimization perspective

Optimal consensus control for unknown second-order multi-agent systems: Using model-free reinforcement learning method

Finite-region dissipative control for two-dimensional Roesser systems via Markov jumping mechanism

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Model‐free distributed optimal control for general discrete‐time linear systems using reinforcement learning

Model-Free Design of Stochastic LQR Controller from Reinforcement Learning and Primal-Dual Optimization Perspective

Learning-based optimal control of linear time-varying systems over large time intervals

Model-based reinforcement learning for infinite-horizon approximate optimal tracking

A Convex Programming Approach to Data-Driven Risk-Averse Reinforcement Learning

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Two‐loop reinforcement learning algorithm for finite‐horizon optimal control of continuous‐time affine nonlinear systems