Abstract:Learning to control unknown nonlinear dynamical systems is a fundamental problem in reinforcement learning and control theory. A commonly applied approach is to first explore the environment (exploration), learn an accurate model of it (system identification), and then compute an optimal controller with the minimum cost on this estimated system (policy optimization). While existing work has shown that it is possible to learn a uniformly good model of the system~\citep{mania2020active}, in practice, if we aim to learn a good controller with a low cost on the actual system, certain system parameters may be significantly more critical than others, and we therefore ought to focus our exploration on learning such parameters. In this work, we consider the setting of nonlinear dynamical systems and seek to formally quantify, in such settings, (a) which parameters are most relevant to learning a good controller, and (b) how we can best explore so as to minimize uncertainty in such parameters. Inspired by recent work in linear systems~\citep{wagenmaker2021task}, we show that minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular, task-dependent metric. Motivated by this, we develop an algorithm able to efficiently explore the system to reduce uncertainty in this metric, and prove a lower bound showing that our approach learns a controller at a near-instance-optimal rate. Our algorithm relies on a general reduction from policy optimization to optimal experiment design in arbitrary systems, and may be of independent interest. We conclude with experiments demonstrating the effectiveness of our method in realistic nonlinear robotic systems.

Policy Gradient Reinforcement Learning for Parameterized Continuous-Time Optimal Control

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Parameter learning: stochastic optimal control approach with reinforcement learning

Neural Network Approaches for Parameterized Optimal Control

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Reinforcement Learning-Based Direct Adaptive Optimal Control of JLQ Model

Effect of stable xenon inhalation on internal carotid artery blood flow in unanesthetized monkeys.

Policy gradient methods for discrete time linear quadratic regulator with random parameters

Optimal Exploration for Model-Based RL in Nonlinear Systems

Globally Convergent Policy Gradient Methods for Linear Quadratic Control of Partially Observed Systems

Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Physics‐informed reinforcement learning for optimal control of nonlinear systems