Abstract:The Bellman equation and its continuous form, the Hamilton-Jacobi-Bellman (HJB) equation, are ubiquitous in reinforcement learning (RL) and control theory. However, these equations quickly become intractable for systems with high-dimensional states and nonlinearity. This paper explores the connection between the data-driven Koopman operator and Markov Decision Processes (MDPs), resulting in the development of two new RL algorithms to address these limitations. We leverage Koopman operator techniques to lift a nonlinear system into new coordinates where the dynamics become approximately linear, and where HJB-based methods are more tractable. In particular, the Koopman operator is able to capture the expectation of the time evolution of the value function of a given system via linear dynamics in the lifted coordinates. By parameterizing the Koopman operator with the control actions, we construct a ``Koopman tensor'' that facilitates the estimation of the optimal value function. Then, a transformation of Bellman's framework in terms of the Koopman tensor enables us to reformulate two max-entropy RL algorithms: soft value iteration and soft actor-critic (SAC). This highly flexible framework can be used for deterministic or stochastic systems as well as for discrete or continuous-time dynamics. Finally, we show that these Koopman Assisted Reinforcement Learning (KARL) algorithms attain state-of-the-art (SOTA) performance with respect to traditional neural network-based SAC and linear quadratic regulator (LQR) baselines on four controlled dynamical systems: a linear state-space system, the Lorenz system, fluid flow past a cylinder, and a double-well potential with non-isotropic stochastic forcing.

Koopman-based Policy Iteration for Robust Optimal Control

Koopman System Approximation Based Optimal Control of Multiple Robots -- Part I: Concepts and Formulations

Data-Driven Stochastic Optimal Control using Linear Transfer Operators

Approximate Finite-Horizon Optimal Control with Policy Iteration

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control

Policy Iteration Algorithm for Online Design of Robust Control for a Class of Continuous-Time Nonlinear Systems

Koopman Representations of Dynamic Systems with Control

Data-Driven Stochastic Optimal Control with Safety Constraints using Linear Transfer Operators

Markov Chain Monte Carlo for Koopman-based Optimal Control: Technical Report

Robust Policy Optimization in Continuous-time Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ Stochastic Control

Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator

Koopman-Hopf Hamilton-Jacobi Reachability and Control

Optimal Synthesis of LTI Koopman Models for Nonlinear Systems with Inputs

Approximate Policy Iteration for Robust Stochastic Control of Multi-agent Markov Decision Processes

Extended Kalman filter -- Koopman operator for tractable stochastic optimal control

Connection among stochastic Hamilton-Jacobi-Bellman equation, path-integral, and Koopman operator on nonlinear stochastic optimal control

Koopman-based control using sum-of-squares optimization: Improved stability guarantees and data efficiency

Robust data-driven control for nonlinear systems using the Koopman operator

Koopman-Assisted Reinforcement Learning

From Optimization to Control: Quasi Policy Iteration

The Koopman Expectation: An Operator Theoretic Method for Efficient Analysis and Optimization of Uncertain Hybrid Dynamical Systems