Abstract:Deep reinforcement learning (RL) is a data-driven method capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications. In particular, the present work is motivated by the goal of reducing energy dissipation in turbulent flows, and the example considered is the spatiotemporally chaotic dynamics of the Kuramoto–Sivashinsky equation (KSE). A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system, making it costly when the system is computationally or experimentally expensive. We mitigate this challenge in a data-driven manner by combining dimensionality reduction via an autoencoder with a neural ODE framework to obtain a low-dimensional dynamical model from just a limited data set. We substitute this data-driven reduced-order model (ROM) in place of the true system during RL training to efficiently estimate the optimal policy, which can then be deployed on the true system. For the KSE actuated with localized forcing ('jets') at four locations, we demonstrate that we are able to learn a ROM that accurately captures the actuated dynamics as well as the underlying natural dynamics just from snapshots of the KSE experiencing random actuations. Using this ROM and a control objective of minimizing dissipation and power cost, we extract a control policy from it using deep RL. We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system. We show that this forced equilibrium captured in the ROM and discovered through RL is related to an existing known equilibrium solution of the natural KSE.

Reinforcement learning-based estimation for spatio-temporal systems

Reinforcement learning-based estimation for partial differential equations

Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

State Estimation For Nonlinear Systems By Using Dynamic Optimization

Estimating spatio-temporal fields through reinforcement learning

Lyapunov-Based Reinforcement Learning State Estimator

Reinforced Optimal Estimator

Dual parametric and state estimation for partial differential equations

LEARNEST: LEARNing Enhanced Model-based State ESTimation for Robots using Knowledge-based Neural Ordinary Differential Equations

Reinforcement Learning-Based Optimal Sensor Placement for Spatiotemporal Modeling

Sparse Reduced-Order Modelling: Sensor-Based Dynamics to Full-State Estimation

Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs

Reduced-order modeling of unsteady fluid flow using neural network ensembles

Reinforcement Learning Approach to Estimation in Linear Systems

Reinforcement Learning Based on Real-Time Iteration NMPC

Deep Reinforcement Learning for Online Control of Stochastic Partial Differential Equations

Deep Reinforcement Learning Sensor Scheduling for Effective Monitoring of Dynamical Systems

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization

PASE: An autonomous sequential framework for the state estimation of dynamical systems

Model-Based Reinforcement Learning via Stochastic Hybrid Models