Abstract:To date, applications of deep reinforcement learning (DRL) to active flow control (AFC) have been largely achieved via model-free DRL wherein the agent's policy is optimized through direct interactions with the actual physical system represented by computational fluid dynamics solvers. However, high computational demands and tendency of numerical divergence can significantly compromise the effectiveness of model-free DRL as the Reynolds number increases. A model-based DRL paradigm, which utilizes neural ordinary differential equations (NODE) to develop an environment model through integration with dimensionality reduction, offers a promising way forward to overcome this problem. This study presents an inaugural application of NODE model-based DRL to control the vortex shedding process from a two-dimensional circular cylinder using two synthetic jet actuators at a freestream Reynolds number of 100. An action-informed episode-based NODE (AENODE) method is developed to overcome the error cascading effect caused by recursive predictions in the existing studies, which typically adopt a single-step prediction NODE (denoted as the time step-based NODE (TNODE) in this paper). Both the AENODE and TNODE methods are employed in this study, and they are amalgamated with three distinct feature extraction approaches, expert-placed velocity sensors, proper orthogonal decomposition, and autoencoders, to construct six low-dimensional dynamical models (LDMs) of the DRL environment. It is found that AENODE resulted in over 90% fewer prediction errors at the end of an episode than TNODE with all LDMs via effectively mitigating the accumulation of long-term prediction errors associated with the recursive use of TNODE, leading to a more robust convergence in training the agents throughout repeated runs. Furthermore, the model-based DRL with either AENODE or TNODE is capable of identifying very similar control strategies to that obtained by the model-free DRL. The AENODE agents achieved 66.2%-72.4% of the rewards obtained by the model-free DRL, whereas the TNODE agents attained merely 43.4%-54.7%, indicating that AENODE provides a more accurate modeling of environment dynamics in DRL. It is also shown that completing a model-based DRL task using either TNODE or AENODE utilized only 10% of the data size requiring either 14% or 33% of the total wall-clock time required by the model-free DRL, and the actual time required for training the agents within the environment model was less than 1% of that required by the model-free DRL. Therefore, the AENODE method developed in this work not only enables a significant saving in computational costs but also outperforms the TNODE method in training convergence and reward. It represents a novel low-dimensional dynamical modeling method tailored for model-based DRL, which would enable the DRL-aided AFC to be applied to more complex flow scenarios occurring at high Reynolds numbers.

Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics

Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations

Reinforcement learning-based estimation for spatio-temporal systems

Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows

Reinforcement learning-based estimation for partial differential equations

Applying Deep Reinforcement Learning To Active Flow Control In Weakly Turbulent Conditions

Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems

Model-based Deep Reinforcement Learning for Active Control of Flow Around a Circular Cylinder Using Action-Informed Episode-Based Neural Ordinary Differential Equations

Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows

Data-based Suboptimal Neuro-control Design with Reinforcement Learning for Dissipative Spatially Distributed Processes

Data-Efficient Reinforcement Learning for Complex Nonlinear Systems

Data-driven low-dimensional dynamic model of Kolmogorov flow

Data-driven prediction of large-scale spatiotemporal chaos with distributed low-dimensional models

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Reinforcement Learning for Active Flow Control in Experiments

Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learning

Data-efficient Deep Reinforcement Learning with Expert Demonstration for Active Flow Control

Deep model predictive flow control with limited sensor data and online learning

Restoring chaos using deep reinforcement learning