Abstract:To date, applications of deep reinforcement learning (DRL) to active flow control (AFC) have been largely achieved via model-free DRL wherein the agent's policy is optimized through direct interactions with the actual physical system represented by computational fluid dynamics solvers. However, high computational demands and tendency of numerical divergence can significantly compromise the effectiveness of model-free DRL as the Reynolds number increases. A model-based DRL paradigm, which utilizes neural ordinary differential equations (NODE) to develop an environment model through integration with dimensionality reduction, offers a promising way forward to overcome this problem. This study presents an inaugural application of NODE model-based DRL to control the vortex shedding process from a two-dimensional circular cylinder using two synthetic jet actuators at a freestream Reynolds number of 100. An action-informed episode-based NODE (AENODE) method is developed to overcome the error cascading effect caused by recursive predictions in the existing studies, which typically adopt a single-step prediction NODE (denoted as the time step-based NODE (TNODE) in this paper). Both the AENODE and TNODE methods are employed in this study, and they are amalgamated with three distinct feature extraction approaches, expert-placed velocity sensors, proper orthogonal decomposition, and autoencoders, to construct six low-dimensional dynamical models (LDMs) of the DRL environment. It is found that AENODE resulted in over 90% fewer prediction errors at the end of an episode than TNODE with all LDMs via effectively mitigating the accumulation of long-term prediction errors associated with the recursive use of TNODE, leading to a more robust convergence in training the agents throughout repeated runs. Furthermore, the model-based DRL with either AENODE or TNODE is capable of identifying very similar control strategies to that obtained by the model-free DRL. The AENODE agents achieved 66.2%-72.4% of the rewards obtained by the model-free DRL, whereas the TNODE agents attained merely 43.4%-54.7%, indicating that AENODE provides a more accurate modeling of environment dynamics in DRL. It is also shown that completing a model-based DRL task using either TNODE or AENODE utilized only 10% of the data size requiring either 14% or 33% of the total wall-clock time required by the model-free DRL, and the actual time required for training the agents within the environment model was less than 1% of that required by the model-free DRL. Therefore, the AENODE method developed in this work not only enables a significant saving in computational costs but also outperforms the TNODE method in training convergence and reward. It represents a novel low-dimensional dynamical modeling method tailored for model-based DRL, which would enable the DRL-aided AFC to be applied to more complex flow scenarios occurring at high Reynolds numbers.

Data-efficient Deep Reinforcement Learning with Expert Demonstration for Active Flow Control

Deep Reinforcement Learning Finds a New Strategy for Vortex-Induced Vibration Control

Applying Deep Reinforcement Learning To Active Flow Control In Weakly Turbulent Conditions

Active Control of Wake-Induced Vibration Using Deep Reinforcement Learning

Reinforcement Learning for Active Flow Control in Experiments

Reinforcement Learning for Bluff Body Active Flow Control in Experiments and Simulations

Deep Reinforcement Learning: A New Beacon for Intelligent Active Flow Control

Enhancing Heat Transfer from a Circular Cylinder Undergoing Vortex Induced Vibration Based on Reinforcement Learning

Active Flow Control Using Deep Reinforcement Learning with Time Delays in Markov Decision Process and Autoregressive Policy

Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers

Active Flow Control for Bluff Body Drag Reduction Using Reinforcement Learning with Partial Measurements

From Active Learning to Deep Reinforcement Learning: Intelligent Active Flow Control in Suppressing Vortex-Induced Vibration

Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning

Model-based Deep Reinforcement Learning for Active Control of Flow Around a Circular Cylinder Using Action-Informed Episode-Based Neural Ordinary Differential Equations

Deep Reinforcement Learning for Active Control of Flow over a Circular Cylinder with Rotational Oscillations

Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach

Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learning

Accelerating and improving deep reinforcement learning-based active flow control: Transfer training of policy network

An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control

Surrogate model-based deep reinforcement learning for experimental study of active flow control of circular cylinder