Abstract:To date, applications of deep reinforcement learning (DRL) to active flow control (AFC) have been largely achieved via model-free DRL wherein the agent's policy is optimized through direct interactions with the actual physical system represented by computational fluid dynamics solvers. However, high computational demands and tendency of numerical divergence can significantly compromise the effectiveness of model-free DRL as the Reynolds number increases. A model-based DRL paradigm, which utilizes neural ordinary differential equations (NODE) to develop an environment model through integration with dimensionality reduction, offers a promising way forward to overcome this problem. This study presents an inaugural application of NODE model-based DRL to control the vortex shedding process from a two-dimensional circular cylinder using two synthetic jet actuators at a freestream Reynolds number of 100. An action-informed episode-based NODE (AENODE) method is developed to overcome the error cascading effect caused by recursive predictions in the existing studies, which typically adopt a single-step prediction NODE (denoted as the time step-based NODE (TNODE) in this paper). Both the AENODE and TNODE methods are employed in this study, and they are amalgamated with three distinct feature extraction approaches, expert-placed velocity sensors, proper orthogonal decomposition, and autoencoders, to construct six low-dimensional dynamical models (LDMs) of the DRL environment. It is found that AENODE resulted in over 90% fewer prediction errors at the end of an episode than TNODE with all LDMs via effectively mitigating the accumulation of long-term prediction errors associated with the recursive use of TNODE, leading to a more robust convergence in training the agents throughout repeated runs. Furthermore, the model-based DRL with either AENODE or TNODE is capable of identifying very similar control strategies to that obtained by the model-free DRL. The AENODE agents achieved 66.2%-72.4% of the rewards obtained by the model-free DRL, whereas the TNODE agents attained merely 43.4%-54.7%, indicating that AENODE provides a more accurate modeling of environment dynamics in DRL. It is also shown that completing a model-based DRL task using either TNODE or AENODE utilized only 10% of the data size requiring either 14% or 33% of the total wall-clock time required by the model-free DRL, and the actual time required for training the agents within the environment model was less than 1% of that required by the model-free DRL. Therefore, the AENODE method developed in this work not only enables a significant saving in computational costs but also outperforms the TNODE method in training convergence and reward. It represents a novel low-dimensional dynamical modeling method tailored for model-based DRL, which would enable the DRL-aided AFC to be applied to more complex flow scenarios occurring at high Reynolds numbers.

Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers

Applying Deep Reinforcement Learning To Active Flow Control In Weakly Turbulent Conditions

Deep Reinforcement Learning Finds a New Strategy for Vortex-Induced Vibration Control

Effect of Synthetic Jets Actuator Parameters on Deep Reinforcement Learning-Based Flow Control Performance in a Square Cylinder

Deep reinforcement transfer learning for active flow control of a 3D square cylinder under state dimension mismatch

Artificially Intelligent Control of Drag Reduction Around a Circular Cylinder Based on Wall Pressure Feedback

Deep reinforcement learning-based active flow control of an elliptical cylinder: transitioning from an elliptical cylinder to a circular cylinder and a flat plate

Deep Reinforcement Learning for Active Flow Control around a Circular Cylinder Using Unsteady-mode Plasma Actuators

Deep reinforcement learning-based active flow control of vortex-induced vibration of a square cylinder

Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing

Deep reinforcement learning based synthetic jet control on disturbed flow over airfoil

Active Flow Control for Bluff Body under High Reynolds Number Turbulent Flow Conditions Using Deep Reinforcement Learning

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

Active Flow Control Using Deep Reinforcement Learning with Time Delays in Markov Decision Process and Autoregressive Policy

An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control

Optimizing Flow Control with Deep Reinforcement Learning: Plasma Actuator Placement around a Square Cylinder

Towards Active Flow Control Strategies Through Deep Reinforcement Learning

Active Control for the Flow Around Various Geometries Through Deep Reinforcement Learning

Model-based Deep Reinforcement Learning for Active Control of Flow Around a Circular Cylinder Using Action-Informed Episode-Based Neural Ordinary Differential Equations

Accelerating and improving deep reinforcement learning-based active flow control: Transfer training of policy network

Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning