Abstract:Active flow control for drag reduction with reinforcement learning (RL) is performed in the wake of a two-dimensional square bluff body at laminar regimes with vortex shedding. Controllers parametrised by neural networks are trained to drive two blowing and suction jets that manipulate the unsteady flow. The RL with full observability (sensors in the wake) discovers successfully a control policy that reduces the drag by suppressing the vortex shedding in the wake. However, a non-negligible performance degradation ( $\sim$ 50 % less drag reduction) is observed when the controller is trained with partial measurements (sensors on the body). To mitigate this effect, we propose an energy-efficient, dynamic, maximum entropy RL control scheme. First, an energy-efficiency-based reward function is proposed to optimise the energy consumption of the controller while maximising drag reduction. Second, the controller is trained with an augmented state consisting of both current and past measurements and actions, which can be formulated as a nonlinear autoregressive exogenous model, to alleviate the partial observability problem. Third, maximum entropy RL algorithms (soft actor critic and truncated quantile critics) that promote exploration and exploitation in a sample-efficient way are used, and discover near-optimal policies in the challenging case of partial measurements. Stabilisation of the vortex shedding is achieved in the near wake using only surface pressure measurements on the rear of the body, resulting in drag reduction similar to that in the case with wake sensors. The proposed approach opens new avenues for dynamic flow control using partial measurements for realistic configurations.

What problem does this paper attempt to address?

This paper aims to solve the problem of how to use Reinforcement Learning (RL) techniques to reduce the drag of bluff bodies under partially observable conditions. Specifically, the research focuses on reducing the drag in the wake of a two - dimensional square bluff body by controlling two blowing - suction injectors located at the trailing edge of the bluff body. In the laminar state, there is vortex shedding in the wake, which will lead to an increase in drag. The paper explores how to effectively apply reinforcement learning to optimize the control strategy to achieve a significant reduction in drag when the controller can only be trained based on partial measurements (for example, only the pressure sensor data on the object surface). ### The main contributions of the paper include: 1. **Improving RL control performance under partially observable conditions**: The paper proposes a method to improve the RL control performance in a partially measured environment by transforming the Partially Observable Markov Decision Process (POMDP) problem into a fully observable Markov Decision Process (MDP) problem. The specific method is to add past state and action information to the state vector to form an augmented state, so that the system can make better use of historical data for decision - making. 2. **Designing an efficient reward function**: In order to optimize the energy efficiency in the dynamic control system, the paper designs a reward function without the need to adjust hyper - parameters. This function not only considers the effect of drag reduction, but also considers the energy consumption of the controller. 3. **Using the maximum - entropy RL algorithm**: The paper adopts two maximum - entropy RL algorithms - Soft Actor - Critic (SAC) and Truncated Quantile Critics (TQC). These algorithms can efficiently explore large action spaces and discover near - optimal control strategies under partially observable conditions. Among them, TQC is considered as an improved version of SAC, with better sample efficiency and stability. ### Research background: - **The importance of drag reduction**: In vehicle aerodynamics, up to 50% of the total energy consumption is caused by aerodynamic drag. Therefore, reducing drag is of great significance for improving energy efficiency. - **Challenges in active flow control**: Although traditional passive control methods are effective, they cannot adapt to environmental changes, resulting in poor performance under different working conditions. Active control methods, especially RL - based control methods, can adjust the control strategy in real time to deal with complex flow environments. ### Experimental setup: - **Flow environment**: Two - dimensional Direct Numerical Simulations (DNS) are used to simulate the flow around a square bluff body, with a Reynolds number of 100. - **Control strategy**: The unstable flow is manipulated by two blowing - suction injectors, which are located at the top and bottom of the trailing edge of the bluff body. - **Observation and reward**: In a partially measured environment, the pressure sensors are only located on the downstream surface of the bluff body; while in a fully measured environment, the sensors are distributed in the wake area. The reward function is designed to maximize drag reduction and minimize energy consumption at the same time. ### Conclusion: The paper shows that under partially measured conditions, by using the maximum - entropy RL algorithm, the drag of bluff bodies can be effectively reduced, and the proposed framework has high sample efficiency and robustness. These results provide theoretical basis and technical support for future applications in more complex flow conditions.

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Active Flow Control for Bluff Body Drag Reduction Using Reinforcement Learning with Partial Measurements

Reinforcement Learning for Bluff Body Active Flow Control in Experiments and Simulations

Active Flow Control for Bluff Body under High Reynolds Number Turbulent Flow Conditions Using Deep Reinforcement Learning

Applying Deep Reinforcement Learning To Active Flow Control In Weakly Turbulent Conditions

Artificially Intelligent Control of Drag Reduction Around a Circular Cylinder Based on Wall Pressure Feedback

Deep reinforcement learning for active control of a three-dimensional bluff body wake

Reducing the Drag of a Bluff Body by Deep Reinforcement Learning

Active Flow Control Using Deep Reinforcement Learning with Time Delays in Markov Decision Process and Autoregressive Policy

Enhancing Heat Transfer from a Circular Cylinder Undergoing Vortex Induced Vibration Based on Reinforcement Learning

Machine-learning flow control with few sensor feedback and measurement noise

Deep Reinforcement Learning Finds a New Strategy for Vortex-Induced Vibration Control

Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning

Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing

Deep reinforcement learning for turbulent drag reduction in channel flows

Active Control of Wake-Induced Vibration Using Deep Reinforcement Learning

Active Control for the Flow Around Various Geometries Through Deep Reinforcement Learning

Drag Reduction of a D-shaped Bluff-Body Using Linear Parameter Varying Control

Acoustics-based Active Control of Unsteady Flow Dynamics using Reinforcement Learning Driven Synthetic Jets

Deep reinforcement transfer learning of active control for bluff body flows at high Reynolds number

Efficient Active Flow Control Strategy for Confined Square Cylinder Wake Using Deep Learning-Based Surrogate Model and Reinforcement Learning