Abstract:This paper investigates the performance of several most popular deep reinforcement learning (DRL) algorithms applied to fluid flow and convective heat transfer systems, providing credible guidance and evaluation on their characteristics and performance. The studied algorithms are selected by considering the popularity, category, and advancement for guaranteeing the significance of the current study. The effectiveness and feasibility of all DRL algorithms are first demonstrated by studying a two-dimensional multi-heat-source cooling problem. Compared with the best manually optimized control, all DRL algorithms can find better control strategies that realize a further temperature reduction of 3–7 K. For problems with complex control objectives and environments, PPO (proximal policy optimization) shows an outstanding performance that accurately and dynamically constrains the oscillation of the solid temperature within 0.5 K around the target value, which is far beyond the capability of the manually optimized control. With the presented performance and the supplemented generalization test, the characteristic and specialty of the DRL algorithms are analyzed. The value-based methods have better training efficiency on simple cooling tasks with linear reward, while the policy-based methods show remarkable convergence on demanding tasks with nonlinear reward. Among the algorithms studied, the single-step PPO and prioritized experience replay deep Q-networks should be highlighted: the former has the advantage of considering multiple control targets and the latter obtains the best result in all generalization testing tasks. In addition, randomly resetting the environment is confirmed to be indispensable for the trained agent executing long-term control, which is strongly recommended to be included in follow-up studies.

A Study of Aero-Engine Control Method Based on Deep Reinforcement Learning.

A Research on Aero-engine Control Based on Deep Q Learning

Accelerating Reinforcement Learning with Local Data Enhancement for Process Control

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Acceleration control strategy for aero-engines based on model-free deep reinforcement learning method

Deep reinforcement learning implementation on IC engine idle speed control

Direct Thrust Inverse Control of Aero-Engine Based on Deep Neural Network

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

A Study on Aero-Engine Direct Thrust Control with Nonlinear Model Predictive Control Based on Deep Neural Network

Deep reinforcement learning based synthetic jet control on disturbed flow over airfoil

The Nonlinear Single Controller Of Dgen380 Aero Engine Design

Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing

Intelligent Control Strategy for Transient Response of a Variable Geometry Turbocharger System Based on Deep Reinforcement Learning

Performance analysis of reinforcement learning algorithms on intelligent closed-loop control on fluid flow and convective heat transfer

Deep reinforcement learning based active surge control for aeroengine compressors

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

Closed-loop forced heat convection control using deep reinforcement learning

Research on Hybrid Optimization and Deep Learning Modeling Method in the Performance Seeking Control

Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows

Active Disturbance Rejection Control Based on Deep Reinforcement Learning of PMSM for More Electric Aircraft

Accelerating and improving deep reinforcement learning-based active flow control: Transfer training of policy network