Abstract:This paper investigates the performance of several most popular deep reinforcement learning (DRL) algorithms applied to fluid flow and convective heat transfer systems, providing credible guidance and evaluation on their characteristics and performance. The studied algorithms are selected by considering the popularity, category, and advancement for guaranteeing the significance of the current study. The effectiveness and feasibility of all DRL algorithms are first demonstrated by studying a two-dimensional multi-heat-source cooling problem. Compared with the best manually optimized control, all DRL algorithms can find better control strategies that realize a further temperature reduction of 3–7 K. For problems with complex control objectives and environments, PPO (proximal policy optimization) shows an outstanding performance that accurately and dynamically constrains the oscillation of the solid temperature within 0.5 K around the target value, which is far beyond the capability of the manually optimized control. With the presented performance and the supplemented generalization test, the characteristic and specialty of the DRL algorithms are analyzed. The value-based methods have better training efficiency on simple cooling tasks with linear reward, while the policy-based methods show remarkable convergence on demanding tasks with nonlinear reward. Among the algorithms studied, the single-step PPO and prioritized experience replay deep Q-networks should be highlighted: the former has the advantage of considering multiple control targets and the latter obtains the best result in all generalization testing tasks. In addition, randomly resetting the environment is confirmed to be indispensable for the trained agent executing long-term control, which is strongly recommended to be included in follow-up studies.

Closed‐loop stability analysis of deep reinforcement learning controlled systems with experimental validation

Actor-Critic Reinforcement Learning for Control With Stability Guarantee

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Model Free Deep Deterministic Policy Gradient Controller for Setpoint Tracking of Non-minimum Phase Systems

Open-Loop Motion Control of a Hydraulic Soft Robotic Arm Using Deep Reinforcement Learning

Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions

Physical Deep Reinforcement Learning Towards Safety Guarantee

Performance analysis of reinforcement learning algorithms on intelligent closed-loop control on fluid flow and convective heat transfer

Safe and Reliable Training of Learning-Based Aerospace Controllers

A modular framework for stabilizing deep reinforcement learning control

Lyapunov-stable neural-network control

Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows

Closed-loop forced heat convection control using deep reinforcement learning

Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem

Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method

Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions

Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates

Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior

A framework for online, stabilizing reinforcement learning

Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems

Deep reinforcement learning with reward shaping for tracking control and vibration suppression of flexible link manipulator