Abstract:Proportional-Integral-Derivative (PID) control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, most traditional PID tuning methods rely on trial and error for complex processes where insights about the system are limited and may not yield the optimal PID parameters. To address the issue, this work proposes an automatic PID tuning framework based on reinforcement learning (RL), particularly the deterministic policy gradient (DPG) method. Different from existing studies on using RL for PID tuning, in this work, we explicitly consider the closed-loop stability throughout the RL-based tuning process. In particular, we propose a novel episodic tuning framework that allows for an episodic closed-loop operation under selected PID parameters where the actor and critic networks are updated once at the end of each episode. To ensure the closed-loop stability during the tuning, we initialize the training with a conservative but stable baseline PID controller and the resultant reward is used as a benchmark score. A supervisor mechanism is used to monitor the running reward (e.g., tracking error) at each step in the episode. As soon as the running reward exceeds the benchmark score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Moreover, we use layer normalization to standardize the input to each layer in actor and critic networks to overcome the issue of policy saturation at action bounds, to ensure the convergence to the optimum. The developed methods are validated through setpoint tracking experiments on a second-order plus dead-time system. Simulation results show that with our scheme, the closed-loop stability can be maintained throughout RL explorations and the explored PID parameters by the RL agent converge quickly to the optimum. Moreover, through simulation verification, the developed RL-based PID tuning method can adapt the PID parameters to changes in the process model automatically without requiring any knowledge about the underlying operating condition, in contrast to other adaptive methods such as the gain scheduling control.

Adaptive temperature control of a reverse flow process by using reinforcement learning approach

A Reinforcement Learning-Based Temperature Control of Fluidized Bed Reactor in Gas-Phase Polyethylene Process

Accelerating Reinforcement Learning with Local Data Enhancement for Process Control

Adaptive Model Predictive Control Scheme for Partially Internal Thermally Coupled Air Separation Column

Controlling Rayleigh–Bénard convection via reinforcement learning

Reinforcement learning based temperature control of a fermentation bioreactor for ethanol production

Control-Informed Reinforcement Learning for Chemical Processes

A Reinforcement Learning-based Economic Model Predictive Control Framework for Autonomous Operation of Chemical Reactors

Performance analysis of reinforcement learning algorithms on intelligent closed-loop control on fluid flow and convective heat transfer

Adaptive control for circulating cooling water system using deep reinforcement learning

Accelerating reinforcement learning with case-based model-assisted experience augmentation for process control

Closed-loop forced heat convection control using deep reinforcement learning

Model-Based Reinforcement Learning Control of Reaction-Diffusion Problems

Close d-loop force d heat convection control using deep reinforcement learning

Computational Discovery of Energy-Efficient Heat Treatment for Microstructure Design using Deep Reinforcement Learning

Control of Superheat of Organic Rankine Cycle under Transient Heat Source Based on Deep Reinforcement Learning

Fuzzy-based predictive deep reinforcement learning for robust and constrained optimal control of industrial solar thermal plants

Safe Transfer-Reinforcement-Learning-Based Optimal Control of Nonlinear Systems

Output Feedback Reinforcement Learning with Parameter Optimisation for Temperature Control in a Material Extrusion Additive Manufacturing system

Deep Reinforcement Learning-Pid Based Supervisor Control Method for Indirect-Contact Heat Transfer Processes in Energy Systems

Stability-preserving automatic tuning of PID control with reinforcement learning