Abstract:Proportional-Integral-Derivative (PID) control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, most traditional PID tuning methods rely on trial and error for complex processes where insights about the system are limited and may not yield the optimal PID parameters. To address the issue, this work proposes an automatic PID tuning framework based on reinforcement learning (RL), particularly the deterministic policy gradient (DPG) method. Different from existing studies on using RL for PID tuning, in this work, we explicitly consider the closed-loop stability throughout the RL-based tuning process. In particular, we propose a novel episodic tuning framework that allows for an episodic closed-loop operation under selected PID parameters where the actor and critic networks are updated once at the end of each episode. To ensure the closed-loop stability during the tuning, we initialize the training with a conservative but stable baseline PID controller and the resultant reward is used as a benchmark score. A supervisor mechanism is used to monitor the running reward (e.g., tracking error) at each step in the episode. As soon as the running reward exceeds the benchmark score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Moreover, we use layer normalization to standardize the input to each layer in actor and critic networks to overcome the issue of policy saturation at action bounds, to ensure the convergence to the optimum. The developed methods are validated through setpoint tracking experiments on a second-order plus dead-time system. Simulation results show that with our scheme, the closed-loop stability can be maintained throughout RL explorations and the explored PID parameters by the RL agent converge quickly to the optimum. Moreover, through simulation verification, the developed RL-based PID tuning method can adapt the PID parameters to changes in the process model automatically without requiring any knowledge about the underlying operating condition, in contrast to other adaptive methods such as the gain scheduling control.

Application of Reinforcement Learning-Based Adaptive PID Controller for Automatic Generation Control of Multi-Area Power System

Automatic generation control of is-landed micro-grid using integral reinforcement learning-based adaptive optimal control strategy

AdapSafe2: Prior-Free Safe-Certified Reinforcement Learning for Multi-Area Frequency Control

A Proposal of Adaptive PID Controller Based on Reinforcement Learning

Stability-preserving automatic tuning of PID control with reinforcement learning

Deep Deterministic Policy Gradient Reinforcement Learning Based Adaptive PID Load Frequency Control of an AC Micro-Grid

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

Adaptive Non-Linear Neural Control of Wide-Area Power Systems

Automatic Generation Control of a Multi-Area Hybrid Renewable Energy System Using a Proposed Novel GA-Fuzzy Logic Self-Tuning PID Controller

Improved frequency control performance of restructured multi-area power system through optimized controller design

Investigation of reciprocal rank method for automatic generation control in two-area interconnected power system

Generation Control of Multi-Area Multi-Source Power System using Fire-Fly Algorithm

Pelican Optimization Algorithm-Based Proportional–Integral–Derivative Controller for Superior Frequency Regulation in Interconnected Multi-Area Power Generating System

Adaptive PI Controller Based on a Reinforcement Learning Algorithm for Speed Control of a DC Motor

Comparing actor-critic deep reinforcement learning controllers for enhanced performance on a ball-and-plate system

A Model-Based Reinforcement Learning Approach for PID Design

Optimal design of automatic generation control based on BBPSO-tuned PI for a restructured environment

Coordinated automatic generation control of interconnected power system with imitation guided exploration multi-agent deep reinforcement learning

Reinforcement Learning in Adaptive Control of Power System Generation

Liberalized automatic generation control of interconnected thermal‐hydro‐gas system using firefly algorithm optimized PID controller

Adaptive Power System Emergency Control using Deep Reinforcement Learning