Abstract:Proportional-Integral-Derivative (PID) control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, most traditional PID tuning methods rely on trial and error for complex processes where insights about the system are limited and may not yield the optimal PID parameters. To address the issue, this work proposes an automatic PID tuning framework based on reinforcement learning (RL), particularly the deterministic policy gradient (DPG) method. Different from existing studies on using RL for PID tuning, in this work, we explicitly consider the closed-loop stability throughout the RL-based tuning process. In particular, we propose a novel episodic tuning framework that allows for an episodic closed-loop operation under selected PID parameters where the actor and critic networks are updated once at the end of each episode. To ensure the closed-loop stability during the tuning, we initialize the training with a conservative but stable baseline PID controller and the resultant reward is used as a benchmark score. A supervisor mechanism is used to monitor the running reward (e.g., tracking error) at each step in the episode. As soon as the running reward exceeds the benchmark score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Moreover, we use layer normalization to standardize the input to each layer in actor and critic networks to overcome the issue of policy saturation at action bounds, to ensure the convergence to the optimum. The developed methods are validated through setpoint tracking experiments on a second-order plus dead-time system. Simulation results show that with our scheme, the closed-loop stability can be maintained throughout RL explorations and the explored PID parameters by the RL agent converge quickly to the optimum. Moreover, through simulation verification, the developed RL-based PID tuning method can adapt the PID parameters to changes in the process model automatically without requiring any knowledge about the underlying operating condition, in contrast to other adaptive methods such as the gain scheduling control.

An ADRC‐based PID Tuning Rule

A Recursive Tuning Approach for the Model-Free PID Controller Design

Robust PID Tuning Using Closed-Loop Identification

A Developed Method of Tuning PID Controllers with Fuzzy Rules for Integrating Processes.

New Tuning Methods of Both PID and ADRC for MIMO Coupled Nonlinear Uncertain Systems

A Design Method of Bang-Bang and PID Integrated Controller Based on Rough Set

Tuning Adrc for Multi-Variable System Based on Existing Pid Parameters

On Tracking and Antidisturbance Ability of PID Controllers

From PID to Active Disturbance Rejection Control

Development status of active disturbance rejection control

From PID to ADRC and back: Expressing error‐based active disturbance rejection control schemes as standard industrial 1DOF and 2DOF controllers

Quantitative Tuning of Active Disturbance Rejection Controller for FOPTD Model With Application to Power Plant Control

Robust Adaptive Iterative Learning Control for Discrete‐time Nonlinear Systems with Both Parametric and Nonparametric Uncertainties

Robust Fine Tuning of Optimal PID Controller With Guaranteed Robustness

Towards a Theoretical Foundation of PID Control for Uncertain Nonlinear Systems

A Tuning Method of Active Disturbance Rejection Control for a Class of High-Order Processes.

Stability-preserving automatic tuning of PID control with reinforcement learning

Extended PID Control of Nonlinear Uncertain Systems

Control of Nonlinear Uncertain Systems by Extended PID

Tracking performance of PID for nonlinear stochastic systems

A new PID controller design with constraints on relative delay margin for first-order plus dead- time systems