Abstract:High-performance optimization algorithms are essential in deep learning. However, understanding the behavior of optimization (i.e., learning process) remains challenging due to the instability and weak interpretability of the algorithms. Since gradient-based optimizations can be interpreted as continuous-time dynamical systems, applying feedback control to the dynamical systems that model the optimizers may provide another perspective for exploring more robust, accurate and explainable optimization algorithms. In this study, we present a framework for optimization called controlled heavy-ball optimizer. By employing the proportional-integral-derivative (PID) controller in the optimizer, we develop a deterministic continuous-time optimizer called Proportional-Integral-Derivative Accelerated Optimizer (PIDAO), and provide theoretical convergence analysis of PIDAO in unconstrained (non-)convex optimizations. As a byproduct, we derive PIDAO-family schemes for training deep neural networks by using specific discretization methods. Compared to classical optimizers, PIDAO can be empirically proven a more aggressive capacity to explore the loss landscape with lower computational costs due to the property of PID controller. Experimental evaluations demonstrate that PIDAO can accelerate the convergence and enhance the accuracy of deep learning, achieving state-of-the-art performance compared with advanced algorithms.

Research on a Proportional-Integral-derivative Neural Network Decoupling Control Based on Genetic Algorithm Optimization for Unified Chaotic System

Accelerated Optimization in Deep Learning with a Proportional-Integral-derivative Controller

A Proposal on Centralised and Distributed Optimisation Via Proportional-Integral-derivative Controllers (PID) Control Perspective

Novel PID Controller Parameters Tuning Method.

Real-Coded Adaptive Genetic Algorithm Applied to PID Parameter Optimization on a 6R Manipulators

A Hybrid Reactive Power Optimization Algorithm Based on Improved Genetic Algorithm and Primal-dual Interior Point Algorithm

Intelligent Optimization Design of PID Controller for Integrating Processes

A Neuron Pid Controller With Fuzzy Gain Scheduling And Its Applications In Electrohydraulic Velocity Servosystems

A Pid Controller With Neuron Tuning Parameters For Multi-Model Plants

Optimization of Decentralized Pi/Pid Controllers Based on Genetic Algorithm

PID Controller Design for Distribution Parameter Systems Based on Genetic Algorithms

Chaos Robust Anti-Control of Nonlinear Continuous Systems with Uncertainties

An improved control strategy of single neuron PID

Design Of Upfc Controller In Largescale Power Systems Based On Immune Genetic Algorithm

Application of Genetic Algorithm to PID Self-adaptive Control System

Hybrid Internal Model Control and Proportional Control of Chaotic Dynamical Systems.

The Adaptive Niche Genetic Algorithm for Optimum Design of PID Controller

PID Control Algorithm Based on Genetic Algorithm and Its Application in Electric Cylinder Control

Deterministic Learning-Based Neural PID Control for Nonlinear Robotic Systems

Simulation studies on the design of optimum PID controllers to suppress chaotic oscillations in a family of Lorenz-like multi-wing attractors

Adaptive robust PID control subject to supervisory decoupled sliding mode control based upon genetic algorithm optimization