Abstract:The policy gradient adaptive dynamic programming (PGADP) technique has gained recognition as an effective approach for optimizing the performance of nonlinear systems. Nonetheless, existing PGADP algorithms often demand a substantial volume of expensive or potentially risky interaction data with the system. Moreover, the utilization of neural networks in these algorithms can result in suboptimal learning efficiency and unstable training procedures. To address these challenges, a novel algorithm, referred to as OptNet-PGADP, has been introduced. This algorithm integrates an initially tailored control policy based on OptNet to tackle the optimization of control problems in discrete-time nonlinear systems. The OptNet-PGADP algorithm operates through a two-step process. Initially, the input–output trajectory of the system is computed using the nonlinear model predictive control (NMPC) method. Subsequently, an initial admissible control policy is acquired through OptNet. This policy is iteratively enhanced using the PGADP algorithm to attain the optimal controller. The resulting closed-loop control policy can be readily deployed in real-time applications. The implementation of the algorithm employs OptNet for the actor network and integrates an experience replay mechanism to bolster the controller’s learning efficiency. Furthermore, a convergence and optimality analysis of the algorithm is included. Simulation and experimental results conducted on two nonlinear systems conclusively demonstrate that the approach outperforms traditional PGADP and NMPC algorithms. These findings underscore the efficacy of OptNet-PGADP in mitigating the constraints of current methods and achieving superior control performance for nonlinear systems.

Discrete-Time Nonlinear Generalized Policy Iteration for Optimal Control Using Neural Networks

Infinite Horizon Self-Learning Optimal Control of Nonaffine Discrete-Time Nonlinear Systems

Modified general policy iteration based adaptive dynamic programming for unknown discrete‐time linear systems

Local Policy Iteration Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems

Discrete-Time Stable Generalized Self-Learning Optimal Control With Approximation Errors.

Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

Discrete-Time Optimal Control Via Local Policy Iteration Adaptive Dynamic Programming

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems with Trajectory-Based Initial Control Policy

Deterministic Policy Gradient Adaptive Dynamic Programming for Model-Free Optimal Control

Generalized Policy Iteration-based Reinforcement Learning Algorithm for Optimal Control of Unknown Discrete-time Systems

Neural-network-based Optimal Control for Discrete-Time Nonlinear Systems Using General Value Iteration

Policy Iteration Algorithm for Constrained Cost Optimal Control of Discrete-Time Nonlinear System

Discrete-time Generalized Policy Iteration ADP Algorithm with Approximation Errors.

Nonlinear Neuro-Optimal Tracking Control via Stable Iterative Q-Learning Algorithm

Twin Deterministic Policy Gradient Adaptive Dynamic Programming for Optimal Control of Affine Nonlinear Discrete-time Systems

A Generalized Policy Iteration Adaptive Dynamic Programming Algorithm for Optimal Control of Discrete-Time Nonlinear Systems with Actuator Saturation.

Adaptive Optimal Control for a Class of Nonlinear Systems: the Online Policy Iteration Approach.

A Novel Policy Iteration Algorithm for Nonlinear Continuous-Time H$\infty$ Control Problem

Finite-approximation-error-based Discrete-Time Iterative Adaptive Dynamic Programming.

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces