Abstract:Given that conventional model-based control methods have some limitations for dynamic systems with unknown model parameters and existing reinforcement learning methods do not take batch and time delay information into account, a novel data-based adaptive Q-learning approach with two-dimensional (2D) state and control policy is proposed to address the optimal tracking control issue for batch processes with time-invariant state delay. The extended delay state space equation, value function, Q function and optimal performance index are initially presented along the time and batch directions. By examining the correlation between the 2D value function and the 2D Q function, a delay-dependent 2D Bellman equation is designed independent of the process model, which is solved to obtain the expression of the control law. Without requiring prior knowledge of the system, the optimal gain matrices of the control law are further learned by using the current and historical state, output error values and time delay information of the timewise and batchwise. It is feasible to achieve accelerated convergence and reduced errors between the optimal control gain matrices and the learning gain matrices, hence enhancing the tracking capabilities of the systems. At the same time, the unbiasedness and convergence of the given adaptive Q-learning approach are strictly proved. The effectiveness of the proposed algorithm is ultimately validated by simulation comparisons of injection molding, specifically regarding the convergence of control gains and the tracking of output.

Nearly Data-Based Optimal Control for Linear Discrete Model-Free Systems with Delays Via Reinforcement Learning

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Time-delayed Feedback Control Optimization for Quasi Linear Systems under Random Excitations

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

Average Cost Optimal Control of Stochastic Systems Using Reinforcement Learning

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Adaptive Constrained Optimal Control Design for Data-Based Nonlinear Discrete-Time Systems With Critic-Only Structure

Data-driven Adaptive Optimal Control for Discrete-Time Linear Time-Invariant Systems

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Optimal tracking control of batch processes with time-invariant state delay: Adaptive Q-learning with two-dimensional state and control policy

Data-Driven Networked Optimal Iterative Learning Control for Discrete Linear Time-Varying Systems with One-Operation Bernoulli-Type Communication Delays

Neural-network-based Optimal Control for Discrete-Time Nonlinear Systems Using General Value Iteration

A Time-Delay Modeling Approach for Data-Driven Predictive Control of Continuous-Time Systems

Data-Based Optimal Tracking Control of Nonaffine Nonlinear Discrete-Time Systems.

Adaptive Neural Network Optimal Backstepping Control of Strict Feedback Nonlinear Systems Via Reinforcement Learning

Data-Driven Optimal Control of Bilinear Systems

Hybrid Reinforcement Learning for Optimal Control of Non-Linear Switching System

Learning Optimal Control Policy for Unknown Discrete-Time Systems

Adaptive dynamic programming-based optimal control for nonlinear state constrained systems with input delay

Data-driven Optimal Preview Output Tracking of Linear Discrete-time Systems