Abstract:Given that conventional model-based control methods have some limitations for dynamic systems with unknown model parameters and existing reinforcement learning methods do not take batch and time delay information into account, a novel data-based adaptive Q-learning approach with two-dimensional (2D) state and control policy is proposed to address the optimal tracking control issue for batch processes with time-invariant state delay. The extended delay state space equation, value function, Q function and optimal performance index are initially presented along the time and batch directions. By examining the correlation between the 2D value function and the 2D Q function, a delay-dependent 2D Bellman equation is designed independent of the process model, which is solved to obtain the expression of the control law. Without requiring prior knowledge of the system, the optimal gain matrices of the control law are further learned by using the current and historical state, output error values and time delay information of the timewise and batchwise. It is feasible to achieve accelerated convergence and reduced errors between the optimal control gain matrices and the learning gain matrices, hence enhancing the tracking capabilities of the systems. At the same time, the unbiasedness and convergence of the given adaptive Q-learning approach are strictly proved. The effectiveness of the proposed algorithm is ultimately validated by simulation comparisons of injection molding, specifically regarding the convergence of control gains and the tracking of output.

A Model-Free Optimal Control Method With Fixed Terminal States and Delay

Time-delayed Feedback Control Optimization for Quasi Linear Systems under Random Excitations

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Model-free optimal controller for discrete-time Markovian jump linear systems: A Q-learning approach

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Optimal State Estimation Using Model-Free Reinforcement Learning

Optimal state-delay control in nonlinear dynamic systems

Q-Learning for Linear Quadratic Optimal Control with Terminal State Constraint

State estimation for control: an approach for output-feedback stochastic MPC

Learning-based optimal control of linear time-varying systems over large time intervals

Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approach

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Learning-Based Optimal Control with Performance Guarantees for Unknown Systems with Latent States

Efficient model-based reinforcement learning for approximate online optimal

Optimal tracking control of batch processes with time-invariant state delay: Adaptive Q-learning with two-dimensional state and control policy

An approach for regime-switching stochastic control problems with memory and terminal conditions

Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

Optimal consensus model-free control for multi-agent systems subject to input delays and switching topologies

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Terminal Constrained Robust Hybrid Iterative Learning Model Predictive Control for Complex Time-Delayed Batch Processes

Online Adaptive Optimization Algorithm for Semi-Markov Control Processes