Dual Adaptive Control

Jun Liang

1997-01-01

Abstract:In this paper, some results on the principles; optimal policies and applications of dual adaptive control are surveyed for stochastic systems with uncertainty in the view of stochastic suboptimal control, and the research trends in the future are pointed.

What problem does this paper attempt to address?

An Alternative Dual Control Approach for Auto-Optimization in Uncertain Environments

Guoqiang Tan,Wen-Hua Chen,Jun Yang

DOI: https://doi.org/10.1109/iai63275.2024.10730016

2024-01-01

Abstract:This paper investigates the problem of dual control for exploitation and exploration (DCEE) for auto-optimization in uncertain environments. Different from existing adaptive control methods, an exploration effect is additionally considered in the DCEE framework. By providing an alternative dual control approach, the system output can be driven to the estimated nominal value and the uncertainty of the predicted optimal operational condition can be reduced. The convergence analysis of optimality tracking process of general linear systems is established using the DCEE framework. Simulation results are provided to verify the effectiveness of the developed DCEE.
Adaptive dual control with online outlier detection for uncertain systems

Xuehui Ma,Fucai Qian,Shiliang Zhang,Li Wu,Lei Liu

DOI: https://doi.org/10.1016/j.isatra.2022.01.021

IF: 7.3

2022-01-01

ISA Transactions

Abstract:This paper proposes an adaptive dual control with outlier detection that is robust to the occurrence of outliers in uncertain systems. Outliers occasionally exist in system process noise and observation noise, which could cause poor parameter estimation and degraded control performance of uncertain systems. For this reason, we devise an online outlier detection mechanism to filter the outliers so as to enhance the parameter estimation of uncertain systems. The devised mechanism makes decisions on outlier detection via the generated predicted regions where the newly arriving data is expected to locate, and the predicted regions are updated in real-time according to the historical data. The detection mechanism is integrated into the design of adaptive dual control, which is derived based on the bicriterial method. Compared with classical dual control merely considering uncertainty in input and output data stream, we are the first to include the uncontrollable excitations into the structure of dual control to fit practical scenarios, and this inclusion also provides an extensive cover on outliers to be detected. The improved performance of the proposed approach is verified using a mathematical model through one-time simulation and Monte Carlo simulations under different conditions, and we also evaluate our method in the control of fermentation sterilization process for more convincing results.

automation & control systems,instruments & instrumentation,engineering, multidisciplinary
Adaptive Control and Intersections with Reinforcement Learning

Anuradha M. Annaswamy

DOI: https://doi.org/10.1146/annurev-control-062922-090153

2023-05-04

Annual Review of Control, Robotics, and Autonomous Systems

Abstract:This article provides an exposition of the field of adaptive control and its intersections with reinforcement learning. Adaptive control and reinforcement learning are two different methods that are both commonly employed for the control of uncertain systems. Historically, adaptive control has excelled at real-time control of systems with specific model structures through adaptive rules that learn the underlying parameters while providing strict guarantees on stability, asymptotic performance, and learning. Reinforcement learning methods are applicable to a broad class of systems and are able to produce near-optimal policies for highly complex control tasks. This is often enabled by significant offline training via simulation or the collection of large input-state datasets. This article attempts to compare adaptive control and reinforcement learning using a common framework. The problem statement in each field and highlights of their results are outlined. Two specific examples of dynamic systems are used to illustrate the details of the two methods, their advantages, and their deficiencies. The need for real-time control methods that leverage tools from both approaches is motivated through the lens of this common framework. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 14 is May 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Dual Control of Exploration and Exploitation for Auto-Optimization Control with Active Learning

Zhongguo Li,Wen-Hua Chen,Jun Yang,Yunda Yan

DOI: https://doi.org/10.1109/tase.2024.3375373

IF: 6.636

2024-01-01

IEEE Transactions on Automation Science and Engineering

Abstract:The quest for optimal operation in environments with unknowns and uncertainties is highly desirable but critically challenging across numerous fields. This paper develops a dual control framework for exploration and exploitation (DCEE) to solve an auto-optimization problem in such complex settings. In general, there is a fundamental conflict between tracking an unknown optimal operational condition and parameter identification. The DCEE framework stands out by eliminating the need for additional perturbation signals, a common requirement in existing adaptive control methods. Instead, it inherently incorporates an exploration mechanism, actively probing the uncertain environment to diminish belief uncertainty. An ensemble based multi-estimator approach is developed to learn the environmental parameters and in the meanwhile quantify the estimation uncertainty in real time. The control action is devised with dual effects, which not only minimizes the tracking error between the current state and the believed unknown optimal operational condition but also reduces belief uncertainty by proactively exploring the environment. Formal properties of the proposed DCEE framework like convergence are established. A numerical example is used to validate the effectiveness of the proposed DCEE. Simulation results for maximum power point tracking are provided to further demonstrate the potential of this new framework in real world applications.
Rolling Horizon Stochastic Optimal Control Strategy for ACC and CACC under Uncertainty

Yang Zhou,Soyoung Ahn,Madhav Chitturi,David A. Noyce

DOI: https://doi.org/10.1016/j.trc.2017.07.011

IF: 9.022

2017-01-01

Transportation Research Part C Emerging Technologies

Abstract:This paper presents a rolling horizon stochastic optimal control strategy for both Adaptive Cruise Control and Cooperative Adaptive Cruise Control under uncertainty based on the constant time gap policy. Specifically, uncertainties that can arise in vehicle control systems and vehicle sensor measurements are represented as normally-distributed disturbances to state and measurement equations in a state-space formulation. Then, acceleration sequence of a controlled vehicle is determined by optimizing an objective function that captures control efficiency and driving comfort over a predictive horizon, constrained by bounded acceleration deceleration and collision protection. The optimization problem is formulated as a linearly constrained linear quadratic Gaussian problem and solved using a separation principle, Lagrangian relaxation, and Kalman filter. A sensitivity analysis and a scenario-based analysis via simulations demonstrate that the proposed control strategy can generate smoother vehicle control and perform better than a deterministic feedback controller, particularly under small system disturbances and large measurement disturbances. Published by Elsevier Ltd.
Dual Control of Exploration and Exploitation for Auto-Optimisation Control with Active Learning

Zhongguo Li,Wen-Hua Chen,Jun Yang,Yunda Yan

DOI: https://doi.org/10.1109/TASE.2024.3375373

2024-03-12

Abstract:The quest for optimal operation in environments with unknowns and uncertainties is highly desirable but critically challenging across numerous fields. This paper develops a dual control framework for exploration and exploitation (DCEE) to solve an auto-optimisation problem in such complex settings. In general, there is a fundamental conflict between tracking an unknown optimal operational condition and parameter identification. The DCEE framework stands out by eliminating the need for additional perturbation signals, a common requirement in existing adaptive control methods. Instead, it inherently incorporates an exploration mechanism, actively probing the uncertain environment to diminish belief uncertainty. An ensemble based multi-estimator approach is developed to learn the environmental parameters and in the meanwhile quantify the estimation uncertainty in real time. The control action is devised with dual effects, which not only minimises the tracking error between the current state and the believed unknown optimal operational condition but also reduces belief uncertainty by proactively exploring the environment. Formal properties of the proposed DCEE framework like convergence are established. A numerical example is used to validate the effectiveness of the proposed DCEE. Simulation results for maximum power point tracking are provided to further demonstrate the potential of this new framework in real world applications.

Systems and Control
Duality Between Large Deviation Control and Risk-Sensitive Control for Markov Decision Processes.

Yanan Dai,Jinwen Chen

DOI: https://doi.org/10.1016/j.sysconle.2023.105490

IF: 2.742

2023-01-01

Systems & Control Letters

Abstract:This paper studies the dual relation between large deviations control of maximizing “up-side chance” probability and risk-sensitive control for Markov Decision Processes. To derive the desired duality, we apply a non-linear extension of the Kreĭn–Rutman Theorem to characterize the optimal risk-sensitive value and prove that an optimal policy exists which is stationary and deterministic. Benchmarks in the “up-side chance” probability which make the duality hold are characterized. It is proved that the optimal policy for the “up-side chance” probability can be approximated by the optimal one for the risk-sensitive control. The right-hand derivative of the optimal risk-sensitive value function plays an important role, and a variational formula for the optimal risk-sensitive value is applied to characterize it. Some essential differences between these two types of optimal control problems are presented.
Universal Adaptive Control of Nonlinear Systems

Brett T. Lopez,Jean-Jacques E. Slotine

DOI: https://doi.org/10.48550/arXiv.2012.15815

2021-11-09

Abstract:This work develops a new direct adaptive control framework that extends the certainty equivalence principle to general nonlinear systems with unmatched model uncertainties. The approach adjusts the rate of adaptation online to eliminate the effects of parameter estimation transients on closed-loop stability. The method can be immediately combined with a previously designed or learned feedback policy if a corresponding model-parameterized Lyapunov function or contraction metric is known. Simulation results of various nonlinear systems with unmatched uncertainties demonstrates the approach.

Systems and Control,Robotics
Composite Adaptive Control for Time-varying Systems with Dual Adaptation

Raghavv Goel,Sayan Basu Roy

DOI: https://doi.org/10.48550/arXiv.2206.01700

2022-06-04

Abstract:This paper proposes a composite adaptive control architecture using dual adaptation scheme for dynamical systems comprising time-varying uncertain parameters. While majority of the adaptive control schemes in literature address the case of constant parameters, recent research has conceptualized improved adaptive control techniques for time-varying systems with rigorous stability proofs. The proposed work is an effort towards a similar direction, where a novel dual adaptation mechanism is introduced to efficiently tackle the time-varying nature of the parameters. Projection and $\sigma$-modification algorithms are strategically combined using congelation of variables to claim a global result for the tracking error space. While the classical adaptive systems demand a restrictive condition of persistence of excitation (PE) for accurate parameter estimation, the proposed work relies on a milder condition, called initial excitation (IE) for the same. A rigorous Lyapunov stability analysis is carried out to establish uniformly ultimately bounded (UUB) stability of the closed-loop system. Further it is analytically shown that the proposed work can recover the performance of previously designed IE-based adaptive controller in case of time invariant systems.

Systems and Control
Active Uncertainty Reduction for Human-Robot Interaction: An Implicit Dual Control Approach

Haimin Hu,Jaime F. Fisac

DOI: https://doi.org/10.48550/arXiv.2202.07720

2022-06-06

Abstract:The ability to accurately predict human behavior is central to the safety and efficiency of robot autonomy in interactive settings. Unfortunately, robots often lack access to key information on which these predictions may hinge, such as people's goals, attention, and willingness to cooperate. Dual control theory addresses this challenge by treating unknown parameters of a predictive model as stochastic hidden states and inferring their values at runtime using information gathered during system operation. While able to optimally and automatically trade off exploration and exploitation, dual control is computationally intractable for general interactive motion planning, mainly due to the fundamental coupling between robot trajectory optimization and human intent inference. In this paper, we present a novel algorithmic approach to enable active uncertainty reduction for interactive motion planning based on the implicit dual control paradigm. Our approach relies on sampling-based approximation of stochastic dynamic programming, leading to a model predictive control problem that can be readily solved by real-time gradient-based optimization methods. The resulting policy is shown to preserve the dual control effect for a broad class of predictive human models with both continuous and categorical uncertainty. The efficacy of our approach is demonstrated with simulated driving examples.

Robotics,Machine Learning,Systems and Control,Optimization and Control
Partially Observable Stochastic Optimal Control

Guangchen Wang,Jie Xiong,Shuaiqi Zhang

2016-01-01

International Journal of Numerical Analysis and Modeling

Abstract:This paper is survey on some recent results in optimal control and stochastic filtering. The goal is not to cover all recent developments in control and filtering, instead we focus on maximum principle for optimality of partial information backward or forward-backward stochastic differential equations and branching particle approximation of nonlinear filtering.
A Dual Adaptive Control Theory Inspired by Hebbian Associative Learning

Jun-e Feng,Chung Tin,Chi-Sang Poon

DOI: https://doi.org/10.1109/cdc.2009.5400831

2009-01-01

Abstract:Hebbian associative learning is a common form of neuronal adaptation in the brain and is important for many physiological functions such as motor learning, classical conditioning and operant conditioning. Here we show that a Hebbian associative learning synapse is an ideal neuronal substrate for the simultaneous implementation of high-gain adaptive control (HGAC) and model-reference adaptive control (MRAC), two classical adaptive control paradigms. The resultant dual adaptive control (DAC) scheme is shown to achieve superior tracking performance compared to both HGAC and MRAC, with increased convergence speed and improved robustness against disturbances and adaptation instability. The relationships between convergence rate and adaptation gain/error feedback gain are demonstrated via numerical simulations. According to these relationships, a tradeoff between the convergence rate and overshoot exists with respect to the choice of adaptation gain and error feedback gain.
Adaptive neural network control for uncertain dual switching nonlinear systems

Qianqian Mu,Fei Long,Lipo Mo,Liang Liu

DOI: https://doi.org/10.1038/s41598-022-21049-y

IF: 4.6

2022-10-06

Scientific Reports

Abstract:Dual switching system is a special hybrid system that contains both deterministic and stochastic switching subsystems. Due to its complex switching mechanism, few studies have been conducted for dual switching systems, especially for systems with uncertainty. Usually, the stochastic subsystems are described as Markov jump systems. Based upon the upstanding identity of RBF neural network on approaching nonlinear data, the tracking models for uncertain subsystems are constructed and the neural network adaptive controller is designed. The global asymptotic stability almost surely (GAS a.s.) and almost surely exponential stability (ES a.s.) of dual switching nonlinear error systems are investigated by using the energy attenuation theory and Lyapunov function method. An uncertain dual switching system with two subsystems, each with two modes, is studied. The uncertain functions of the subsystems are approximated well, and the approximation error is controlled to be below 0.05. Under the control of the designed adaptive controller and switching rules, the error system can obtain a good convergence rate. The tracking error is quite small compared with the original uncertain dual switching system.

multidisciplinary sciences
Dual fault‐tolerant control for a class of stochastic systems with partial loss‐of‐control effectiveness

Lei Liu,Guo Xie,Fucai Qian,Xiaohong Guo,Xuehui Ma

DOI: https://doi.org/10.1002/rnc.5857

IF: 3.8973

2021-10-27

International Journal of Robust and Nonlinear Control

Abstract:Loss‐of‐control effectiveness is a common fault in the engineering field. In the existing adaptive fault‐tolerant control algorithms, observers and Lyapunov functions are often used to estimate fault parameters. However, they can only ensure the convergence of fault parameters as time goes to infinity. This leads to the fact that the controller can only play a better fault‐tolerant role in the steady‐state process, and cannot guarantee the system performance in the transient state. Therefore, a dual fault‐tolerant control (DFTC) algorithm is proposed for a class of stochastic systems with partial loss‐of‐control effectiveness. The algorithm introduces a fault‐tolerant objective on the basis of the control objective. By establishing a new objective function, the controller is forced to have the characteristics of active learning to improve the transient performance of the fault system. An example simulation verifies the validity of DFTC. The simulation results show that DFTC can drive the system to the desired output, especially when partial loss‐of‐control efficiency occurs in the system, it can learn the unknown parameters actively, quickly, and accurately, which is significantly better than the conventional control based on the certainty equivalence principle and improves the reliability of the system.

automation & control systems,engineering, electrical & electronic,mathematics, applied
Adaptive Robust Control of Nonholonomic Systems with Stochastic Disturbances

Wang Jiang,Gao Hanqiao,Li Huiyan

DOI: https://doi.org/10.1007/s11432-006-0189-5

2006-01-01

Science in China Series F Information Sciences

Abstract:This paper deals with nonholonomic systems in chained form with unknown covariance stochastic disturbances. The objective is to design the almost global adaptive asymptotical controllers in probability u 0 and u 1 for the systems by using discontinuous control. A switching control law u 0 is designed to almost globally asymptotically stabilize the state x 0 in both the singular x 0(t 0)=0 case and the non-singular x 0(t 0)≠0 case. Then the state scaling technique is introduced for the discontinuous feedback into the (x 1, x2, ⋯, x n )-subsystem. Thereby, by using backstepping technique the global adaptive asymptotical control law u 1 has been presented for (x 1, x2, ⋯, x n )-subsystem for both different u 0 in non-singular x 0(t 0)≠0 case and the singular case x 0(t 0)=0. The control algorithm validity is proved by simulation.
Adaptive Neural Control for Stochastic Nonholonomic Systems with Full-State Constraints and Unknown Covariance Noise

Qinghui Du,Lixia Wang,Chaofeng Zhao

DOI: https://doi.org/10.1080/00036811.2021.2011242

2021-01-01

Applicable Analysis

Abstract:In this paper, adaptive neural control is investigated for a class of stochastic nonholonomic systems disturbed by unknown covariance noise under the condition of full-state constraints. Compared with the related literatures, this paper largely generalizes the results in recent works on stochastic nonholonomic systems. The distinctive features of this paper are three folds: the full-state constrains are introduced; the restriction assumed on incremental covariance is removed; the growth assumptions imposed on drift and diffusion terms are somewhat weakened. In the control design procedure, the barrier Lyapunov function (BLF) is applied to conquer the effect of full-state constraints to system performance, unknown covariance noise is compensated with the aid of adaptive control design, the radial basis function neural networks are utilized to approximate unknown nonlinear functions and the backstepping technique is used to construct the desired controller. Then, by adopting the switching strategy to eliminate the phenomenon of uncontrollability, the proposed adaptive neural controller renders the closed-loop system to be semi-globally uniformly ultimately bounded. Moreover, the system states remain in the defined compact sets and the output tracks the reference signal well. Finally, the simulation example shows the effectiveness of the proposed scheme.
Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control

Yuheng Wang,Margaret P. Chapman

DOI: https://doi.org/10.48550/arXiv.2109.08947

2022-07-12

Abstract:We present an historical overview about the connections between the analysis of risk and the control of autonomous systems. We offer two main contributions. Our first contribution is to propose three overlapping paradigms to classify the vast body of literature: the worst-case, risk-neutral, and risk-averse paradigms. We consider an appropriate assessment for the risk of an autonomous system to depend on the application at hand. In contrast, it is typical to assess risk using an expectation, variance, or probability alone. Our second contribution is to unify the concepts of risk and autonomous systems. We achieve this by connecting approaches for quantifying and optimizing the risk that arises from a system's behaviour across academic fields. The survey is highly multidisciplinary. We include research from the communities of reinforcement learning, stochastic and robust control theory, operations research, and formal verification. We describe both model-based and model-free methods, with emphasis on the former. Lastly, we highlight fruitful areas for further research. A key direction is to blend risk-averse model-based and model-free methods to enhance the real-time adaptive capabilities of systems to improve human and environmental welfare.

Artificial Intelligence
Data-Driven Adaptive Optimal Control for Linear Systems with Structured Time-Varying Uncertainty.

Meng Zhang,Ming-Gang Gan

DOI: https://doi.org/10.1109/access.2019.2891575

IF: 3.9

2019-01-01

IEEE Access

Abstract:In this paper, a data-driven adaptive optimal control strategy is proposed for a class of linear systems with structured time-varying uncertainty, minimizing the upper bound of a pre-defined cost function while maintaining the closed-loop stability. An off-policy data-driven reinforcement learning algorithm is presented, which uses repeatedly the online state signal on some fixed time intervals without knowing system information, yielding a guaranteed cost control (GCC) law with quadratic stability for the system. This law is further optimized through a particle swarm optimization (PSO) method, the parameters of which are adaptively adjusted by a fuzzy logic mechanism, and an optimal GCC law with the minimum upper bound of the cost function is finally obtained. The effectiveness of this strategy is verified on the dynamic model of a two-degree-of-freedom helicopter, showing that both stability and convergence of the closed-loop system are guaranteed and that the cost is minimized with much less iteration than the conventional PSO method with constant parameters.
A survey of adaptive optimal control theory

Xiaoxuan Pei,Kewen Li,Yongming Li

DOI: https://doi.org/10.3934/mbe.2022561

2022-01-01

Mathematical Biosciences and Engineering

Abstract:This paper makes a survey about the recent development of optimal control based on adaptive dynamic programming (ADP). First of all, based on DP algorithm and reinforcement learning (RL) algorithm, the origin and development of the optimization idea and its application in the control field are introduced. The second part introduces achievements in the optimal control direction, then we classify and summarize the research results of optimization method, constraint problem, structure design in control algorithm and practical engineering process based on optimal control. Finally, the possible future research topics are discussed. Through a comprehensive and complete investigation of its application in many existing fields, this survey fully demonstrates that the optimal control algorithms via ADP with critic-actor neural network (NN) structure, which also have a broad application prospect, and some developed optimal control design algorithms have been applied to practical engineering fields.
An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control

Elena Arcari,Lukas Hewing,Melanie N. Zeilinger

DOI: https://doi.org/10.1016/j.ifacol.2020.12.2280

2020-01-01

Abstract:Dual control explicitly addresses the problem of trading off active exploration and exploitation in the optimal control of partially unknown systems. While the problem can be cast in the framework of stochastic dynamic programming, exact solutions are only tractable for discrete state and action spaces of very small dimension due to a series of nested minimization and expectation operations. We propose an approximate dual control method for systems with continuous state and input domain based on a rollout dynamic programming approach, splitting the control horizon into a dual and an exploitation part. The dual part is approximated using a scenario tree generated by sampling the process noise and the unknown system parameters, for which the underlying distribution is updated via Bayesian estimation along the horizon. In the exploitation part, we fix the resulting parameter estimate of each scenario branch and compute an open-loop control sequence for the remainder of the horizon. The key benefit of the proposed sampling-based approximation is that it enables the formulation as one optimization problem that computes a collection of control sequences over the scenario tree, leading to a dual model predictive control formulation.

Dual Adaptive Control

An Alternative Dual Control Approach for Auto-Optimization in Uncertain Environments

Adaptive dual control with online outlier detection for uncertain systems

Adaptive Control and Intersections with Reinforcement Learning

Dual Control of Exploration and Exploitation for Auto-Optimization Control with Active Learning

Rolling Horizon Stochastic Optimal Control Strategy for ACC and CACC under Uncertainty

Dual Control of Exploration and Exploitation for Auto-Optimisation Control with Active Learning

Duality Between Large Deviation Control and Risk-Sensitive Control for Markov Decision Processes.

Universal Adaptive Control of Nonlinear Systems

Composite Adaptive Control for Time-varying Systems with Dual Adaptation

Active Uncertainty Reduction for Human-Robot Interaction: An Implicit Dual Control Approach

Partially Observable Stochastic Optimal Control

A Dual Adaptive Control Theory Inspired by Hebbian Associative Learning

Adaptive neural network control for uncertain dual switching nonlinear systems

Dual fault‐tolerant control for a class of stochastic systems with partial loss‐of‐control effectiveness

Adaptive Robust Control of Nonholonomic Systems with Stochastic Disturbances

Adaptive Neural Control for Stochastic Nonholonomic Systems with Full-State Constraints and Unknown Covariance Noise

Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control

Data-Driven Adaptive Optimal Control for Linear Systems with Structured Time-Varying Uncertainty.

A survey of adaptive optimal control theory

An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control