Abstract:Dual control denotes a class of control problems where the parameters governing the system are imperfectly known. The challenge is to find the optimal balance between probing, i.e. exciting the system to understand it more, and caution, i.e. selecting conservative controls based on current knowledge to achieve the control objective. Dynamic programming techniques can achieve this optimal trade-off. However, while dynamic programming performs well with discrete state and time, it is not well-suited to problems with continuous time-frames or continuous or unbounded state spaces. Another limitation is that multidimensional states often cause the dynamic programming approaches to be intractable. In this paper, we investigate whether continuous-time optimal control tools could help circumvent these caveats whilst still achieving the probing-caution balance. We introduce a stylized problem where the state is governed by one of two differential equations. It is initially unknown which differential equation governs the system, so we must simultaneously determine the true differential equation and control the system to the desired state. We show how this problem can be transformed to apply optimal control tools, and compare the performance of this approach to a dynamic programming approach. Our results suggest that the optimal control algorithm rivals dynamic programming on small problems, achieving the right balance between aggressive and smoothly varying controls. In contrast to dynamic programming, the optimal control approach remains tractable when several states are to be controlled simultaneously.

Deterministic control of randomly-terminated processes

Optimal control of production-dependent failure-prone manufacturing systems

The existence of optimal control for continuous-time Markov decision processes in random environments

Drift Control with Discretionary Stopping for a Diffusion Process

Stochastic control up to a hitting time: optimality and rolling-horizon implementation

Asymptotic Stability with Probability One of Random-Time-Delay-Controlled Quasi-Integrable Hamiltonian Systems

Stochastic Intermittent Control with Uncertainty

Average Continuous Control of Piecewise Deterministic Markov Processes

Deterministic Trajectory Optimization through Probabilistic Optimal Control

Existence and Synchronization of Coupled Stochastic Infinite-Dimensional Systems Via Aperiodically Intermittent Control.

Mixed Deterministic and Random Optimal Control of Linear Stochastic Systems with Quadratic Costs

Finite-time horizon, stopper vs. singular-controller games on the half-line

A continuous time formulation of stochastic dual control to avoid the curse of dimensionality

Semi-Infinite Programs for Robust Control and Optimization: Efficient Solutions and Extensions to Existence Constraints

On Stochastic Model Predictive Control with Bounded Control Inputs

Optimal control formulation of transition path problems for Markov Jump Processes

Continuous time Stochastic optimal control under discrete time partial observations

Stopping Time and Control for a Type of Impulsive Stochastic Differential Equation

On Probabilistic Termination of Functional Programs with Continuous Distributions

Control in Probability for SDE Models of Growth Population

Stochastic Near-Optimal Controls for Path-Dependent Systems