Abstract:This paper presents an online method that learns optimal decisions for a discrete time Markov decision problem with an opportunistic structure. The state at time $t$ is a pair $(S(t),W(t))$ where $S(t)$ takes values in a finite set $\mathcal{S}$ of basic states, and $\{W(t)\}_{t=0}^{\infty}$ is an i.i.d. sequence of random vectors that affect the system and that have an unknown distribution. Every slot $t$ the controller observes $(S(t),W(t))$ and chooses a control action $A(t)$. The triplet $(S(t),W(t),A(t))$ determines a vector of costs and the transition probabilities for the next state $S(t+1)$. The goal is to minimize the time average of an objective function subject to additional time average cost constraints. We develop an algorithm that acts on a corresponding virtual system where $S(t)$ is replaced by a decision variable. An equivalence between virtual and actual systems is established by enforcing a collection of time averaged global balance equations. For any desired $\epsilon>0$, we prove the algorithm achieves an $\epsilon$-optimal solution on the virtual system with a convergence time of $O(1/\epsilon^2)$. The actual system runs at the same time, its actions are informed by the virtual system, and its conditional transition probabilities and costs are proven to be the same as the virtual system at every instant of time. Also, its unconditional probabilities and costs are shown in simulation to closely match the virtual system. Our simulations consider online control of a robot that explores a region of interest. Objects with varying rewards appear and disappear and the robot learns what areas to explore and what objects to collect and deliver to a home base.

Optimal Learning under Robustness and Time-Consistency.

Optimal Stopping under Model Ambiguity: a Time-Consistent Equilibrium Approach

Randomized Optimal Stopping Problem in Continuous Time and Reinforcement Learning Algorithm

Time-Consistency of Optimal Investment under Smooth Ambiguity

Learning to Optimally Stop a Diffusion Process

Prolonged Learning and Hasty Stopping: The Wald Problem with Ambiguity

Cost-efficient payoffs under model ambiguity

Robust optimal investment and consumption strategies with portfolio constraints and stochastic environment

Opportunistic Learning for Markov Decision Systems with Application to Smart Robots

Robust Time-inconsistent Linear-Quadratic Stochastic Controls: A Stochastic Differential Game Approach

Learning Underspecified Models

Robust experimentation in the continuous time bandit problem

Optimal Control of Robust Team Stochastic Games

Robust Portfolio Choice and Indifference Valuation

Robustness of Stochastic Optimal Control to Approximate Diffusion Models under Several Cost Evaluation Criteria

Exploratory Optimal Stopping: A Singular Control Formulation

Robust optimal stopping with regime switching

Robust Stochastic Stability in Dynamic and Reactive Environments

Optimal Long-Term Contracting with Learning

Robust Reinforcement Learning with Wasserstein Constraint

Non-concave distributionally robust stochastic control in a discrete time finite horizon setting