Abstract:This paper presents an online method that learns optimal decisions for a discrete time Markov decision problem with an opportunistic structure. The state at time $t$ is a pair $(S(t),W(t))$ where $S(t)$ takes values in a finite set $\mathcal{S}$ of basic states, and $\{W(t)\}_{t=0}^{\infty}$ is an i.i.d. sequence of random vectors that affect the system and that have an unknown distribution. Every slot $t$ the controller observes $(S(t),W(t))$ and chooses a control action $A(t)$. The triplet $(S(t),W(t),A(t))$ determines a vector of costs and the transition probabilities for the next state $S(t+1)$. The goal is to minimize the time average of an objective function subject to additional time average cost constraints. We develop an algorithm that acts on a corresponding virtual system where $S(t)$ is replaced by a decision variable. An equivalence between virtual and actual systems is established by enforcing a collection of time averaged global balance equations. For any desired $\epsilon>0$, we prove the algorithm achieves an $\epsilon$-optimal solution on the virtual system with a convergence time of $O(1/\epsilon^2)$. The actual system runs at the same time, its actions are informed by the virtual system, and its conditional transition probabilities and costs are proven to be the same as the virtual system at every instant of time. Also, its unconditional probabilities and costs are shown in simulation to closely match the virtual system. Our simulations consider online control of a robot that explores a region of interest. Objects with varying rewards appear and disappear and the robot learns what areas to explore and what objects to collect and deliver to a home base.

What problem does this paper attempt to address?

The paper mainly addresses the following issues: ### Research Background and Objectives - **Research Subject**: The paper studies a class of discrete-time Markov Decision Problems (MDP), which includes a basic state set $S$ and a sequence of random vectors $\{W(t)\}_{t=0}^{\infty}$ that influence the system's evolution but whose distribution is unknown. - **Objective**: The goal of the paper is to minimize the time average of an objective function while satisfying additional cost constraint conditions on the time average. Specifically, at each time step $t$, the system needs to select a control action $A(t)$ based on the current basic state $S(t)$ and the random vector $W(t)$. ### Main Contributions - **Virtual System Method**: A new online learning algorithm is proposed, which operates in a virtual system where decision variables replace the actual system's state variables $S(t)$. A set of global balance equations is enforced to ensure the equivalence between the virtual system and the actual system. - **Algorithm Performance**: For any positive number $\epsilon > 0$, it is proven that the algorithm can achieve an $\epsilon$-optimal solution in the virtual system with a convergence time of $O(1/\epsilon^2)$. The actual system runs synchronously with the virtual system, and the transition probabilities and costs of the actual system are shown to be the same as those of the virtual system. - **Robot Example**: The effectiveness of the algorithm is validated through a simulation of an intelligent robot exploring areas of interest. The robot needs to learn which areas are worth exploring and which items should be collected and brought back to the base. Experimental results show that the algorithm can effectively learn the optimal strategy, achieving or approaching the optimal time-average reward. ### Methodology and Theoretical Foundation - **Algorithm Design**: The algorithm design is based on Lyapunov drift techniques, particularly the extended drift-plus-penalty method, to minimize the time-average cost while maintaining the stability of certain virtual queues. - **Optimization Problem**: The algorithm's performance is described by a deterministic non-convex optimization problem, which isolates the basic state variables into a finite set $S$ and uses compact and convex sets $\Gamma_i$ to handle opportunistic scheduling issues. - **Theoretical Guarantees**: Through theoretical analysis, the paper proves the equivalence between the virtual system and the actual system and provides explicit guarantees on the algorithm's performance. In summary, this paper addresses the learning and optimization problem of a class of discrete-time Markov decision problems with opportunistic structures. By designing a new online learning algorithm, it achieves efficient learning of unknown distributions and validates the algorithm's effectiveness and superiority through theoretical analysis and experimental verification.

Opportunistic Learning for Markov Decision Systems with Application to Smart Robots

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Learning to Switch Among Agents in a Team via 2-Layer Markov Decision Processes

Learning Anticipatory Decision for Distributed Systems with Robustness Guarantees

Decision-Oriented Learning Using Differentiable Submodular Maximization for Multi-Robot Coordination

Optimal Learning for Sequential Decision Making for Expensive Cost Functions with Stochastic Binary Feedbacks

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Approximate Policy Iteration for Robust Stochastic Control of Multi-agent Markov Decision Processes

Optimal decision making in robotic assembly and other trial-and-error tasks

Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes

Online Reinforcement Learning in Markov Decision Process Using Linear Programming

Online Optimization and Ambiguity-Based Learning of Distributionally Uncertain Dynamic Systems

Online Pareto-Optimal Decision-Making for Complex Tasks using Active Inference

Reinforcement Learning and Adaptive Optimization of a Class of Markov Jump Systems with Completely Unknown Dynamic Information

Online Markov decision processes with Kullback-Leibler control cost

A learning-based approach to stochastic optimal control under reach-avoid constraint

Acting in Delayed Environments with Non-Stationary Markov Policies

Model-Based Reinforcement Learning In Continuous Environments Using Real-Time Constrained Optimization

A Structure-aware Online Learning Algorithm for Markov Decision Processes

Stochastic Online Optimization for Cyber-Physical and Robotic Systems