Abstract:We study a general Markov game with metric switching costs: in each round, the player adaptively chooses one of several Markov chains to advance with the objective of minimizing the expected cost for at least k chains to reach their target states. If the player decides to play a di erent chain, an additional switching cost is incurred. e special case in which there is no switching cost was solved optimally by Dumitriu, Tetali and Winkler [DTW03] by a variant of the celebrated Gi ins Index for the classical multi-armed bandit (MAB) problem with Markovian rewards [Git74, Git79]. However, for multi-armed bandit (MAB) with nontrivial switching cost, even if the switching cost is a constant, the classic paper by Banks and Sundaram [BS94] showed that no index strategy can be optimal. 1 In this paper, we complement their result and show there is a simple index strategy that achieves a constant approximation factor if the switching cost is constant and k = 1. To the best of our knowledge, this is the rst index strategy that achieves a constant approximation factor for a general MAB variant with switching costs. For the general metric, we propose a more involved constant-factor approximation algorithm, via an nontrivial reduction to the stochastic k-TSP problem, in which a Markov chain is approximated by a random variable. Our analysis makes extensive use of various interesting properties of Gi ins index. ∗Institute for Interdisciplinary Information Sciences, TsinghuaUniversity. Email:lijian83@mail.tsinghua.edu.cn. †Paul G. Allen School of Computer Science & Engineering, University of Washington. Part of work was done while visiting Shanghai Qi Zhi Institute. Email: dgliu@cs.washington.edu. 1 eir proof is for the discounted version of MAB, but can be extended to our se ing. See Appendix D for the details. ar X iv :2 10 7. 05 82 2v 1 [ cs .D S] 1 3 Ju l 2 02 1

Switching Costs in Infinitely Repeated Games

Switching Costs in Frequently Repeated Games

Strategic Invasion in Markets with Switching Costs

Solving finite time horizon Dynkin games by optimal switching

Explicit solution to an optimal two-player switching game in infinite horizon

Dynamic programming for discrete-time finite horizon optimal switching problems with negative switching costs

Switching Games Of Stochastic Differential Systems

Bounded Computational Capacity Equilibrium

An Impulse-Regime Switching Game Model of Vertical Competition

Payoff Control in Repeated Games

On the estimation of the switching moment of utility functions in cooperative differential games

Bandits with Switching Costs: T^{2/3} Regret.

Markov Game with Switching Costs

Discrete-time risk-aware optimal switching with non-adapted costs

Switching Games of Backward Stochastic Differential Equations

Evolutionary Rotation in Switching Incentive Zero-Sum Games

Multi-token Markov Game with Switching Costs

Inertia In Strategy Switching Transforms The Strategy Evolution

The Price of History-Independent Strategies in Games with Inter-Temporal Externalities

A Revision Game of Experimentation on a Common Threshold.

Closed-loop Equilibria for Mean-Field Games in Randomly Switching Environments with General Discounting Costs