A posterior sampling framework for interactive decision making

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, Tong Zhang

2022-01-01

Abstract:

What problem does this paper attempt to address?

Asymptotic Instance-Optimal Algorithms for Interactive Decision Making

Kefan Dong,Tengyu Ma

2023-06-12

Abstract:Past research on interactive decision making problems (bandits, reinforcement learning, etc.) mostly focuses on the minimax regret that measures the algorithm's performance on the hardest instance. However, an ideal algorithm should adapt to the complexity of a particular problem instance and incur smaller regrets on easy instances than worst-case instances. In this paper, we design the first asymptotic instance-optimal algorithm for general interactive decision making problems with finite number of decisions under mild conditions. On every instance $f$, our algorithm outperforms all consistent algorithms (those achieving non-trivial regrets on all instances), and has asymptotic regret $\mathcal{C}(f) \ln n$, where $\mathcal{C}(f)$ is an exact characterization of the complexity of $f$. The key step of the algorithm involves hypothesis testing with active data collection. It computes the most economical decisions with which the algorithm collects observations to test whether an estimated instance is indeed correct; thus, the complexity $\mathcal{C}(f)$ is the minimum cost to test the instance $f$ against other instances. Our results, instantiated on concrete problems, recover the classical gap-dependent bounds for multi-armed bandits [Lai and Robbins, 1985] and prior works on linear bandits [Lattimore and Szepesvari, 2017], and improve upon the previous best instance-dependent upper bound [Xu et al., 2021] for reinforcement learning.

Machine Learning
Efficient Sampling Policy for Selecting a Subset with the Best

Gongbo Zhang,Bin Chen,Qing-Shan Jia,Yijie Peng

DOI: https://doi.org/10.1109/tac.2022.3207871

IF: 6.549

2022-01-01

IEEE Transactions on Automatic Control

Abstract:The note studies the problem of selecting a subset with the best of a finite number of alternatives under a fixed simulation budget. Our work aims to maximize the posterior probability of correctly selecting such a subset. We formulate the dynamic sampling decision as a stochastic control problem in a Bayesian setting. In an approximate dynamic programming paradigm, we propose a sequential sampling policy based on value function approximation. We analyze the asymptotic property of the proposed sampling policy. Numerical experiments demonstrate the efficiency of the proposed procedure.

automation & control systems,engineering, electrical & electronic
Efficient Sampling Policy for Selecting a Good Enough Subset

Gongbo Zhang,Bin Chen,Qing-shan Jia,Yijie Peng

DOI: https://doi.org/10.1109/TAC.2022.3207871

2021-11-29

Abstract:The note studies the problem of selecting a good enough subset out of a finite number of alternatives under a fixed simulation budget. Our work aims to maximize the posterior probability of correctly selecting a good subset. We formulate the dynamic sampling decision as a stochastic control problem in a Bayesian setting. In an approximate dynamic programming paradigm, we propose a sequential sampling policy based on value function approximation. We analyze the asymptotic property of the proposed sampling policy. Numerical experiments demonstrate the efficiency of the proposed procedure.

Optimization and Control
Offline Simulation Online Application: A New Framework of Simulation-Based Decision Making

L. Jeff Hong,Guangxin Jiang

DOI: https://doi.org/10.1142/S0217595919400153

2019-01-01

Asia-Pacific Journal of Operational Research

Abstract:Traditionally, simulation has been used as a tool of design to estimate, compare and optimize the performance of different system designs. It is rarely used in making real-time decisions due to the long computation delay of executing simulation models. However, with the fast growth of computing capability, we have observed more and more works on reusing simulation efforts for repeated experiments with the help of data analytics tools, and the target of these works is to solve real-time decision problems. In this paper, we distill the important features of these works and summarize a new simulation framework, called offline-simulation-online-application (OSOA) framework, which treats simulation as a data generator, applies state-of-the-art analytics tools to build predictive models, and then uses the predictive models for real-time applications. In this paper, we illustrate how to apply the OSOA framework on estimation, ranking and selection and simulation optimization, and provide a prospect of this new framework.
Assouad, Fano, and Le Cam with Interaction: A Unifying Lower Bound Framework and Characterization for Bandit Learnability

Fan Chen,Dylan J. Foster,Yanjun Han,Jian Qian,Alexander Rakhlin,Yunbei Xu

2024-10-07

Abstract:In this paper, we develop a unified framework for lower bound methods in statistical estimation and interactive decision making. Classical lower bound techniques -- such as Fano's inequality, Le Cam's method, and Assouad's lemma -- have been central to the study of minimax risk in statistical estimation, yet they are insufficient for the analysis of methods that collect data in an interactive manner. The recent minimax lower bounds for interactive decision making via the Decision-Estimation Coefficient (DEC) appear to be genuinely different from the classical methods. We propose a unified view of these distinct methodologies through a general algorithmic lower bound method. We further introduce a novel complexity measure, decision dimension, which facilitates the derivation of new lower bounds for interactive decision making. In particular, decision dimension provides a characterization of bandit learnability for any structured bandit model class. Further, we characterize the sample complexity of learning convex model class up to a polynomial gap with the decision dimension, addressing the remaining gap between upper and lower bounds in Foster et al. (2021, 2023).

Machine Learning,Information Theory,Statistics Theory
GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond

Han Zhong,Wei Xiong,Sirui Zheng,Liwei Wang,Zhaoran Wang,Zhuoran Yang,Tong Zhang

2023-06-30

Abstract:We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making, which includes Markov decision process (MDP), partially observable Markov decision process (POMDP), and predictive state representation (PSR) as special cases. Toward finding the minimum assumption that empowers sample efficient learning, we propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation in online interactive decision making. In specific, GEC captures the hardness of exploration by comparing the error of predicting the performance of the updated policy with the in-sample training error evaluated on the historical data. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR, where generalized regular PSR, a new tractable PSR class identified by us, includes nearly all known tractable POMDPs and PSRs. Furthermore, in terms of algorithm design, we propose a generic posterior sampling algorithm, which can be implemented in both model-free and model-based fashion, under both fully observable and partially observable settings. The proposed algorithm modifies the standard posterior sampling algorithm in two aspects: (i) we use an optimistic prior distribution that biases towards hypotheses with higher values and (ii) a loglikelihood function is set to be the empirical loss evaluated on the historical data, where the choice of loss function supports both model-free and model-based learning. We prove that the proposed algorithm is sample efficient by establishing a sublinear regret upper bound in terms of GEC. In summary, we provide a new and unified understanding of both fully observable and partially observable RL.

Machine Learning,Artificial Intelligence,Optimization and Control
Dynamic Sampling Allocation and Design Selection.

Yijie Peng,Chun-Hung Chen,Michael C. Fu,Jian-Qiang Hu

DOI: https://doi.org/10.1287/ijoc.2015.0673

IF: 3.288

2016-01-01

INFORMS Journal on Computing

Abstract:We formulate the statistical selection problem in a general dynamic framework comprising fully sequential sampling allocation and optimal design selection. Because the traditional probability of correct selection measure is not sufficient to capture both aspects in this more general framework, we introduce the integrated probability of correct selection to better characterize the objective. As a result, the usual selection policy of choosing the design with the largest sample mean as the estimate of the best is no longer necessarily optimal. Rather, the optimal selection policy is to choose the design that maximizes the posterior integrated probability of correct selection, which is a function of the posterior mean and the correlation structure induced by the posterior variance. Because determining the optimal selection policy is generally intractable, we also devise an approximation scheme to efficiently approximate the optimal selection policy. For the allocation policy, we study an asymptotic policy called general Bayesian budget allocation, which is comprised of a sampling statistic and a sequential rule. The optimal computing budget allocation algorithm can be interpreted as a special case of the asymptotical sampling statistics. Numerical examples are provided to illustrate the potential performance improvements, especially in small sample behavior.
Posterior Sampling for Multi-Agent Reinforcement Learning: Solving Extensive Games with Imperfect Information

Yichi Zhou,Jialian Li,Jun Zhu

2020-01-01

Abstract:Posterior sampling for reinforcement learning (PSRL) is a useful framework for making decisions in an unknown environment. PSRL maintains a posterior distribution of the environment and then makes planning on the environment sampled from the posterior distribution. Though PSRL works well on single-agent reinforcement learning problems, how to apply PSRL to multi-agent reinforcement learning problems is relatively unexplored. In this work, we extend PSRL to two-player zero-sum extensive-games with imperfect information (TZIEG), which is a class of multi-agent systems. More specifically, we combine PSRL with counterfactual regret minimization (CFR), which is the leading algorithm for TZIEG with a known environment. Our main contribution is a novel design of interaction strategies. With our interaction strategies, our algorithm provably converges to the Nash Equilibrium at a rate of $O(\sqrt{\log T/T})$. Empirical results show that our algorithm works well.
Cost-Efficient Online Decision Making: A Combinatorial Multi-Armed Bandit Approach

Arman Rahbar,Niklas Åkerblom,Morteza Haghir Chehreghani

2024-02-05

Abstract:Online decision making plays a crucial role in numerous real-world applications. In many scenarios, the decision is made based on performing a sequence of tests on the incoming data points. However, performing all tests can be expensive and is not always possible. In this paper, we provide a novel formulation of the online decision making problem based on combinatorial multi-armed bandits and take the (possibly stochastic) cost of performing tests into account. Based on this formulation, we provide a new framework for cost-efficient online decision making which can utilize posterior sampling or BayesUCB for exploration. We provide a theoretical analysis of Thompson Sampling for cost-efficient online decision making, and present various experimental results that demonstrate the applicability of our framework to real-world problems.

Machine Learning
A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Games

Wei Xiong,Han Zhong,Chengshuai Shi,Cong Shen,Tong Zhang

DOI: https://doi.org/10.48550/arXiv.2210.01907

IF: 5.414

2022-10-04

Machine Learning

Abstract:Existing studies on provably efficient algorithms for Markov games (MGs) almost exclusively build on the "optimism in the face of uncertainty" (OFU) principle. This work focuses on a different approach of posterior sampling, which is celebrated in many bandits and reinforcement learning settings but remains under-explored for MGs. Specifically, for episodic two-player zero-sum MGs, a novel posterior sampling algorithm is developed with general function approximation. Theoretical analysis demonstrates that the posterior sampling algorithm admits a $\sqrt{T}$-regret bound for problems with a low multi-agent decoupling coefficient, which is a new complexity measure for MGs, where $T$ denotes the number of episodes. When specialized to linear MGs, the obtained regret bound matches the state-of-the-art results. To the best of our knowledge, this is the first provably efficient posterior sampling algorithm for MGs with frequentist regret guarantees, which enriches the toolbox for MGs and promotes the broad applicability of posterior sampling.
Stochastic Control Framework for Determining Feasible Alternatives in Sampling Allocation

Yijie Peng,Jie Song,Jie Xu,Edwin K. P. Chong

DOI: https://doi.org/10.1109/tac.2019.2942005

IF: 6.549

2020-01-01

IEEE Transactions on Automatic Control

Abstract:We formulate the optimal dynamic sampling allocation decision problem for feasibility determination as a stochastic control problem in a Bayesian setting. This new formulation addresses the limitations of previous static optimization formulations. In an approximate dynamic programming paradigm, we propose an approximately optimal allocation policy that maximizes a single feature of the value function one step ahead. Numerical results demonstrate the efficiency of the proposed method.
Collective Sampling: An Ex Ante Perspective

Yangfan Zhou

DOI: https://doi.org/10.48550/arXiv.2311.05758

2024-06-24

Abstract:I study collective dynamic information acquisition. Players determine when to end sequential sampling via a collective stopping rule. With two players, collective stopping is determined either unilaterally or unanimously. I develop a methodology to characterize equilibria using an ex ante perspective on posterior distributions. Under unilateral stopping, each player chooses a mean-preserving contraction of the other's posterior distribution; under unanimous stopping, they choose mean-preserving spreads. Equilibrium sampling regions are determined via concavification. Preference misalignment generates Pareto inefficiency: there is too little learning under unilateral stopping and possibly too much under unanimous stopping. Learning inefficiencies are amplified when players' preferences become less aligned. I demonstrate the value of my methodology in three applications: committee search, dynamic persuasion, and competition in persuasion.

Theoretical Economics
Markov-chain Monte-Carlo Sampling for Optimal Fidelity Determination in Dynamic Decision-Making

Sara Masoud,Bijoy Chowdhury,Young-Jun Son,Russell Tronstad

DOI: https://doi.org/10.48550/arXiv.1911.03719

2019-11-09

Applications

Abstract:Decision making for dynamic systems is challenging due to the scale and dynamicity of such systems, and it is comprised of decisions at strategic, tactical, and operational levels. One of the most important aspects of decision making is incorporating real time information that reflects immediate status of the system. This type of decision making, which may apply to any dynamic system, needs to comply with the system's current capabilities and calls for a dynamic data driven planning framework. Performance of dynamic data driven planning frameworks relies on the decision making process which in return is relevant to the quality of the available data. This means that the planning framework should be able to set the level of decision making based on the current status of the system, which is learned through the continuous readings of sensory data. In this work, a Markov chain Monte Carlo sampling method is proposed to determine the optimal fidelity of decision making in a dynamic data driven framework. To evaluate the performance of the proposed method, an experiment is conducted, where the impact of workers performance on the production capacity and the fidelity level of decision making are studied.
A Reduction-based Framework for Sequential Decision Making with Delayed Feedback

Yunchang Yang,Han Zhong,Tianhao Wu,Bin Liu,Liwei Wang,Simon S. Du

2024-03-06

Abstract:We study stochastic delayed feedback in general multi-agent sequential decision making, which includes bandits, single-agent Markov decision processes (MDPs), and Markov games (MGs). We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision making. By plugging different multi-batched algorithms into our framework, we provide several examples demonstrating that our framework not only matches or improves existing results for bandits, tabular MDPs, and tabular MGs, but also provides the first line of studies on delays in sequential decision making with function approximation. In summary, we provide a complete set of sharp results for multi-agent sequential decision making with delayed feedback.

Machine Learning
Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling

Danil Provodin,Maurits Kaptein,Mykola Pechenizkiy

2024-05-29

Abstract:We present a new algorithm based on posterior sampling for learning in Constrained Markov Decision Processes (CMDP) in the infinite-horizon undiscounted setting. The algorithm achieves near-optimal regret bounds while being advantageous empirically compared to the existing algorithms. Our main theoretical result is a Bayesian regret bound for each cost component of $\tilde{O} (DS\sqrt{AT})$ for any communicating CMDP with $S$ states, $A$ actions, and diameter $D$. This regret bound matches the lower bound in order of time horizon $T$ and is the best-known regret bound for communicating CMDPs achieved by a computationally tractable algorithm. Empirical results show that our posterior sampling algorithm outperforms the existing algorithms for constrained reinforcement learning.

Machine Learning
Dynamic Sampling Allocation for Selecting a Good Enough Alternative.

Gongbo Zhang,Chun-Hung Chen,Qing-Shan Jia,Yijie Peng

DOI: https://doi.org/10.1109/case48305.2020.9216838

2020-01-01

Abstract:We consider the problem of selecting a good enough alternative from a finite set of alternatives. Instead of selecting the exactly best alternative, our work aims to maximize the probability of correctly selecting an alternative in an acceptable subset. Under a Bayesian framework, we formulate the problem as a stochastic control problem. We propose a dynamic allocation scheme for selecting a good enough alternative, which optimizes a value function approximation one-step ahead. Numerical results demonstrate the proposed sampling procedure is more efficient than other sampling allocation methods.
Sampling to Achieve the Goal: An Age-aware Remote Markov Decision Process

Aimin Li,Shaohua Wu,Gary C.F. Lee,Xiaomeng Cheng,Sumei Sun

2024-08-08

Abstract:Age of Information (AoI) has been recognized as an important metric to measure the freshness of information. Central to this consensus is that minimizing AoI can enhance the freshness of information, thereby facilitating the accuracy of subsequent decision-making processes. However, to date the direct causal relationship that links AoI to the utility of the decision-making process is unexplored. To fill this gap, this paper provides a sampling-control co-design problem, referred to as an age-aware remote Markov Decision Process (MDP) problem, to explore this unexplored relationship. Our framework revisits the sampling problem in [1] with a refined focus: moving from AoI penalty minimization to directly optimizing goal-oriented remote decision-making process under random delay. We derive that the age-aware remote MDP problem can be reduced to a standard MDP problem without delays, and reveal that treating AoI solely as a metric for optimization is not optimal in achieving remote decision making. Instead, AoI can serve as important side information to facilitate remote decision making.

Information Theory
Amortized Bayesian Decision Making for simulation-based models

Mila Gorecki,Jakob H. Macke,Michael Deistler

2023-12-18

Abstract:Simulation-based inference (SBI) provides a powerful framework for inferring posterior distributions of stochastic simulators in a wide range of domains. In many settings, however, the posterior distribution is not the end goal itself -- rather, the derived parameter values and their uncertainties are used as a basis for deciding what actions to take. Unfortunately, because posterior distributions provided by SBI are (potentially crude) approximations of the true posterior, the resulting decisions can be suboptimal. Here, we address the question of how to perform Bayesian decision making on stochastic simulators, and how one can circumvent the need to compute an explicit approximation to the posterior. Our method trains a neural network on simulated data and can predict the expected cost given any data and action, and can, thus, be directly used to infer the action with lowest cost. We apply our method to several benchmark problems and demonstrate that it induces similar cost as the true posterior distribution. We then apply the method to infer optimal actions in a real-world simulator in the medical neurosciences, the Bayesian Virtual Epileptic Patient, and demonstrate that it allows to infer actions associated with low cost after few simulations.

Machine Learning,Artificial Intelligence
AutoSAM: Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems

Hao Zhang,Mingyue Cheng,Qi Liu,Zhiding Liu,Junzhe Jiang,Enhong Chen

2024-07-16

Abstract:Sequential recommender systems (SRS) have gained widespread popularity in recommendation due to their ability to effectively capture dynamic user preferences. One default setting in the current SRS is to uniformly consider each historical behavior as a positive interaction. Actually, this setting has the potential to yield sub-optimal performance, as each item makes a distinct contribution to the user's interest. For example, purchased items should be given more importance than clicked ones. Hence, we propose a general automatic sampling framework, named AutoSAM, to non-uniformly treat historical behaviors. Specifically, AutoSAM augments the standard sequential recommendation architecture with an additional sampler layer to adaptively learn the skew distribution of the raw input, and then sample informative sub-sets to build more generalizable SRS. To overcome the challenges of non-differentiable sampling actions and also introduce multiple decision factors for sampling, we further introduce a novel reinforcement learning based method to guide the training of the sampler. We theoretically design multi-objective sampling rewards including Future Prediction and Sequence Perplexity, and then optimize the whole framework in an end-to-end manner by combining the policy gradient. We conduct extensive experiments on benchmark recommender models and four real-world datasets. The experimental results demonstrate the effectiveness of the proposed approach. We will make our code publicly available after the acceptance.

Information Retrieval
Dynamic Game-Theoretical Decision-Making Framework for Vehicle-Pedestrian Interaction with Human Bounded Rationality

Meiting Dang,Dezong Zhao,Yafei Wang,Chongfeng Wei

2024-09-24

Abstract:Human-involved interactive environments pose significant challenges for autonomous vehicle decision-making processes due to the complexity and uncertainty of human behavior. It is crucial to develop an explainable and trustworthy decision-making system for autonomous vehicles interacting with pedestrians. Previous studies often used traditional game theory to describe interactions for its interpretability. However, it assumes complete human rationality and unlimited reasoning abilities, which is unrealistic. To solve this limitation and improve model accuracy, this paper proposes a novel framework that integrates the partially observable markov decision process with behavioral game theory to dynamically model AV-pedestrian interactions at the unsignalized intersection. Both the AV and the pedestrian are modeled as dynamic-belief-induced quantal cognitive hierarchy (DB-QCH) models, considering human reasoning limitations and bounded rationality in the decision-making process. In addition, a dynamic belief updating mechanism allows the AV to update its understanding of the opponent's rationality degree in real-time based on observed behaviors and adapt its strategies accordingly. The analysis results indicate that our models effectively simulate vehicle-pedestrian interactions and our proposed AV decision-making approach performs well in safety, efficiency, and smoothness. It closely resembles real-world driving behavior and even achieves more comfortable driving navigation compared to our previous virtual reality experimental data.

Robotics

A posterior sampling framework for interactive decision making

Asymptotic Instance-Optimal Algorithms for Interactive Decision Making

Efficient Sampling Policy for Selecting a Subset with the Best

Efficient Sampling Policy for Selecting a Good Enough Subset

Offline Simulation Online Application: A New Framework of Simulation-Based Decision Making

Assouad, Fano, and Le Cam with Interaction: A Unifying Lower Bound Framework and Characterization for Bandit Learnability

GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond

Dynamic Sampling Allocation and Design Selection.

Posterior Sampling for Multi-Agent Reinforcement Learning: Solving Extensive Games with Imperfect Information

Cost-Efficient Online Decision Making: A Combinatorial Multi-Armed Bandit Approach

A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Games

Stochastic Control Framework for Determining Feasible Alternatives in Sampling Allocation

Collective Sampling: An Ex Ante Perspective

Markov-chain Monte-Carlo Sampling for Optimal Fidelity Determination in Dynamic Decision-Making

A Reduction-based Framework for Sequential Decision Making with Delayed Feedback

Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling

Dynamic Sampling Allocation for Selecting a Good Enough Alternative.

Sampling to Achieve the Goal: An Age-aware Remote Markov Decision Process

Amortized Bayesian Decision Making for simulation-based models

AutoSAM: Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems

Dynamic Game-Theoretical Decision-Making Framework for Vehicle-Pedestrian Interaction with Human Bounded Rationality