Abstract:This paper studies decision-making in two-player scenarios where the type (e.g. adversary, neutral, or teammate) of the other agent (opponent) is uncertain to the decision-making agent (protagonist), which is an abstraction of security-domain applications. In these settings, the reward for the protagonist agent depends on the type of the opponent, but this is private information known only to the opponent itself, and thus hidden from the protagonist. In contrast, as is often the case, the type of the protagonist agent is assumed to be known to the opponent, and this information-asymmetry significantly complicates the protagonist's decision-making. In particular, to determine the best actions to take, the protagonist agent must infer the opponent type from the observations and agent modeling. To address this problem, this paper presents an opponent-type deduction module based on Bayes' rule. This inference module takes as input the imagined opponent's decision-making rule (opponent model) as well as the observed opponent's history of actions and states, and outputs a belief over the opponent's hidden type. A multiagent reinforcement learning approach is used to develop this game-theoretic opponent model through self-play, which avoids the expensive data collection step that requires interaction with a real opponent. Besides, this multiagent approach also captures the strategy interaction and reasoning between agents. In addition, we apply ensemble training to avoid over-fitting to a single opponent model during the training. As a result, the learned protagonist policy is also effective against unseen opponents. Experimental results show that the proposed game-theoretic modeling, explicit opponent type inference and the ensemble training significantly improves the decision-making performance over baseline approaches, and generalizes well against adversaries that have not been seen during the training.

Building Endgame Data Set to Improve Opponent Modeling Approach.

Ancillary Mechanism for Autonomous Decision-Making Process in Asymmetric Confrontation: a View from Gomoku

Opponent Modeling in Poker Games

Opponent Modeling in Multiplayer Imperfect-Information Games

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

Auto-encoder Neural Network Based Prediction of Texas Poker Opponent's Behavior

L2E: Learning to Exploit Your Opponent

To Create Neuro-Controlled Game Opponent from UCT-created Data

Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games

Opponent Modeling in Deep Reinforcement Learning

Modeling Rationality: Toward Better Performance Against Unknown Agents in Sequential Games

Model-Based Opponent Modeling

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

Playing Extensive Games with Learning of Opponent's Cognition

Efficient Opponent Exploitation in No-Limit Texas Hold’em Poker: A Neuroevolutionary Method Combined with Reinforcement Learning

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Neural Auto-Curricula in Two-Player Zero-Sum Games.

Neural Auto-Curricula

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning

Opponent portrait for multiagent reinforcement learning in competitive environment

Imperfect Information Game in Multiplayer No-limit Texas Hold’em Based on Mean Approximation and Deep CFVnet