Abstract:This paper studies decision-making in two-player scenarios where the type (e.g. adversary, neutral, or teammate) of the other agent (opponent) is uncertain to the decision-making agent (protagonist), which is an abstraction of security-domain applications. In these settings, the reward for the protagonist agent depends on the type of the opponent, but this is private information known only to the opponent itself, and thus hidden from the protagonist. In contrast, as is often the case, the type of the protagonist agent is assumed to be known to the opponent, and this information-asymmetry significantly complicates the protagonist's decision-making. In particular, to determine the best actions to take, the protagonist agent must infer the opponent type from the observations and agent modeling. To address this problem, this paper presents an opponent-type deduction module based on Bayes' rule. This inference module takes as input the imagined opponent's decision-making rule (opponent model) as well as the observed opponent's history of actions and states, and outputs a belief over the opponent's hidden type. A multiagent reinforcement learning approach is used to develop this game-theoretic opponent model through self-play, which avoids the expensive data collection step that requires interaction with a real opponent. Besides, this multiagent approach also captures the strategy interaction and reasoning between agents. In addition, we apply ensemble training to avoid over-fitting to a single opponent model during the training. As a result, the learned protagonist policy is also effective against unseen opponents. Experimental results show that the proposed game-theoretic modeling, explicit opponent type inference and the ensemble training significantly improves the decision-making performance over baseline approaches, and generalizes well against adversaries that have not been seen during the training.

Modeling Friends and Foes

Ancillary Mechanism for Autonomous Decision-Making Process in Asymmetric Confrontation: a View from Gomoku

Opponent Modeling in Multiplayer Imperfect-Information Games

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning

Modelling the creation of friends and foes groups in small real social networks

A Game Model for Adversarial Classification in Spam Filtering

Friend- and Enemy-oriented Hedonic Games With Strangers Full Version

Playing Extensive Games with Learning of Opponent's Cognition

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Moody Learners -- Explaining Competitive Behaviour of Reinforcement Learning Agents

Finding Friend and Foe in Multi-Agent Games

A survey of decision making in adversarial games

Adversarial Reconnaissance Mitigation and Modeling

Detecting and Deterring Manipulation in a Cognitive Hierarchy

Opponent Modeling in Deep Reinforcement Learning

A Model of Risk and Mental State Shifts during Social Interaction

Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control

Adversarial Coordination on Social Networks

Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning

Adversaries in Online Learning Revisited: with applications in Robust Optimization and Adversarial training