Abstract:Large language models (LLMs) are now increasingly utilized for role-playing tasks, especially in impersonating domain-specific experts, primarily through role-playing prompts. When interacting in real-world scenarios, the decision-making abilities of a role significantly shape its behavioral patterns. In this paper, we concentrate on evaluating the decision-making abilities of LLMs post role-playing thereby validating the efficacy of role-playing. Our goal is to provide metrics and guidance for enhancing the decision-making abilities of LLMs in role-playing tasks. Specifically, we first use LLMs to generate virtual role descriptions corresponding to the 16 personality types of Myers-Briggs Type Indicator (abbreviated as MBTI) representing a segmentation of the population. Then we design specific quantitative operations to evaluate the decision-making abilities of LLMs post role-playing from four aspects: adaptability, exploration$\&$exploitation trade-off ability, reasoning ability, and safety. Finally, we analyze the association between the performance of decision-making and the corresponding MBTI types through GPT-4. Extensive experiments demonstrate stable differences in the four aspects of decision-making abilities across distinct roles, signifying a robust correlation between decision-making abilities and the roles emulated by LLMs. These results underscore that LLMs can effectively impersonate varied roles while embodying their genuine sociological characteristics.

Simulating Ability: Representing Skills in Games

A State-Space Perspective on Modelling and Inference for Online Skill Rating

A hybrid fuzzy logic/constraint satisfaction problem approach to automatic decision making in simulation game models

Capacity ATL

Analyzing Decision-Making in Cognitive Agent Simulations Using Generalized Linear Mixed-Effects Models

People use fast, goal-directed simulation to reason about novel games

Understanding Complex Adaptive Systems by Playing Games.

Realism of Simulation Models in Serious Gaming: Two case studies from Urban Water Management Higher Education

Game Theory with Simulation of Other Players

Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMs

Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning

On the Decision-Making Abilities in Role-Playing using Large Language Models

Imagined Potential Games: A Framework for Simulating, Learning and Evaluating Interactive Behaviors

A general framework of agent-based simulation for analyzing behavior of players in games

Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

Autoformalizing and Simulating Game-Theoretic Scenarios using LLM-augmented Agents

Generalised Player Modelling: Why Artificial Intelligence in Games Should Incorporate Meaning, with a Formalism for so Doing

Informal Safety Guarantees for Simulated Optimizers Through Extrapolation from Partial Simulations

BeSimulator: A Large Language Model Powered Text-based Behavior Simulator

On Languaging a Simulation Engine

Modeling Individual Differences in Game Behavior using HMM