On the Decision-Making Abilities in Role-Playing using Large Language Models

Chenglei Shen,Guofu Xie,Xiao Zhang,Jun Xu

2024-02-29

Abstract:Large language models (LLMs) are now increasingly utilized for role-playing tasks, especially in impersonating domain-specific experts, primarily through role-playing prompts. When interacting in real-world scenarios, the decision-making abilities of a role significantly shape its behavioral patterns. In this paper, we concentrate on evaluating the decision-making abilities of LLMs post role-playing thereby validating the efficacy of role-playing. Our goal is to provide metrics and guidance for enhancing the decision-making abilities of LLMs in role-playing tasks. Specifically, we first use LLMs to generate virtual role descriptions corresponding to the 16 personality types of Myers-Briggs Type Indicator (abbreviated as MBTI) representing a segmentation of the population. Then we design specific quantitative operations to evaluate the decision-making abilities of LLMs post role-playing from four aspects: adaptability, exploration$\&$exploitation trade-off ability, reasoning ability, and safety. Finally, we analyze the association between the performance of decision-making and the corresponding MBTI types through GPT-4. Extensive experiments demonstrate stable differences in the four aspects of decision-making abilities across distinct roles, signifying a robust correlation between decision-making abilities and the roles emulated by LLMs. These results underscore that LLMs can effectively impersonate varied roles while embodying their genuine sociological characteristics.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of evaluating the decision-making capabilities of large language models (LLMs) in role-playing tasks. Specifically, the authors aim to provide metrics and guidance methods to enhance LLMs' decision-making abilities in role-playing tasks. The study first utilizes LLMs to generate virtual character descriptions corresponding to the 16 personality types in the Myers-Briggs Type Indicator (MBTI). Then, specific quantitative methods are designed to assess these models' decision-making abilities after role-playing, including adaptability, exploration-exploitation trade-off ability, reasoning ability, and safety. Through extensive experiments, stable differences in decision-making abilities between different roles were verified, and it was demonstrated that LLMs could reflect their true sociological characteristics when imitating various roles.

On the Decision-Making Abilities in Role-Playing using Large Language Models

Character is Destiny: Can Large Language Models Simulate Persona-Driven Decisions in Role-Playing?

Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Thinking Before Speaking: A Role-playing Model with Mindset

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games

Identifying Multiple Personalities in Large Language Models with External Evaluation

Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context

RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

On the Modeling Capabilities of Large Language Models for Sequential Decision Making

Bias and Toxicity in Role-Play Reasoning

The Emergence of Strategic Reasoning of Large Language Models

Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Play

Benchmarking Bias in Large Language Models during Role-Playing

Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

Introspective Tips: Large Language Model for In-Context Decision Making

Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models

Optimal Decision Making Through Scenario Simulations Using Large Language Models

The Oscars of AI Theater: A Survey on Role-Playing with Language Models

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits