Abstract:Extended Abstract Large Language Models (LLMs)—by way of their training and design—can be thought of as implicit computational models of humans [2] and studies are already exploring how these LLMs can be seen as effective proxies for speciﬁc human sub-populations [1]. This is largely because these models were designed to respond to prompts in a similar fashion to how a person would react—which makes them very appealing for applications like chatbots. In that context, an appealing property of LLMs is that their adaptivity to take on the character of different individuals based on certain traits, e.g. personality traits. Research shows that designing chatbots with curated personality proﬁles provides an improved personalized and engaging user experience [5]. Despite the need and clear applications, little work has been done to evaluate whether the behavior of LLM-generated personas can reﬂect certain personality traits accurately and consistently. In this work, we design a case study to address this gap. In this work, we consider studying the behavior of LLM-based simulated agents which refer to as LLM personas and aim to answer the following questions: When GPT-3.5 (text-davinci-003) is assigned a Big Five personality type, (1) do LLM personas consistently express the assigned personality traits in personality tests and writing tasks? (2) Does assigning a gender role have an additional effect on LLM personas’ behavior? To investigate these research questions, we build upon prior work in text-based personality analysis [4] by studying the ability of LLMs to generate content with curated personality traits. Speciﬁcally, we create 10 personas (5

Large language models know how the personality of public figures is perceived by the general public

Large language models and humans converge in judging public figures' personalities

Artificial Intelligence and Personality: Large Language Models’ Ability to Predict Personality Type

Large Language Models Can Infer Psychological Dispositions of Social Media Users

Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Large Language Models Can Infer Personality from Free-Form User Interactions

Large language models can replicate cross-cultural differences in personality

Humanity in AI: Detecting the Personality of Large Language Models

Identifying and Manipulating the Personality Traits of Language Models

Dynamic Generation of Personalities with Large Language Models

PersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences

Revisiting the Reliability of Psychological Scales on Large Language Models

Can large language models help predict results from a complex behavioural science study?

Personality Traits in Large Language Models

Challenging the Validity of Personality Tests for Large Language Models

Is persona enough for personality? Using ChatGPT to reconstruct an agent's latent personality from simple descriptions

Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Personality Predictions Based on User Behavior on the Facebook Social Media Platform