PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Hang Jiang,Xiajie Zhang,Xubo Cao,Cynthia Breazeal,Deb Roy,Jad Kabbara

2024-04-02

Abstract:Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas' writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%. Interestingly, the accuracy drops significantly when the annotators were informed of AI authorship.

Computation and Language,Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The paper aims to explore the ability of large language models (LLMs) to express personality traits. Specifically, researchers define LLM-based agents (referred to as LLM personas) that are prompted to generate content reflecting specific personality traits. The core question of the study is to verify whether these LLM personas can accurately reflect the personality traits they are assigned when completing the Big Five Inventory (BFI) test. Additionally, the study explores how these agents with specific personality traits are perceived by humans, i.e., whether humans can accurately perceive these personality traits through the stories generated by the agents.

The main objectives of the study include:
1. **Verifying whether LLM personas can reflect the specified personality traits when completing the BFI assessment** (RQ1).
2. **Analyzing whether there are specific language patterns in the stories generated by LLM personas** (RQ2).
3. **Evaluating the perceptions of human and LLM evaluators regarding the stories generated by LLM personas** (RQ3).
4. **Examining whether humans and LLMs can accurately identify the Big Five personality traits from the stories** (RQ4).