Abstract:As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.

LLMs' ways of seeing User Personas

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs

Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization

Understanding User Experience in Large Language Model Interactions

Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

On the steerability of large language models toward data-driven personas

Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

"I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices

Hate Personified: Investigating the role of LLMs in content moderation

Persona-L has Entered the Chat: Leveraging LLM and Ability-based Framework for Personas of People with Complex Needs

Identifying Multiple Personalities in Large Language Models with External Evaluation

Apprentices to Research Assistants: Advancing Research with Large Language Models

How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions

Large Language Models Humanize Technology

Improved prompting and process for writing user personas with LLMs, using qualitative interviews: Capturing behaviour and personality traits of users

Understanding the Role of User Profile in the Personalization of Large Language Models

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

LLMs as Academic Reading Companions: Extending HCI Through Synthetic Personae

Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agents