Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Bojana Bodroza,Bojana M. Dinic,Ljubisa Bojic
2024-07-28
Abstract:As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.
Artificial Intelligence,Computation and Language,Human-Computer Interaction
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is the time - stability issue of large language models (LLMs) in personality tests. Specifically, researchers hope to evaluate whether these models' responses to personality test tools are consistent at different time points and how their personality characteristics compare with human normative data. In addition, the study also explores the impression management ability of LLMs in the fields of agency and community, as well as their political tendencies. The study evaluates the temporal stability of LLMs' personality traits through the time interval between two tests and uses a variety of personality test tools, such as the Big Five personality model, the HEXACO (Six - Factor Personality Model), the Dark Triad traits, etc., to comprehensively understand the personality characteristics of LLMs. At the same time, the study also focuses on the performance of LLMs in terms of private and public self - awareness, impression management, etc., as well as their political stances, especially whether they tend to be more liberal or progressive political attitudes. Through this study, the authors hope to provide evidence regarding the stability of LLMs' psychological characteristics, which is of great significance for understanding the social impacts, potential opportunities and risks of these models, and the security in different application scenarios.