A procedure for the strategic planning of locations, capacities and districting of jails: application to Chile

V. Marianov,F. Fresard

DOI: https://doi.org/10.1057/palgrave.jors.2601790

IF: 3.6

2005-03-01

Journal of the Operational Research Society

Abstract:

What problem does this paper attempt to address?

Large Language Models Can Infer Psychological Dispositions of Social Media Users

Heinrich Peters,Sandra Matz

2024-06-05

Abstract:Large Language Models (LLMs) demonstrate increasingly human-like abilities across a wide variety of tasks. In this paper, we investigate whether LLMs like ChatGPT can accurately infer the psychological dispositions of social media users and whether their ability to do so varies across socio-demographic groups. Specifically, we test whether GPT-3.5 and GPT-4 can derive the Big Five personality traits from users' Facebook status updates in a zero-shot learning scenario. Our results show an average correlation of r = .29 (range = [.22, .33]) between LLM-inferred and self-reported trait scores - a level of accuracy that is similar to that of supervised machine learning models specifically trained to infer personality. Our findings also highlight heterogeneity in the accuracy of personality inferences across different age groups and gender categories: predictions were found to be more accurate for women and younger individuals on several traits, suggesting a potential bias stemming from the underlying training data or differences in online self-expression. The ability of LLMs to infer psychological dispositions from user-generated text has the potential to democratize access to cheap and scalable psychometric assessments for both researchers and practitioners. On the one hand, this democratization might facilitate large-scale research of high ecological validity and spark innovation in personalized services. On the other hand, it also raises ethical concerns regarding user privacy and self-determination, highlighting the need for stringent ethical frameworks and regulation.

Computation and Language,Artificial Intelligence,Computers and Society,Human-Computer Interaction,Machine Learning,Social and Information Networks
Large Language Models Can Infer Personality from Free-Form User Interactions

Heinrich Peters,Moran Cerf,Sandra C. Matz

2024-05-20

Abstract:This study investigates the capacity of Large Language Models (LLMs) to infer the Big Five personality traits from free-form user interactions. The results demonstrate that a chatbot powered by GPT-4 can infer personality with moderate accuracy, outperforming previous approaches drawing inferences from static text content. The accuracy of inferences varied across different conversational settings. Performance was highest when the chatbot was prompted to elicit personality-relevant information from users (mean r=.443, range=[.245, .640]), followed by a condition placing greater emphasis on naturalistic interaction (mean r=.218, range=[.066, .373]). Notably, the direct focus on personality assessment did not result in a less positive user experience, with participants reporting the interactions to be equally natural, pleasant, engaging, and humanlike across both conditions. A chatbot mimicking ChatGPT's default behavior of acting as a helpful assistant led to markedly inferior personality inferences and lower user experience ratings but still captured psychologically meaningful information for some of the personality traits (mean r=.117, range=[-.004, .209]). Preliminary analyses suggest that the accuracy of personality inferences varies only marginally across different socio-demographic subgroups. Our results highlight the potential of LLMs for psychological profiling based on conversational interactions. We discuss practical implications and ethical challenges associated with these findings.

Human-Computer Interaction,Artificial Intelligence,Computation and Language,Computers and Society,Machine Learning
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Hang Jiang,Xiajie Zhang,Xubo Cao,Cynthia Breazeal,Deb Roy,Jad Kabbara

2024-04-02

Abstract:Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas' writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%. Interestingly, the accuracy drops significantly when the annotators were informed of AI authorship.

Computation and Language,Artificial Intelligence,Human-Computer Interaction
Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

Nikolay B Petrov,Gregory Serapio-García,Jason Rentfrow

DOI: https://doi.org/10.48550/arXiv.2405.07248

2024-05-12

Abstract:The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles, show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs' ability to simulate individual-level human behaviour across multiple-choice question answering tasks.

Computation and Language,Artificial Intelligence,Computers and Society,Human-Computer Interaction
PersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences

Hang Jiang,Xubo Cao,Jad Kabbara,Xiajie Zhang,Dwaipayan Roy

DOI: https://doi.org/10.48550/arXiv.2305.02547

Abstract:Extended Abstract Large Language Models (LLMs)—by way of their training and design—can be thought of as implicit computational models of humans [2] and studies are already exploring how these LLMs can be seen as effective proxies for speciﬁc human sub-populations [1]. This is largely because these models were designed to respond to prompts in a similar fashion to how a person would react—which makes them very appealing for applications like chatbots. In that context, an appealing property of LLMs is that their adaptivity to take on the character of different individuals based on certain traits, e.g. personality traits. Research shows that designing chatbots with curated personality proﬁles provides an improved personalized and engaging user experience [5]. Despite the need and clear applications, little work has been done to evaluate whether the behavior of LLM-generated personas can reﬂect certain personality traits accurately and consistently. In this work, we design a case study to address this gap. In this work, we consider studying the behavior of LLM-based simulated agents which refer to as LLM personas and aim to answer the following questions: When GPT-3.5 (text-davinci-003) is assigned a Big Five personality type, (1) do LLM personas consistently express the assigned personality traits in personality tests and writing tasks? (2) Does assigning a gender role have an additional effect on LLM personas’ behavior? To investigate these research questions, we build upon prior work in text-based personality analysis [4] by studying the ability of LLMs to generate content with curated personality traits. Speciﬁcally, we create 10 personas (5

Computer Science,Psychology
Artificial Intelligence and Personality: Large Language Models’ Ability to Predict Personality Type

Max Murphy

DOI: https://doi.org/10.1177/27523543241257291

2024-06-14

Abstract:Micro-targeting, which is when messages are specifically tailored to an individual based on the information that can be derived from their digital footprint, has become a prevalent practice in digital spaces. This communicative approach often centers around personality, the rationale being that individuals resonate with messages differently depending on their personality type. There has been research into whether machine learning methods can improve the detection of personality types, but insufficient analysis has focused on whether large language models enhance this technique even further. And so, this study examines the ability of GPT-3.5 and GPT-4, two of the models underpinning OpenAI's ChatGPT, to classify personality type. Using a publicly available dataset collected through the ‘PersonalityCafe’ forum, this paper found that GPT-3.5 and GPT-4 were able to perfectly classify 73% and 76% of the sample's personality type, respectively, simply by analyzing an individual's 50 most recent tweets—a level which is significantly better than random guessing and other machine learning models. This has important implications for communication scholarship because if LLMs enable actors to detect personality types more accurately, micro-targeting could become even more powerful and widespread. There is also a possibility for misuse by bad actors and privacy threats, so understanding these mechanisms is crucial for building safe artificial intelligence systems and safeguards against this.
Perils and opportunities in using large language models in psychological research

Suhaib Abdurahman,Mohammad Atari,Farzan Karimi-Malekabadi,Mona J Xue,Jackson Trager,Peter S Park,Preni Golazizian,Ali Omrani,Morteza Dehghani

DOI: https://doi.org/10.1093/pnasnexus/pgae245

2024-07-16

PNAS Nexus

Abstract:The emergence of large language models (LLMs) has sparked considerable interest in their potential application in psychological research, mainly as a model of the human psyche or as a general text-analysis tool. However, the trend of using LLMs without sufficient attention to their limitations and risks, which we rhetorically refer to as "GPTology", can be detrimental given the easy access to models such as ChatGPT. Beyond existing general guidelines, we investigate the current limitations, ethical implications, and potential of LLMs specifically for psychological research, and show their concrete impact in various empirical studies. Our results highlight the importance of recognizing global psychological diversity, cautioning against treating LLMs (especially in zero-shot settings) as universal solutions for text analysis, and developing transparent, open methods to address LLMs' opaque nature for reliable, reproducible, and robust inference from AI-generated data. Acknowledging LLMs' utility for task automation, such as text annotation, or to expand our understanding of human psychology, we argue for diversifying human samples and expanding psychology's methodological toolbox to promote an inclusive, generalizable science, countering homogenization, and over-reliance on LLMs.
Humanity in AI: Detecting the Personality of Large Language Models

Baohua Zhan,Yongyi Huang,Wenyao Cui,Huaping Zhang,Jianyun Shang

2024-10-11

Abstract:Questionnaires are a common method for detecting the personality of Large Language Models (LLMs). However, their reliability is often compromised by two main issues: hallucinations (where LLMs produce inaccurate or irrelevant responses) and the sensitivity of responses to the order of the presented options. To address these issues, we propose combining text mining with questionnaires method. Text mining can extract psychological features from the LLMs' responses without being affected by the order of options. Furthermore, because this method does not rely on specific answers, it reduces the influence of hallucinations. By normalizing the scores from both methods and calculating the root mean square error, our experiment results confirm the effectiveness of this approach. To further investigate the origins of personality traits in LLMs, we conduct experiments on both pre-trained language models (PLMs), such as BERT and GPT, as well as conversational models (ChatLLMs), such as ChatGPT. The results show that LLMs do contain certain personalities, for example, ChatGPT and ChatGLM exhibit the personality traits of 'Conscientiousness'. Additionally, we find that the personalities of LLMs are derived from their pre-trained data. The instruction data used to train ChatLLMs can enhance the generation of data containing personalities and expose their hidden personality. We compare the results with the human average personality score, and we find that the personality of FLAN-T5 in PLMs and ChatGPT in ChatLLMs is more similar to that of a human, with score differences of 0.34 and 0.22, respectively.

Computation and Language,Artificial Intelligence
AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories

Max Pellert,Clemens M. Lechner,Claudia Wagner,Beatrice Rammstedt,Markus Strohmaier

DOI: https://doi.org/10.1177/17456916231214460

2024-01-03

Perspectives on Psychological Science

Abstract:Perspectives on Psychological Science, Ahead of Print. We illustrate how standard psychometric inventories originally designed for assessing noncognitive human traits can be repurposed as diagnostic tools to evaluate analogous traits in large language models (LLMs). We start from the assumption that LLMs, inadvertently yet inevitably, acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained. Such corpora contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts, which LLMs learn through a complex training process. The traits that LLMs acquire in such a way can potentially influence their behavior, that is, their outputs in downstream tasks and applications in which they are employed, which in turn may have real-world consequences for individuals and social groups. By eliciting LLMs' responses to language-based psychometric inventories, we can bring their traits to light. Psychometric profiling enables researchers to study and compare LLMs in terms of noncognitive characteristics, thereby providing a window into the personalities, values, beliefs, and biases these models exhibit (or mimic). We discuss the history of similar ideas and outline possible psychometric approaches for LLMs. We demonstrate one promising approach, zero-shot classification, for several LLMs and psychometric inventories. We conclude by highlighting open challenges and future avenues of research for AI Psychometrics.

psychology, multidisciplinary
Large Language Models Show Human-like Social Desirability Biases in Survey Responses

Aadesh Salecha,Molly E. Ireland,Shashanka Subrahmanya,João Sedoc,Lyle H. Ungar,Johannes C. Eichstaedt

2024-05-10

Abstract:As Large Language Models (LLMs) become widely used to model and simulate human behavior, understanding their biases becomes critical. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e., increased extraversion, decreased neuroticism, etc). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4's survey responses changing by 1.20 (human) standard deviations and Llama 3's by 0.98 standard deviations-very large effects. This bias is robust to randomization of question order and paraphrasing. Reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on using LLMs as proxies for human participants.

Artificial Intelligence,Computation and Language,Computers and Society,Human-Computer Interaction
Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Haocong Rao,Cyril Leung,Chunyan Miao

2023-10-13

Abstract:Large Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. Specifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation. Then, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and GPT-4. Our experiments reveal ChatGPT's ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT.

Computation and Language,Artificial Intelligence
Personality Traits in Large Language Models

Greg Serapio-García,Mustafa Safdari,Clément Crepy,Luning Sun,Stephen Fitz,Peter Romero,Marwa Abdulhai,Aleksandra Faust,Maja Matarić

2023-09-22

Abstract:The advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant human-like text. As LLMs increasingly power conversational agents used by the general public world-wide, the synthetic personality embedded in these models, by virtue of training on large amounts of human data, is becoming increasingly important. Since personality is a key factor determining the effectiveness of communication, we present a comprehensive method for administering and validating personality tests on widely-used LLMs, as well as for shaping personality in the generated text of such LLMs. Applying this method, we found: 1) personality measurements in the outputs of some LLMs under specific prompting configurations are reliable and valid; 2) evidence of reliability and validity of synthetic LLM personality is stronger for larger and instruction fine-tuned models; and 3) personality in LLM outputs can be shaped along desired dimensions to mimic specific human personality profiles. We discuss application and ethical implications of the measurement and shaping method, in particular regarding responsible AI.

Computation and Language,Artificial Intelligence,Computers and Society,Human-Computer Interaction
Can large language models help predict results from a complex behavioural science study?

Steffen Lippert,Anna Dreber,Magnus Johannesson,Warren Tierney,Wilson Cyrus-Lai,Eric Luis Uhlmann,Emotion Expression Collaboration,Thomas Pfeiffer

DOI: https://doi.org/10.1098/rsos.240682

IF: 3.5

2024-09-26

Royal Society Open Science

Abstract:We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate—at scale and minimal cost—which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human–AI collaboration in science.

multidisciplinary sciences
Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Bojana Bodroza,Bojana M. Dinic,Ljubisa Bojic

2024-07-28

Abstract:As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.

Artificial Intelligence,Computation and Language,Human-Computer Interaction
Revisiting the Reliability of Psychological Scales on Large Language Models

Jen-tse Huang,Wenxiang Jiao,Man Ho Lam,Eric John Li,Wenxuan Wang,Michael R. Lyu

2024-10-04

Abstract:Recent research has focused on examining Large Language Models' (LLMs) characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory, indicating a satisfactory level of reliability. Furthermore, our research explores the potential of GPT-3.5 to emulate diverse personalities and represent various groups-a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions.

Computation and Language
GPT is an effective tool for multilingual psychological text analysis

Steve Rathje,Dan-Mircea Mirea,Ilia Sucholutsky,Raja Marjieh,Claire E Robertson,Jay J Van Bavel

DOI: https://doi.org/10.1073/pnas.2308950121

2024-08-20

Abstract:The social and behavioral sciences have been increasingly using automated text analysis to measure psychological constructs in text. We explore whether GPT, the large-language model (LLM) underlying the AI chatbot ChatGPT, can be used as a tool for automated psychological text analysis in several languages. Across 15 datasets (n = 47,925 manually annotated tweets and news headlines), we tested whether different versions of GPT (3.5 Turbo, 4, and 4 Turbo) can accurately detect psychological constructs (sentiment, discrete emotions, offensiveness, and moral foundations) across 12 languages. We found that GPT (r = 0.59 to 0.77) performed much better than English-language dictionary analysis (r = 0.20 to 0.30) at detecting psychological constructs as judged by manual annotators. GPT performed nearly as well as, and sometimes better than, several top-performing fine-tuned machine learning models. Moreover, GPT's performance improved across successive versions of the model, particularly for lesser-spoken languages, and became less expensive. Overall, GPT may be superior to many existing methods of automated text analysis, since it achieves relatively high accuracy across many languages, requires no training data, and is easy to use with simple prompts (e.g., "is this text negative?") and little coding experience. We provide sample code and a video tutorial for analyzing text with the GPT application programming interface. We argue that GPT and other LLMs help democratize automated text analysis by making advanced natural language processing capabilities more accessible, and may help facilitate more cross-linguistic research with understudied languages.
ChatGPT vs Social Surveys: Probing the Objective and Subjective Human Society

Muzhi Zhou,Lu Yu,Xiaomin Geng,Lan Luo

2024-09-04

Abstract:The extent to which Large Language Models (LLMs) can simulate the data-generating process for social surveys remains unclear. Current research has not thoroughly assessed potential biases in the sociodemographic population represented within the language model's framework. Additionally, the subjective worlds of LLMs often show inconsistencies in how closely their responses match those of groups of human respondents. In this paper, we used ChatGPT-3.5 to simulate the sampling process and generated six socioeconomic characteristics from the 2020 US population. We also analyzed responses to questions about income inequality and gender roles to explore GPT's subjective attitudes. By using repeated random sampling, we created a sampling distribution to identify the parameters of the GPT-generated population and compared these with Census data. Our findings show some alignment in gender and age means with the actual 2020 US population, but we also found mismatches in the distributions of racial and educational groups. Furthermore, there were significant differences between the distribution of GPT's responses and human self-reported attitudes. While the overall point estimates of GPT's income attitudinal responses seem to align with the mean of the population occasionally, their response distributions follow a normal distribution that diverges from human responses. In terms of gender relations, GPT's answers tend to cluster in the most frequently answered category, demonstrating a deterministic pattern. We conclude by emphasizing the distinct design philosophies of LLMs and social surveys: LLMs aim to predict the most suitable answers, while social surveys seek to reveal the heterogeneity among social groups.

Computers and Society
Using large language models in psychology

Dorottya Demszky,Diyi Yang,David S. Yeager,Christopher J. Bryan,Margarett Clapper,Susannah Chandhok,Johannes C. Eichstaedt,Cameron Hecht,Jeremy Jamieson,Meghann Johnson,Michaela Jones,Danielle Krettek-Cobb,Leslie Lai,Nirel JonesMitchell,Desmond C. Ong,Carol S. Dweck,James J. Gross,James W. Pennebaker

DOI: https://doi.org/10.1038/s44159-023-00241-5

2023-10-14

Nature Reviews Psychology

Abstract:Large language models (LLMs), such as OpenAI's GPT-4, Google's Bard or Meta's LLaMa, have created unprecedented opportunities for analysing and generating language data on a massive scale. Because language data have a central role in all areas of psychology, this new technology has the potential to transform the field. In this Perspective, we review the foundations of LLMs. We then explain how the way that LLMs are constructed enables them to effectively generate human-like linguistic output without the ability to think or feel like a human. We argue that although LLMs have the potential to advance psychological measurement, experimentation and practice, they are not yet ready for many of the most transformative psychological applications — but further research and development may enable such use. Next, we examine four major concerns about the application of LLMs to psychology, and how each might be overcome. Finally, we conclude with recommendations for investments that could help to address these concerns: field-initiated 'keystone' datasets; increased standardization of performance benchmarks; and shared computing and analysis infrastructure to ensure that the future of LLM-powered research is equitable.
Inferring Human Traits From Facebook Statuses

Andrew Cutler,Brian Kulis

DOI: https://doi.org/10.48550/arXiv.1805.08718

2018-05-22

Social and Information Networks

Abstract:This paper explores the use of language models to predict 20 human traits from users' Facebook status updates. The data was collected by the myPersonality project, and includes user statuses along with their personality, gender, political identification, religion, race, satisfaction with life, IQ, self-disclosure, fair-mindedness, and belief in astrology. A single interpretable model meets state of the art results for well-studied tasks such as predicting gender and personality; and sets the standard on other traits such as IQ, sensational interests, political identity, and satisfaction with life. Additionally, highly weighted words are published for each trait. These lists are valuable for creating hypotheses about human behavior, as well as for understanding what information a model is extracting. Using performance and extracted features we analyze models built on social media. The real world problems we explore include gendered classification bias and Cambridge Analytica's use of psychographic models.
Challenging the Validity of Personality Tests for Large Language Models

Tom Sühr,Florian E. Dorner,Samira Samadi,Augustin Kelava

2024-06-05

Abstract:With large language models (LLMs) like GPT-4 appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate personality traits of LLMs using questionnaires originally developed for humans. While reusing measures is a resource-efficient way to evaluate LLMs, careful adaptations are usually required to ensure that assessment results are valid even across human subpopulations. In this work, we provide evidence that LLMs' responses to personality tests systematically deviate from human responses, implying that the results of these tests cannot be interpreted in the same way. Concretely, reverse-coded items ("I am introverted" vs. "I am extraverted") are often both answered affirmatively. Furthermore, variation across prompts designed to "steer" LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe that it is important to investigate tests' validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs' "personality".

Computation and Language,Artificial Intelligence,Machine Learning

A procedure for the strategic planning of locations, capacities and districting of jails: application to Chile

Large Language Models Can Infer Psychological Dispositions of Social Media Users

Large Language Models Can Infer Personality from Free-Form User Interactions

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

PersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences

Artificial Intelligence and Personality: Large Language Models’ Ability to Predict Personality Type

Perils and opportunities in using large language models in psychological research

Humanity in AI: Detecting the Personality of Large Language Models

AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories

Large Language Models Show Human-like Social Desirability Biases in Survey Responses

Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Personality Traits in Large Language Models

Can large language models help predict results from a complex behavioural science study?

Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Revisiting the Reliability of Psychological Scales on Large Language Models

GPT is an effective tool for multilingual psychological text analysis

ChatGPT vs Social Surveys: Probing the Objective and Subjective Human Society

Using large language models in psychology

Inferring Human Traits From Facebook Statuses

Challenging the Validity of Personality Tests for Large Language Models