PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychology

Edoardo Sebastiano De Duro,Enrique Taietta,Riccardo Improta,Massimo Stella
2024-11-07
Abstract:Machine psychology aims to reconstruct the mindset of Large Language Models (LLMs), i.e. how these artificial intelligences perceive and associate ideas. This work introduces PhDGPT, a prompting framework and synthetic dataset that encapsulates the machine psychology of PhD researchers and professors as perceived by OpenAI's GPT-3.5. The dataset consists of 756,000 datapoints, counting 300 iterations repeated across 15 academic events, 2 biological genders, 2 career levels and 42 unique item responses of the Depression, Anxiety, and Stress Scale (DASS-42). PhDGPT integrates these psychometric scores with their explanations in plain language. This synergy of scores and texts offers a dual, comprehensive perspective on the emotional well-being of simulated academics, e.g. male/female PhD students or professors. By combining network psychometrics and psycholinguistic dimensions, this study identifies several similarities and distinctions between human and LLM data. The psychometric networks of simulated male professors do not differ between physical and emotional anxiety subscales, unlike humans. Other LLMs' personification can reconstruct human DASS factors with a purity up to 80%. Furthemore, LLM-generated personifications across different scenarios are found to elicit explanations lower in concreteness and imageability in items coding for anxiety, in agreement with past studies about human psychology. Our findings indicate an advanced yet incomplete ability for LLMs to reproduce the complexity of human psychometric data, unveiling convenient advantages and limitations in using LLMs to replace human participants. PhDGPT also intriguingly capture the ability for LLMs to adapt and change language patterns according to prompted mental distress contextual features, opening new quantitative opportunities for assessing the machine psychology of these artificial intelligences.
Human-Computer Interaction,Artificial Intelligence,Computation and Language,Computers and Society
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to evaluate how large language models (LLMs) perceive the mental health status of doctoral students and professors in the field of psychology by introducing a new framework named PhDGPT and a synthetic dataset. Specifically, the research aims to: 1. **Differences in Psychological Stress Score in Different Academic Situations**: Explore the impact of different academic situations (such as examinations, research, publications, work - life balance, interpersonal relationships, etc.) on the mental health scores (depression, anxiety, and stress measured by the DASS - 42 scale) of doctoral students and professors, and whether there are statistically significant differences in these scores among different genders and situations. 2. **Whether LLMs Can Accurately Reproduce Psychometric Dimensions**: Verify whether LLMs can accurately simulate the psychometric dimensions (i.e., depression, anxiety, and stress) of humans in specific academic situations, and compare them with real human data to assess the accuracy of LLMs in psychometric measurement. 3. **Psycholinguistic Patterns in the Text Generated by LLMs**: Analyze the psycholinguistic patterns (such as emotional arousal, concreteness, vividness, etc.) in the text generated by LLMs in different academic situations, and how these patterns are associated with mental health states (such as depression, anxiety, stress). Through the research of these problems, the paper aims to explore the potential and limitations of LLMs in simulating human mental health, and provide new perspectives and tools for future research in machine psychology and other related fields.