Abstract:The advent of Artificial Intelligence (AI) technologies has precipitated the rise of asynchronous video interviews (AVIs) as an alternative to conventional job interviews. These one-way video interviews are conducted online and can be analyzed using AI algorithms to automate and speed up the selection procedure. In particular, the swift advancement of Large Language Models (LLMs) has significantly decreased the cost and technical barrier to developing AI systems for automatic personality and interview performance evaluation. However, the generative and task-unspecific nature of LLMs might pose potential risks and biases when evaluating humans based on their AVI responses. In this study, we conducted a comprehensive evaluation of the validity, reliability, fairness, and rating patterns of two widely-used LLMs, GPT-3.5 and GPT-4, in assessing personality and interview performance from an AVI. We compared the personality and interview performance ratings of the LLMs with the ratings from a task-specific AI model and human annotators using simulated AVI responses of 685 participants. The results show that LLMs can achieve similar or even better zero-shot validity compared with the task-specific AI model when predicting personality traits. The verbal explanations for predicting personality traits generated by LLMs are interpretable by the personality items that are designed according to psychological theories. However, LLMs also suffered from uneven performance across different traits, insufficient test-retest reliability, and the emergence of certain biases. Thus, it is necessary to exercise caution when applying LLMs for human-related application scenarios, especially for significant decisions such as employment.

Large language models and humans converge in judging public figures' personalities

Large language models know how the personality of public figures is perceived by the general public

Large Language Models Can Infer Personality from Free-Form User Interactions

Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Artificial Intelligence and Personality: Large Language Models’ Ability to Predict Personality Type

Humanity in AI: Detecting the Personality of Large Language Models

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Large Language Models Can Infer Psychological Dispositions of Social Media Users

Large language models can replicate cross-cultural differences in personality

Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation

Can AI Understand Human Personality? -- Comparing Human Experts and AI Systems at Predicting Personality Correlations

Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Dynamic Generation of Personalities with Large Language Models

Can Large Language Models Assess Personality from Asynchronous Video Interviews? A Comprehensive Evaluation of Validity, Reliability, Fairness, and Rating Patterns

Is persona enough for personality? Using ChatGPT to reconstruct an agent's latent personality from simple descriptions

Is ChatGPT a Good Personality Recognizer? A Preliminary Study

Personality Traits in Large Language Models

Can large language models help predict results from a complex behavioural science study?

Challenging the Validity of Personality Tests for Large Language Models