Abstract:Personality determines a wide variety of human daily and working behaviours, and is crucial for understanding human internal and external states. In recent years, a large number of automatic personality computing approaches have been developed to predict either the apparent personality or self-reported personality of the subject based on non-verbal audio-visual behaviours. However, the majority of them suffer from complex and dataset-specific pre-processing steps and model training tricks. In the absence of a standardized benchmark with consistent experimental settings, it is not only impossible to fairly compare the real performances of these personality computing models but also makes them difficult to be reproduced. In this paper, we present the first reproducible audio-visual benchmarking framework to provide a fair and consistent evaluation of eight existing personality computing models (e.g., audio, visual and audio-visual) and seven standard deep learning models on both self-reported and apparent personality recognition tasks. Building upon a set of benchmarked models, we also investigate the impact of two previously-used long-term modelling strategies for summarising short-term/frame-level predictions on personality computing results. We conduct a comprehensive investigation into all the benchmarked models to demonstrate their capabilities in modelling personality traits on two publicly available datasets, audio-visual apparent personality (ChaLearn First Impression) and self-reported personality (UDIVA) datasets. The experimental results conclude: (i) apparent personality traits, inferred from facial behaviours by most benchmarked deep learning models, show more reliability than self-reported ones; (ii) visual models frequently achieved superior performances than audio models on personality recognition; (iii) non-verbal behaviours contribute differently in predicting different personality traits; and (iv) our reproduced personality computing models generally achieved worse performances than their original reported results. We make all the code and settings of this personality computing benchmark publicly available at https://github.com/liaorongfan/DeepPersonality.

Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition

Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks

Bi-modal First Impressions Recognition using Temporally Ordered Deep Audio and Stochastic Visual Features

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning

Deep Bimodal Regression For Apparent Personality Analysis

CR-Net: A Deep Classification-Regression Network for Multimodal Apparent Personality Analysis

Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings

Deep Bimodal Regression of Apparent Personality Traits from Short Video Sequences.

Video-based multimodal personality analysis

Multimodal analysis of personality traits on videos of self-presentation and induced behavior

An Open-source Benchmark of Deep Learning Models for Audio-visual Apparent and Self-reported Personality Recognition

Contextualized Multidimensional Personality Recognition using Combination of Deep Neural Network and Ensemble Learning

Predicting and visualizing psychological attributions with a deep neural network

Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction

Adaptive information fusion network for multi‐modal personality recognition

Deep learning-based personality recognition from text posts of online social networks

Background Hardly Matters: Understanding Personality Attribution in Deep Residual Networks

PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship

End-to-End Modeling and Transfer Learning for Audiovisual Emotion Recognition in-the-Wild

The Deeper, the Better: Analysis of Person Attributes Recognition

IMPACT OF VISUAL MODALITIES IN MULTIMODAL PERSONALITY AND AFFECTIVE COMPUTING