Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselines

Yaochen Zhu,Xiangqing Shen,Rui Xia

2024-03-04

Abstract:Personality traits, emotions, and beliefs shape individuals' behavioral choices and decision-making processes. However, for one thing, the affective computing community normally focused on predicting personality traits but overlooks their application in behavior prediction. For another, the multimodal reasoning task emphasized the prediction of future states and behaviors but often neglected the incorporation of individual personality traits. In this work, we introduce a new task called Personality-aware Human-centric Multimodal Reasoning (PHMR) (T1), with the goal of forecasting the future behavior of a particular individual using multimodal information from past instances, while integrating personality factors. We accordingly construct a new dataset based on six television shows, encompassing 225 characters and 12k samples. To establish a benchmark for the task, we propose seven baseline methods: three adapted from related tasks, two pre-trained model, and two multimodal large language models. The experimental results demonstrate that incorporating personality traits enhances human-centric multimodal reasoning performance. To further solve the lack of personality annotation in real-life scenes, we introduce an extension task called Personality-predicted Human-centric Multimodal Reasoning task (T2) along with the corresponding dataset and method. We will make our dataset and code available on GitHub.

Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address two main issues:

1. **Insufficient Application of Personality Factors in Behavior Prediction**:
- In the field of affective computing, research typically focuses on predicting personality traits but neglects the application of these traits in behavior prediction.
- Multimodal reasoning tasks (such as social IQ, video language event prediction, etc.) emphasize predicting future states and behaviors but often overlook the integration of individual personality traits.

2. **Lack of Personality Annotations in Real-World Scenarios**:
- In real life, obtaining personality annotations for individuals is very difficult. This limits the application of personality information in multimodal reasoning tasks.

### Solutions

To address the above issues, the authors propose the following solutions:

1. **Introducing a New Task: Personality-Aware Human-Centric Multimodal Reasoning (PHMR)**:
- **Task Objective**: Predict the most likely behavior of a specific individual in future complex social interactions by integrating past multimodal information (video, dialogue, audio) and personality traits.
- **Dataset**: A new dataset (PHMRD) was constructed based on six TV series, containing 225 characters and 12,000 samples.
- **Baseline Methods**: Seven baseline methods were designed, including methods adapted from related tasks, pre-trained models, and multimodal large language models.

2. **Introducing an Extended Task: Personality Prediction Human-Centric Multimodal Reasoning (T2)**:
- **Task Objective**: Predict individual personality traits using multimodal information and use these predicted traits as surrogate annotations to enhance the reasoning process.
- **Dataset**: A new dataset (MPPD) was constructed for the personality prediction task.

### Experimental Results

- **Experimental Results**: Experiments show that integrating personality traits can significantly improve the performance of multimodal reasoning. The proposed models outperform baseline methods under multiple modality settings.
- **Extended Task Results**: In the absence of personality annotations, predicting personality traits using multimodal information can effectively enhance the performance of multimodal reasoning.

### Conclusion

By introducing the Personality-Aware Human-Centric Multimodal Reasoning task (PHMR) and the Personality Prediction Human-Centric Multimodal Reasoning task (T2), this paper successfully addresses the insufficient application of personality factors in behavior prediction and the lack of personality annotations in real-world scenarios. Experimental results validate the effectiveness of these methods.