Laetitia A. Renier,Kumar Shubham,Rahil Satyanarayan Vijay,Swasti Shreya Mishra,Emmanuelle P. Kleinlogel,Dinesh Babu Jayagopi,Marianne Schmid Mast
Abstract:Artificial intelligence (AI)-generated media is used in entertainment, art, education, and marketing. AI-generated faces or facial expressions using deepfake-based technologies might also contribute to nonverbal behavior studies. As a proof of concept, in this research, we test the replicability of past results regarding the positive effects of facial expressiveness (e.g., gazing, nodding, and smiling) on social outcomes. Because the facial expressions when listening to someone (i.e., nonverbal immediacy) encompass several behaviors that are typically related (i.e., smiling while nodding), the effect of combinations of these behaviors is challenging to study. We thus test the effect of facial expressiveness (gazing, nodding, and smiling vs. none) on first impression judgements in a job interview setting using AI-generated experimental video material. We measure how competent, warm, and favorably independent observers ( n = 823) perceive the targets ( n = 159) showing AI-generated facial expressions. Our results replicate past research showing that the more targets are facially expressive, the more they are perceived favorably. Apart from supporting evidence of the importance of facial expressiveness for conveying a positive impression, we show the benefits of relying on AI-generated experimental video material for the study of nonverbal behavior.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the role of facial expressions in social interactions, especially their impact on interview results. Specifically, the researchers used deepfake technology to generate video materials with rich facial expressions to test the influence of facial expressions (such as gazing, nodding, and smiling) on first - impression judgments. The main objective of the study was to verify whether people with rich facial expressions are more popular than those with less expressive faces, are considered more capable and warm, and leave a better overall impression.
### Research Background and Problem Definition
1. **Limitations of Traditional Methods**:
- Traditional non - verbal behavior research usually uses real video clips in which the target objects (i.e., actors or untrained participants) vary in expressiveness, and then asks observers for their impressions.
- This method has several problems:
- Non - verbal behaviors show individual differences (for example, one person's smile may be wider or more subtle than another's).
- Many non - verbal expressive behaviors are related to other target characteristics. For example, women are usually more non - verbally expressive than men.
- In order to separate the gender effect and the non - verbal behavior effect, researchers need to be able to precisely control these behaviors (for example, make a female face and a male face smile or gaze with the same intensity and frequency).
2. **Advantages of Deepfake Technology**:
- Deepfake technology allows researchers to generate highly standardized facial expression videos, thereby controlling temporality (for example, when and how long to make eye contact), frequency (for example, smile twice or five times), intensity (for example, a slight or obvious nod), and co - occurrence (for example, nod while smiling).
- Using AI - generated videos can avoid the individual differences and correlation problems in traditional methods, thus more accurately assessing the impact of facial expressions on social outcomes.
### Research Hypotheses
1. **Main Hypothesis**:
- Hypothesis 1: People with rich facial expressions (i.e., gazing, nodding, and smiling) receive higher evaluations in terms of ability, warmth, and overall likability than those with less expressive faces (i.e., not gazing, not nodding, and not smiling).
2. **Secondary Hypothesis**:
- Hypothesis 2: The impact of facial expressions on social outcomes remains consistent across different cultures and genders.
### Research Methods
- **Experimental Design**:
- The researchers collected self - portraits of 159 target objects and used deepfake technology to generate two sets of videos: one set showed the target objects with rich facial expressions (gazing, nodding, and smiling), and the other set showed the target objects with less expressive faces (not gazing, not nodding, and not smiling).
- 823 observers watched these videos and reported their perceptions of the target objects' ability, warmth, and overall likability.
- **Data Analysis**:
- Through paired t - tests and mixed ANOVA analyses, the researchers evaluated the impact of facial expressions on observers' perceptions and explored the moderating effects of culture (Switzerland and India) and gender (male and female).
### Research Results
- **Main Findings**:
- The target objects with rich facial expressions received significantly higher evaluations in terms of ability, warmth, and overall likability.
- These effects remained consistent across different cultures and genders, but there were some interaction effects. In particular, in the case of less expressive facial expressions, there were differences in evaluations between different cultures and genders.
### Conclusions
- **Theoretical Contributions**:
- This study provides additional evidence to support the positive impact of rich facial expressions on interview results.
- The research results show that this effect is stable across different cultures and genders.
- **Methodological Contributions**:
- This study provides a new method of using deepfake technology to generate highly standardized video materials, thus overcoming the limitations of traditional research methods.
- This technology not only improves the standardization level of the research but also reduces the cost of experimental material development.
- **Practical Significance**:
- The findings of this study have important guiding significance for the recruitment and interview processes, emphasizing the important role of non - verbal behaviors in forming positive first impressions.