Abstract:Faces hold significant importance as visual stimuli in human daily experiences. Although there is substantial research on emotional processing and social evaluations of faces, most studies have focused on each of these topics separately. In this review, we advocate for the adoption of multimodal cognitive neuroscience methods to investigate these phenomena. We illustrate this approach for two specific research areas: ambiguity in facial expressions of emotion and social trait judgment of faces. Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event‐related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single‐neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality‐dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in‐depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

A uniform human multimodal dataset for emotion perception and judgment

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

Multimodal investigations of emotional face processing and social trait judgment of faces

A Multimodal Dataset for Mixed Emotion Recognition

Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

SEED-VII: A Multimodal Dataset of Six Basic Emotions with Continuous Labels for Emotion Recognition

EmpathicSchool: A multimodal dataset for real-time facial expressions and physiological data analysis under different stress conditions

PhyMER: Physiological Dataset for Multimodal Emotion Recognition With Personality as a Context

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

Emotion recognition framework using multiple modalities for an effective human–computer interaction

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks

Multi-modal Approach for Affective Computing