Abstract:Emotion is a rapidly changing psychological and physical phenomenon. In daily life, people need to use the information of various sensory modalities (visual, auditory, etc.) to perceive non-verbal emotional information. Non-verbal emotion from faces and voices are often complex and varied. Previous studies have revealed that there are common and distinct neural networks underlying perception of human faces and voices. However, the neural mechanisms underlying visual and auditory emotional perception have not been well studied. Furthermore, despite researches on audiovisual integration of cross-modal emotional information, the multisensory cortex of the visual and auditory emotional information remains elusive. Therefore, it is necessary to study the similarities and differences between the neural mechanisms of emotion perception in visual and auditory modalities, and to explore the multiple sensory cortex of cross-modal emotion perception.The present fMRI (functional magnetic resonance imaging) study adopted a 2*3 (stimulus presentation modality: Visual, auditory; emotional valence: Happy, sad, fear) event-related design to investigate the neural mechanisms of emotion perception in visual and auditory modalities. When the stimulus (an emotional face or voice) was visually or aurally presented, participants were required to make a gender judgement. The results showed that the activation intensity of emotional faces in V1−V4, bilateral fusiform gyrus and bilateral superior temporal sulcus (STS) was significantly higher than that of emotional voices. Conversely, the activation intensity of emotional voices in auditory cortex (AC) was significantly higher than that of emotional faces. The results from multivoxel pattern analysis (MVPA) showed that the activation patterns of the right STS could discriminate the perception of human faces with emotional valence (happy, sad and fear face), indicating that the rSTG plays important role in perception of faces with different emotional valence; the activation patterns of the right FFA were different for happy and sad faces, indicating that the rFFA is crucial for positive and negative emotional face perception. A voxel-based whole brain analysis was further performed to examine the cortical areas that modulated perception of emotional valence. The whole brain analysis showed that the main effects for emotional valence was significant in the left opercular part of inferior frontal gyrus, indicating that this region might be a multisensory cortex of visual-auditory emotional perception. In summary, our study provided important evidence for further understanding the processing of emotion perception in different modalities and multisensory cortex of cross-channel emotion perception.

Spatial representation of multidimensional information in emotional faces revealed by fMRI

Gender Differences in the Processing of Standard Emotional Visual Stimuli: Integrating ERP and Fmri Results

The Fusiform Face Area is Engaged in Holistic, Not Parts-Based, Representation of Faces.

Identifying the Hierarchical Emotional Areas in the Human Brain Through Information Fusion

Study on multidimensional emotion recognition fusing dynamic brain network features in EEG signals

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Emotion recognition based on brain-like multimodal hierarchical perception

Neural Substrates for Explicit Recognition of Dynamic Facial Expression by Fmri

Decoding six basic emotions from brain functional connectivity patterns

Dissociable Roles of Internal Feelings and Face Recognition Ability in Facial Expression Decoding.

Decoding facial expressions based on face-selective and motion-sensitive areas

Neural Representations of Emotions in Visual, Auditory, and Modality‐Independent Regions Reflect Idiosyncratic Conceptual Knowledge

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

Neural Univariate Activity and Multivariate Pattern in the Posterior Superior Temporal Sulcus Differentially Encode Facial Expression and Identity

Similar and distinct neural mechanisms of visual and auditory emotion perception

Distributing expressional faces in 2-D emotional space

A study of neural mechanism in emotion regulation by simultaneous recording of EEG and fMRI based on ICA

Layer-specific, retinotopically-diffuse modulation in human visual cortex by emotional faces

Decoding the temporal representation of facial expression in face-selective regions

Study of Emotional Brain Mechanism Based on Functional Magnetic Resonance Imaging