Abstract:Emotion is a rapidly changing psychological and physical phenomenon. In daily life, people need to use the information of various sensory modalities (visual, auditory, etc.) to perceive non-verbal emotional information. Non-verbal emotion from faces and voices are often complex and varied. Previous studies have revealed that there are common and distinct neural networks underlying perception of human faces and voices. However, the neural mechanisms underlying visual and auditory emotional perception have not been well studied. Furthermore, despite researches on audiovisual integration of cross-modal emotional information, the multisensory cortex of the visual and auditory emotional information remains elusive. Therefore, it is necessary to study the similarities and differences between the neural mechanisms of emotion perception in visual and auditory modalities, and to explore the multiple sensory cortex of cross-modal emotion perception.The present fMRI (functional magnetic resonance imaging) study adopted a 2*3 (stimulus presentation modality: Visual, auditory; emotional valence: Happy, sad, fear) event-related design to investigate the neural mechanisms of emotion perception in visual and auditory modalities. When the stimulus (an emotional face or voice) was visually or aurally presented, participants were required to make a gender judgement. The results showed that the activation intensity of emotional faces in V1−V4, bilateral fusiform gyrus and bilateral superior temporal sulcus (STS) was significantly higher than that of emotional voices. Conversely, the activation intensity of emotional voices in auditory cortex (AC) was significantly higher than that of emotional faces. The results from multivoxel pattern analysis (MVPA) showed that the activation patterns of the right STS could discriminate the perception of human faces with emotional valence (happy, sad and fear face), indicating that the rSTG plays important role in perception of faces with different emotional valence; the activation patterns of the right FFA were different for happy and sad faces, indicating that the rFFA is crucial for positive and negative emotional face perception. A voxel-based whole brain analysis was further performed to examine the cortical areas that modulated perception of emotional valence. The whole brain analysis showed that the main effects for emotional valence was significant in the left opercular part of inferior frontal gyrus, indicating that this region might be a multisensory cortex of visual-auditory emotional perception. In summary, our study provided important evidence for further understanding the processing of emotion perception in different modalities and multisensory cortex of cross-channel emotion perception.

Neural correlates of perceived emotions in human insula and amygdala

Neural Correlates of Perceived Emotions in Human Insula and Amygdala for Auditory Emotion Recognition

The Roles of Subdivisions of Human Insula in Emotion Perception and Auditory Processing

Hearing Emotional Sounds: Category Representation in the Human Amygdala.

Emotion recognition from sound stimuli based on back-propagation neural networks and electroencephalograms

Insula uses overlapping codes for emotion in self and others

How Linguistic and Nonlinguistic Vocalizations Shape the Perception of Emotional Faces—An EEG Study

Disentangling emotional signals in the brain: an ALE meta-analysis of vocal affect perception

Neurocognitive Dynamics of Prosodic Salience over Semantics during Explicit and Implicit Processing of Basic Emotions in Spoken Words

Integrative Interaction of Emotional Speech in Audio-Visual Modality

Amygdala and insula contributions to dorsal-ventral pathway integration in the prosodic neural network

The Neural Processing of Vocal Emotion After Hearing Reconstruction in Prelingual Deaf Children: A Functional Near-Infrared Spectroscopy Brain Imaging Study

Neural Mechanisms of Social Emotion Perception: an EEG Hyper-Scanning Study

Similar and distinct neural mechanisms of visual and auditory emotion perception

Insular dichotomy in the implicit detection of emotions in human faces

Communication Channel, Emotion Category and Attentional Requirement Modulate Emotional Speech Perception: Electrophysiological and Behavioral Evidence

Weighted RSA: An Improved Framework on the Perception of Audio-visual Affective Speech in Left Insula and Superior Temporal Gyrus

Frequency Dependent Emotion Differentiation and Directional Coupling in Amygdala, Orbitofrontal and Medial Prefrontal Cortex Network with Intracranial Recordings

Hierarchical Integration of Interoception and Exteroception in the Anterior Insula During Naturalistic Emotional Experience

Perceived Communicative Context and Emotional Content Amplify Visual Word Processing in the Fusiform Gyrus

The Neural Network Underlying the Processing of Affective Vocalizations