Abstract:Emotion is a rapidly changing psychological and physical phenomenon. In daily life, people need to use the information of various sensory modalities (visual, auditory, etc.) to perceive non-verbal emotional information. Non-verbal emotion from faces and voices are often complex and varied. Previous studies have revealed that there are common and distinct neural networks underlying perception of human faces and voices. However, the neural mechanisms underlying visual and auditory emotional perception have not been well studied. Furthermore, despite researches on audiovisual integration of cross-modal emotional information, the multisensory cortex of the visual and auditory emotional information remains elusive. Therefore, it is necessary to study the similarities and differences between the neural mechanisms of emotion perception in visual and auditory modalities, and to explore the multiple sensory cortex of cross-modal emotion perception.The present fMRI (functional magnetic resonance imaging) study adopted a 2*3 (stimulus presentation modality: Visual, auditory; emotional valence: Happy, sad, fear) event-related design to investigate the neural mechanisms of emotion perception in visual and auditory modalities. When the stimulus (an emotional face or voice) was visually or aurally presented, participants were required to make a gender judgement. The results showed that the activation intensity of emotional faces in V1−V4, bilateral fusiform gyrus and bilateral superior temporal sulcus (STS) was significantly higher than that of emotional voices. Conversely, the activation intensity of emotional voices in auditory cortex (AC) was significantly higher than that of emotional faces. The results from multivoxel pattern analysis (MVPA) showed that the activation patterns of the right STS could discriminate the perception of human faces with emotional valence (happy, sad and fear face), indicating that the rSTG plays important role in perception of faces with different emotional valence; the activation patterns of the right FFA were different for happy and sad faces, indicating that the rFFA is crucial for positive and negative emotional face perception. A voxel-based whole brain analysis was further performed to examine the cortical areas that modulated perception of emotional valence. The whole brain analysis showed that the main effects for emotional valence was significant in the left opercular part of inferior frontal gyrus, indicating that this region might be a multisensory cortex of visual-auditory emotional perception. In summary, our study provided important evidence for further understanding the processing of emotion perception in different modalities and multisensory cortex of cross-channel emotion perception.

Perception of facial expressions involves emotion specific somatosensory cortex activations which are shaped by alexithymia

Distinct neural correlates of accuracy and bias in the perception of facial emotion expressions

Dissociable neural responses to facial expressions of sadness and anger

Specific brain processing of facial expressions in people with alexithymia: an H215O‐PET study

Somatosensory and motor contributions to emotion representation

Alexithymia modulates emotion concept activation during facial expression processing

Perceptive and affective impairments in emotive eye-region processing in alexithymia

Visual perception of facial expressions of emotions and alexithymia among people suffering from schizophrenia

Emotional face recognition, EMG response, and medial prefrontal activity in empathic behaviour

Facial Expression Enhances Emotion Perception Compared to Vocal Prosody: Behavioral and Fmri Studies

Similar and distinct neural mechanisms of visual and auditory emotion perception

Alexithymia predicts arousal-based processing deficits and discordance between emotion response systems during emotional imagery.

The role of the Somatosensory system in the feeling of emotions: a neurostimulation study

Neural Activities Associated with Emotion Recognition Observed in Men and Women.

The angry versus happy recognition advantage: the role of emotional and physical properties

The Recognition of Cross-Cultural Emotional Faces Is Affected by Intensity and Ethnicity in a Japanese Sample

Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities

Neural Substrates Underlying Intentional Empathy

Understanding Others' Feelings: The Role of the Right Primary Somatosensory Cortex in Encoding the Affective Valence of Others' Touch

Empathy in schizophrenia: neural alterations during emotion recognition and affective sharing

The prefrontal cortex conscious and unconscious response to social/emotional facial expressions involve sex, hemispheric laterality, and selective activation of the central cardiac modulation