Abstract:Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.

Incidental Self-Processing Modulates the Interaction of Emotional Valence and Arousal.

The neural correlates of incidental self-processing induced by handwritten negative words

The Temporal Features of Self-referential Processing Evoked by Chinese Handwriting

Self-esteem Modulates Dorsal Medial Prefrontal Cortical Response to Self-Positivity Bias in Implicit Self-Relevant Processing.

Cortical Mechanisms Involved in Selective Interactions of Negative Emotion with Spatial and Verbal Working Memory: an Fmri Study

How distinct functional insular subdivisions mediate interacting neurocognitive systems.

The Role of Mid-Insula in the Relationship Between Cardiac Interoceptive Attention and Anxiety: Evidence from an Fmri Study

Spatiotemporal Cortical Activation Underlying Self-Referencial Processing Evoked by Self-Hand

Self-consciousness mediated the role of the insula in self-disclosure: Evidence from rs-fMRI data

How processing emotion affects language control in bilinguals

Emotion Perception and Executive Control Interact in the Salience Network During Emotionally Charged Working Memory Processing.

From affect to control: Functional specialization of the insula in motivation and regulation

Separate neural networks of implicit emotional processing between pictures and words: A coordinate-based meta-analysis of brain imaging studies

Self-referential Processing in Unipolar Depression: Distinct Roles of Subregions of the Medial Prefrontal Cortex

Integrative Interaction of Emotional Speech in Audio-Visual Modality

Neural Substrates of the Interaction of Emotional Stimulus Processing and Motor Inhibitory Control: an Emotional Linguistic Go/no-Go Fmri Study

Intrinsic Functional Brain Connectivity Patterns Underlying Enhanced Interoceptive Sensibility.

The Neural Mechanisms of Explicit and Implicit Processing of Chinese Emotion-Label and Emotion-Laden Words: Evidence from Emotional Categorisation and Emotional Stroop Tasks

Dissociation of Writing Processes: A Functional Magnetic Resonance Imaging Study on the Neural Substrates for the Handwritten Production of Chinese Characters

Self-esteem Modulates Dorsal Anterior Cingulate Cortical Response in Self-Referential Processing

Anterior insula and dorsal anterior cingulate cortex as a hub of self-regulation: combining activation likelihood estimation meta-analysis and meta-analytic connectivity modeling analysis