Abstract:Purpose Emotional speech communication involves multisensory integration of linguistic (e.g., semantic content) and paralinguistic (e.g., prosody and facial expressions) messages. Previous studies on linguistic versus paralinguistic salience effects in emotional speech processing have produced inconsistent findings. In this study, we investigated the relative perceptual saliency of emotion cues in cross-channel auditory alone task (i.e., semantics-prosody Stroop task) and cross-modal audiovisual task (i.e., semantics-prosody-face Stroop task). Method Thirty normal Chinese adults participated in two Stroop experiments with spoken emotion adjectives in Mandarin Chinese. Experiment 1 manipulated auditory pairing of emotional prosody (happy or sad) and lexical semantic content in congruent and incongruent conditions. Experiment 2 extended the protocol to cross-modal integration by introducing visual facial expression during auditory stimulus presentation. Participants were asked to judge emotional information for each test trial according to the instruction of selective attention. Results Accuracy and reaction time data indicated that, despite an increase in cognitive demand and task complexity in Experiment 2, prosody was consistently more salient than semantic content for emotion word processing and did not take precedence over facial expression. While congruent stimuli enhanced performance in both experiments, the facilitatory effect was smaller in Experiment 2. Conclusion Together, the results demonstrate the salient role of paralinguistic prosodic cues in emotion word processing and congruence facilitation effect in multisensory integration. Our study contributes tonal language data on how linguistic and paralinguistic messages converge in multisensory speech processing and lays a foundation for further exploring the brain mechanisms of cross-channel/modal emotion integration with potential clinical applications.

Communication Channel, Emotion Category and Attentional Requirement Modulate Emotional Speech Perception: Electrophysiological and Behavioral Evidence

Gender Differences in the Processing of Standard Emotional Visual Stimuli: Integrating ERP and Fmri Results

Neurocognitive Dynamics of Prosodic Salience over Semantics during Explicit and Implicit Processing of Basic Emotions in Spoken Words

Gender Differences in Identifying Facial, Prosodic, and Semantic Emotions Show Category- and Channel-Specific Effects Mediated by Encoder's Gender

Prosody Dominates over Semantics in Emotion Word Processing: Evidence from Cross-Channel and Cross-Modal Stroop Effects

Sex Differences in Processing Emotional Speech Prosody: Preliminary Findings from a Multi-Feature Oddball Study

Effects of Channel Dominance and Gender Differences on Impaired Emotion Perception in Schizophrenic Patients

Effects of communication channels and actor's gender on emotion identification by native Mandarin speakers

Affective Prosody Guides Facial Emotion Processing

Influence of Attention on Bimodal Integration During Emotional Change Decoding: ERP Evidence.

Unisensory and Multisensory Stroop Effects Modulate Gender Differences in Verbal and Nonverbal Emotion Perception.

Investigating Duration Effects of Emotional Speech Stimuli in a Tonal Language by Using Event-Related Potentials

Emotional Intensity Modulates Vocal Emotion Decoding in a Late Stage of Processing: an Event-Related Potential Study.

Impaired Emotion Perception in Schizophrenia Shows Sex Differences with Channel- and Category-Specific Effects: A Pilot Study

Age and Sex Differences in Emotion Perception Are Influenced by Emotional Category and Communication Channel.

Category-Sensitive Age-Related Shifts Between Prosodic and Semantic Dominance in Emotion Perception Linked to Cognitive Capacities

How Linguistic and Nonlinguistic Vocalizations Shape the Perception of Emotional Faces—An EEG Study

Evaluating the Relative Perceptual Salience of Linguistic and Emotional Prosody in Quiet and Noisy Contexts

Emotional Prosody Processing Between Artificial Voices and Human Voices

The Neural Mechanisms of Explicit and Implicit Processing of Chinese Emotion-Label and Emotion-Laden Words: Evidence from Emotional Categorisation and Emotional Stroop Tasks

When Brain Differentiates Happy from Neutral in Prosody?