Abstract:Emotional audio-visual integration is an effective way to transmit and decode emotional information, which is of great significance to human survival and evolution. In multisensory integration, it is generally understood that one sensory signal promotes or alters the perception of other sensory signals. Studies have shown that the cross-sensory pre-preparation effect between visual and auditory is a key influencing factor of emotional audio-visual integration. However, it has not received much attention in the pre-preparation effect, particularly with regard to the emotional dimension.Moreover, previous research did not distinguish the emotional pre-preparation effect from the cross-sensory pre-preparation effect. Thus the present two studies started from the perspective of these two pre-preparation effects to gain insight into the emotional audio-visual integration processing mechanism. Experiment 1 adopted the time order judgment and simultaneous judgment tasks. Participants were asked to determine the chronological sequence of the visual and auditory stimuli, or whether they were presented at the same time. Experiment 2 adopted the emotion discrimination tasks. Participants were required to determine the emotional valence of visual, auditory, or audio-visual stimuli. The purpose of these two studies was to examine the influence of pre-preparation effects on emotional audio-visual integration either in the time or emotional cognitive dimensions. In Experiment 1, the response time of the visual-leading condition was significantly slower than that of the auditory-leading condition, and the accuracy was significantly lower than that of the auditory-leading condition. However, the amount of relative multisensory response enhancement（rMRE） for each condition of audio-visual integration was negative. It meant that audio-visual integration did not have many advantages compared with the single visual or auditory sensory. In Experiment 2, the rMRE of the negative emotion in auditory-leading condition was significantly greater than that in the visual-leading condition; The rMRE of the positive emotion in visual-leading condition was significantly greater than that in the auditory-leading condition. However, the difference between the visual-and the auditory-leading conditions on the rMRE of the neutral emotion was not significant.These research results showed that in the temporal dimension, the mechanism of the audio-visual integration was manifested as the feature matching processing across the visual and auditory, and a better pre-preparation effect for the presentation time was in auditory than in visual sensory. In the emotional cognitive dimension, the integration of the emotional information among the visual and auditory sensory occurred with the positive rMRE amount, and the better emotional preparation effect had been discovered in the visual sensory than in the auditory. Moreover, auditory channels played a greater role when integrating negative emotions, while visual channels dominated when integrating positive emotions. In summary, present experimental evidence showed that the enhancement of the emotional audiovisual integration was based on the cognitive processing of emotional information, and the temporal discrimination inhibited the emotional audiovisual integration. Either the pre-preparation effect of cross-sensory matching or emotional integration promoted the processing of visual and auditory information respectively and depended on the different neural circuits under visual-or auditory-leading independently. At last, the advantage of audio-visual integration showed that the cross-sensory and emotional pre-preparation effect was mediated by the tasks, demonstrating a separation between them.

The audio-visual integration effect on music emotion: Behavioral and physiological evidence

The Audio-Visual Integration and Pre-Preparation Effects in Emotion and Timing Discrimination Tasks

An Eye Tracking Study: Positive Emotional Interface Design Facilitates Learning Outcomes in Multimedia Learning?

The Behavioral Presentations of Audio-Visual Integration in Rats

The Influence of the Cross-Modal Emotional Pre-Preparation Effect on Audiovisual Integration

The role of audiovisual congruence in aesthetic appreciation of contemporary music and visual art

The Integration of Facial and Vocal Cues During Emotional Change Perception: EEG Markers.

Intervention Effect of Color and Sound Cross-Modal Correspondence Between Interaction of Emotion and Ambient.

The effect of dichotic music presentation on ratings of emotional facial expressions

Crossmodal Integration Enhances Neural Representation of Task-Relevant Features in Audiovisual Face Perception

Effects of the Musical Sound Environment on Communicating Emotion

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention.

Dissonant music engages early visual processing

Music-Induced Mood Modulates the Strength of Emotional Negativity Bias: an Erp Study

The Contribution Of Sound Intensity In Vocal Emotion Perception: Behavioral And Electrophysiological Evidence

Creativity and Perception: Unveiling the Role of Cross‐Modal Audiovisual Integration

The effect of music background on the emotional appraisal of film sequences

Audiovisual Integration of Natural Auditory and Visual Stimuli in the Real-World Situation

Influence of Attention on Bimodal Integration During Emotional Change Decoding: ERP Evidence.

Do emotions evoked by music modulate visuospatial working memory capacity? A physiological study

Listening to music modulates visual perception of paintings during binocular rivalry