The Audio-Visual Integration and Pre-Preparation Effects in Emotion and Timing Discrimination Tasks
Zhang Zhao,Li Yuchen,Yi Xi,Luo Wenbo
DOI: https://doi.org/10.16719/j.cnki.1671-6981.20230101
2023-01-01
Abstract:Emotional audio-visual integration is an effective way to transmit and decode emotional information, which is of great significance to human survival and evolution. In multisensory integration, it is generally understood that one sensory signal promotes or alters the perception of other sensory signals. Studies have shown that the cross-sensory pre-preparation effect between visual and auditory is a key influencing factor of emotional audio-visual integration. However, it has not received much attention in the pre-preparation effect, particularly with regard to the emotional dimension.Moreover, previous research did not distinguish the emotional pre-preparation effect from the cross-sensory pre-preparation effect. Thus the present two studies started from the perspective of these two pre-preparation effects to gain insight into the emotional audio-visual integration processing mechanism. Experiment 1 adopted the time order judgment and simultaneous judgment tasks. Participants were asked to determine the chronological sequence of the visual and auditory stimuli, or whether they were presented at the same time. Experiment 2 adopted the emotion discrimination tasks. Participants were required to determine the emotional valence of visual, auditory, or audio-visual stimuli. The purpose of these two studies was to examine the influence of pre-preparation effects on emotional audio-visual integration either in the time or emotional cognitive dimensions. In Experiment 1, the response time of the visual-leading condition was significantly slower than that of the auditory-leading condition, and the accuracy was significantly lower than that of the auditory-leading condition. However, the amount of relative multisensory response enhancement（rMRE） for each condition of audio-visual integration was negative. It meant that audio-visual integration did not have many advantages compared with the single visual or auditory sensory. In Experiment 2, the rMRE of the negative emotion in auditory-leading condition was significantly greater than that in the visual-leading condition; The rMRE of the positive emotion in visual-leading condition was significantly greater than that in the auditory-leading condition. However, the difference between the visual-and the auditory-leading conditions on the rMRE of the neutral emotion was not significant.These research results showed that in the temporal dimension, the mechanism of the audio-visual integration was manifested as the feature matching processing across the visual and auditory, and a better pre-preparation effect for the presentation time was in auditory than in visual sensory. In the emotional cognitive dimension, the integration of the emotional information among the visual and auditory sensory occurred with the positive rMRE amount, and the better emotional preparation effect had been discovered in the visual sensory than in the auditory. Moreover, auditory channels played a greater role when integrating negative emotions, while visual channels dominated when integrating positive emotions. In summary, present experimental evidence showed that the enhancement of the emotional audiovisual integration was based on the cognitive processing of emotional information, and the temporal discrimination inhibited the emotional audiovisual integration. Either the pre-preparation effect of cross-sensory matching or emotional integration promoted the processing of visual and auditory information respectively and depended on the different neural circuits under visual-or auditory-leading independently. At last, the advantage of audio-visual integration showed that the cross-sensory and emotional pre-preparation effect was mediated by the tasks, demonstrating a separation between them.