Abstract:Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior research mostly utilized pictures and focused on the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences to evaluate target detection in terms of accuracy and response times. Two experimental tasks were developed using PsychoPy, where participants had to indicate whether a target (noun for Experiment 1, verb for Experiment 2) was present or absent. In trials without a target, target-related information was always present, either through one of the two senses (vision or audition; incongruent condition) or through both senses (congruent condition). We observed superior performance when the target was absent generally (M =93.9%, SD =0.04, M =84.2%, SD =0.182) compared to when it was present (M =82.3%, SD =0.143, M =73.7%, SD =0.184). Moreover, superior performance was noted in incongruent target-related movieclips, which significantly decreased during congruent target-related movieclips. In Experiment 1, we observed that in the audio condition when the target-related word was a noun, participant performance was superior compared to when it was a verb (M=99.4% vs. M=86.7%; t =-8.428, p=.001). In Experiment 2, the judgment scores were similarly high in incongruent movieclips and significantly lower in congruent ones regardless of whether the target-related information presented was a verb or noun. The present results provide evidence regarding the role of complexity of semantics, and especially the diverse role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the potential confusion that the complexity of semantic information can induce to perception experience.

Semantically Congruent Visual Stimuli Can Improve Auditory Memory

Semantic Congruency Improves Recognition Memory Performance for Both Audiovisual and Visual Stimuli

School-aged children can benefit from audiovisual semantic congruency during memory encoding

Semantically Congruent Audiovisual Integration with Modal-Based Attention Accelerates Auditory Short-Term Memory Retrieval.

Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval

The Effect of Semantic Congruence for Visual-Auditory Bimodal Stimuli

Semantically congruent bimodal presentation modulates cognitive control over attentional guidance by working memory

The Effects of Semantic Congruency: a Research of Audiovisual P300-speller.

The audiovisual competition effect induced by temporal asynchronous encoding weakened the visual dominance in working memory retrieval

Multisensory processing impacts memory for objects and their sources

Semantic congruency modulates the speed-up of multisensory responses

Visual Cues Contribute Differentially to Audiovisual Perception of Consonants and Vowels in Improving Recognition and Reducing Cognitive Demands in Listeners With Hearing Impairment Using Hearing Aids

Reproducibility And Discriminability Of Brain Patterns Of Semantic Categories Enhanced By Congruent Audiovisual Stimuli

The effect of semantic content on the perception of audiovisual movieclips

An Investigation of Cross-Modality Effects in Implicit and Explicit Memory

Multisensory benefits for speech recognition in noisy environments

The fidelity of visual and auditory memory

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

Effects of Semantic Congruency and Stimulus-Onset Asynchrony on Audiovisual Bimodal Stimuli Based BCI Speller

Semantic Bimodal Presentation Differentially Slows Working Memory Retrieval

A Novel Audiovisual P300-Speller Paradigm Based on Cross-Modal Spatial and Semantic Congruence