Abstract:Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior research mostly utilized pictures and focused on the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences to evaluate target detection in terms of accuracy and response times. Two experimental tasks were developed using PsychoPy, where participants had to indicate whether a target (noun for Experiment 1, verb for Experiment 2) was present or absent. In trials without a target, target-related information was always present, either through one of the two senses (vision or audition; incongruent condition) or through both senses (congruent condition). We observed superior performance when the target was absent generally (M =93.9%, SD =0.04, M =84.2%, SD =0.182) compared to when it was present (M =82.3%, SD =0.143, M =73.7%, SD =0.184). Moreover, superior performance was noted in incongruent target-related movieclips, which significantly decreased during congruent target-related movieclips. In Experiment 1, we observed that in the audio condition when the target-related word was a noun, participant performance was superior compared to when it was a verb (M=99.4% vs. M=86.7%; t =-8.428, p=.001). In Experiment 2, the judgment scores were similarly high in incongruent movieclips and significantly lower in congruent ones regardless of whether the target-related information presented was a verb or noun. The present results provide evidence regarding the role of complexity of semantics, and especially the diverse role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the potential confusion that the complexity of semantic information can induce to perception experience.

Event boundary perception in audio described films by people without sight

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Impacts on quality: Enjoyment factors in blind and low vision audience entertainment ratings: A qualitative study

Boundaries in the eyes: measure event segmentation during naturalistic video watching using eye tracking

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing

Insights From Mental Model Theory and Cognitive Narratology as a Tool for Content Selection in Audio Description

Audio-Visual Spatial Aligment Requirements of Central and Peripheral Object Events

Describe Now: User-Driven Audio Description for Blind and Low Vision Individuals

Audio describing the mental dimension of narrative characters. Insights from a Flemish case study.

Auditory areas are recruited for naturalistic visual meaning in early deaf people

Making Accessible Movies Easily: an Intelligent Tool for Authoring and Integrating Audio Descriptions to Movies

NowYouSee Me: Context-Aware Automatic Audio Description

The effect of Surround sound on embodiment and sense of presence in cinematic experience: a behavioral and HD-EEG study

"It's Kind of Context Dependent": Understanding Blind and Low Vision People's Video Accessibility Preferences Across Viewing Scenarios

The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience

AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description

Visual boundaries in sign motion: processing with and without lip-reading cues

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

Audio-visual interaction in visual motion detection: Synchrony versus Asynchrony

Audio Facilitates the Perception of Cinematic Continuity by First-Time Viewers

The effect of semantic content on the perception of audiovisual movieclips