Abstract:Abstract The aim of this study was (1) to provide behavioral evidence for multimodal feature integration in an object recognition task in humans and (2) to characterize the processing stages and the neural structures where multisensory interactions take place. Event-related potentials (ERPs) were recorded from 30 scalp electrodes while subjects performed a forced-choice reaction-time categorization task: At each trial, the subjects had to indicate which of two objects was presented by pressing one of two keys. The two objects were defined by auditory features alone, visual features alone, or the combination of auditory and visual features. Subjects were more accurate and rapid at identifying multimodal than unimodal objects. Spatiotemporal analysis of ERPs and scalp current densities revealed several auditory-visual interaction components temporally, spatially, and functionally distinct before 200 msec poststimulus. The effects observed were (1) in visual areas, new neural activities (as early as 40 msec poststimulus) and modulation (amplitude decrease) of the N185 wave to unimodal visual stimulus, (2) in the auditory cortex, modulation (amplitude increase) of subcomponents of the unimodal auditory N1 wave around 90 to 110 msec, and (3) new neural activity over the right fronto-temporal area (140 to 165 msec). Furthermore, when the subjects were separated into two groups according to their dominant modality to perform the task in unimodal conditions (shortest reaction time criteria), the integration effects were found to be similar for the two groups over the nonspecific fronto-temporal areas, but they clearly differed in the sensory-specific cortices, affecting predominantly the sensory areas of the nondominant modality. Taken together, the results indicate that multisensory integration is mediated by flexible, highly adaptive physiological processes that can take place very early in the sensory processing chain and operate in both sensory-specific and nonspecific cortical structures in different ways.

Bi-modal Enhancement Effect from Combination of Visual and Auditory Stimulus

The Bi-modal Enhancement Effect Between Visual and Somatosensory Stimulus

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Combining Motor Imagery with Selective Sensation Toward a Hybrid-Modality BCI

Crossmodal Modulation As a Basis for Visual Enhancement of Auditory Performance

The Effect of Semantic Congruence for Visual-Auditory Bimodal Stimuli

Effects of Temporal Congruity Between Auditory and Visual Stimuli Using Rapid Audio-Visual Serial Presentation

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

Influence of Attention on Bimodal Integration During Emotional Change Decoding: ERP Evidence.

Auditory change detection and visual selective attention: association between MMN and N2pc

Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study

Cross-modal Enhancement of Spatially Unpredictable Visual Target Discrimination During the Attentional Blink

Research on Cross-sensory Information Enhancement Effect Based on the Analysis of Multi-mode Brain Imaging

Spatio-temporal measures of electrophysiological correlates for behavioral multisensory enhancement during visual, auditory and somatosensory stimulation: A behavioral and ERP study

The interplay between audiovisual temporal synchrony and semantic congruency in the cross-modal boost of the visual target discrimination during the attentional blink

Cross-Modal Selective Attention to Visual and Auditory Stimuli Modulates Endogenous Erp Components

A Brain-Computer Interface Based on Multi-Modal Attention

The Effects of Bilateral and Ipsilateral Auditory Stimuli on the Subcomponents of Visual Attention

Semantically congruent bimodal presentation modulates cognitive control over attentional guidance by working memory

Semantically Congruent Audiovisual Integration with Modal-Based Attention Accelerates Auditory Short-Term Memory Retrieval.

The Effect of Stimulus-Onset Asynchrony on Audiovisual Brain-Computer Interface System