Abstract:Our brain perceives the world by exploiting multiple sensory modalities to extract information about various aspects of external stimuli. If these sensory cues are from the same stimulus of interest, they should be integrated to improve perception; otherwise, they should be segregated to distinguish different stimuli. In reality, however, the brain faces the challenge of recognizing stimuli without knowing in advance whether sensory cues come from the same or different stimuli. To address this challenge and to recognize stimuli rapidly, we argue that the brain should carry out multisensory integration and segregation concurrently with complementary neuron groups. Studying an example of inferring heading-direction via visual and vestibular cues, we develop a concurrent multisensory processing neural model which consists of two reciprocally connected modules, the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area (VIP), and that at each module, there exists two distinguishing groups of neurons, congruent and opposite neurons. Specifically, congruent neurons implement cue integration, while opposite neurons compute the cue disparity, both optimally as described by Bayesian inference. The two groups of neurons provide complementary information which enables the neural system to assess the validity of cue integration and, if necessary, to recover the lost information associated with individual cues without re-gathering new inputs. Through this process, the brain achieves rapid stimulus perception if the cues come from the same stimulus of interest, and differentiates and recognizes stimuli based on individual cues with little time delay if the cues come from different stimuli of interest. Our study unveils the indispensable role of opposite neurons in multisensory processing and sheds light on our understanding of how the brain achieves multisensory processing efficiently and rapidly. Significance Statement Our brain perceives the world by exploiting multiple sensory cues. These cues need to be integrated to improve perception if they come from the same stimulus and otherwise be segregated. To address the challenge of recognizing whether sensory cues come from the same or different stimuli that are unknown in advance, we propose that the brain should carry out multisensory integration and segregation concurrently with two different neuron groups. Specifically, congruent neurons implement cue integration, while opposite neurons compute the cue disparity, and the interplay between them achieves rapid stimulus recognition without information loss. We apply our model to the example of inferring heading-direction based on visual and vestibular cues and reproduce the experimental data successfully.

Congruence and Simultaneity Judgment with a Single Multisensory Target: Preliminary Results

Neural Processes Underlying the "Same"-"Different'' Judgment of Two Simultaneously Presented Objects- an Eeg Study

Semantic Incongruency Interferes with Endogenous Attention in Cross-Modal Integration of Semantically Congruent Objects

Effect of Target Semantic Consistency in Different Sequence Positions and Processing Modes on T2 Recognition: Integration and Suppression Based on Cross-Modal Processing

Do Congruent Auditory Stimuli Facilitate Visual Search in Dynamic Environments? an Experimental Study Based on Multisensory Interaction.

The Effect of Spatial Consistence on Visual Detection and Discretion

Concurrent Multisensory Integration and Segregation with Complementary Congruent and Opposite Neurons

Does cross-modal correspondence modulate modality-specific perceptual processing? Study using timing judgment tasks

Perception of temporal synchrony not a prerequisite for multisensory integration

Locating the Sources for Cross-Modal Interactions and Decision Making During Judging the Visual-Affected Auditory Intensity Change.

Audiovisual Congruency and Incongruency Effects on Auditory Intensity Discrimination.

Synesthetic Correspondence: An Overview

On Mechanism of the Same-Different Judgment Based on Dimension Switch

The interplay between audiovisual temporal synchrony and semantic congruency in the cross-modal boost of the visual target discrimination during the attentional blink

Multiple Concurrent Temporal Recalibrations Driven by Audiovisual Stimuli with Apparent Physical Differences

Semantically Congruent Audiovisual Integration with Modal-Based Attention Accelerates Auditory Short-Term Memory Retrieval.

Quantified Acoustic–optical Speech Signal Incongruity Identifies Cortical Sites of Audiovisual Speech Processing

The recalibration patterns of perceptual synchrony and multisensory integration after exposure to asynchronous speech.

Multisensory Information Facilitates the Categorization of Untrained Stimuli.

Olfactory-gustatory simultaneity judgments: A preliminary study on the congruency-dependent temporal window of multisensory binding

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink