Abstract:While interacting with the world our senses and nervous system are constantly challenged to identify the origin and coherence of sensory input signals of various intensities. This problem becomes apparent when stimuli from different modalities need to be combined, e.g., to find out whether an auditory stimulus and a visual stimulus belong to the same object. To cope with this problem, humans and most other animal species are equipped with complex neural circuits to enable fast and reliable combination of signals from various sensory organs. This multisensory integration starts in the brain stem to facilitate unconscious reflexes and continues on ascending pathways to cortical areas for further processing. To investigate the underlying mechanisms in detail, we developed a canonical neural network model for multisensory integration that resembles neurophysiological findings. For example, the model comprises multisensory integration neurons that receive excitatory and inhibitory inputs from unimodal auditory and visual neurons, respectively, as well as feedback from cortex. Such feedback projections facilitate multisensory response enhancement and lead to the commonly observed inverse effectiveness of neural activity in multisensory neurons. Two versions of the model are implemented, a rate-based neural network model for qualitative analysis and a variant that employs spiking neurons for deployment on a neuromorphic processing. This dual approach allows to create an evaluation environment with the ability to test model performances with real world inputs. As a platform for deployment we chose IBM's neurosynaptic chip TrueNorth. Behavioral studies in humans indicate that temporal and spatial offsets as well as reliability of stimuli are critical parameters for integrating signals from different modalities. The model reproduces such behavior in experiments with different sets of stimuli. In particular, model performance for stimuli with varying spatial offset is tested. In addition, we demonstrate that due to the emergent properties of network dynamics model performance is close to optimal Bayesian inference for integration of multimodal sensory signals. Furthermore, the implementation of the model on a neuromorphic processing chip enables a complete neuromorphic processing cascade from sensory perception to multisensory integration and the evaluation of model performance for real world inputs.

Development of a Bayesian Estimator for Audio-Visual Integration: A Neurocomputational Study

Spatiotemporal models for multisensory integration

Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception

From Near-Optimal Bayesian Integration to Neuromorphic Hardware: A Neural Network Model of Multisensory Integration

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

To integrate or not to integrate: Temporal dynamics of hierarchical Bayesian causal inference

Neural Integration of Audiovisual Sensory Inputs in Macaque Amygdala and Adjacent Regions

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Reciprocally Coupled Local Estimators Implement Bayesian Information Integration Distributively.

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Closing the loop on multisensory interactions: A neural architecture for multisensory causal inference and recalibration

Unveiling Dynamic System Strategies for Multisensory Processing: from Neuronal Fixed-Criterion Integration to Population Bayesian Inference

Bayesian Prior Uncertainty and Surprisal Elicit Distinct Neural Patterns During Sound Localization in Dynamic Environments

Multisensory integration across reference frames with additive feed-forward networks

Bayesian inference by visuomotor neurons in prefrontal cortex

Audio-visual localization with hierarchical topographic maps: Modeling the superior colliculus

Multisensory integration operates on correlated input from unimodal transients channels

Visual-Based Spatial Coordinate Dominates Probabilistic Multisensory Inference in Macaque MST-d Disparity Encoding

Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study

Neural Processing of Naturalistic Audiovisual Events in Space and Time

Perceptual Stimulus — A Bayesian-based Integration of Multi-Visual-cue Approach and Its Application