Abstract:Most perceptual decisions rely on the active acquisition of evidence from the environment involving stimulation from multiple senses. However, our understanding of the neural mechanisms underlying this process is limited. Crucially, it remains elusive how different sensory representations interact in the formation of perceptual decisions. To answer these questions, we used an active sensing paradigm coupled with neuroimaging, multivariate analysis, and computational modeling to probe how the human brain processes multisensory information to make perceptual judgments. Participants of both sexes actively sensed to discriminate two texture stimuli using visual (V) or haptic (H) information or the two sensory cues together (VH). Crucially, information acquisition was under the participants' control, who could choose where to sample information from and for how long on each trial. To understand the neural underpinnings of this process, we first characterized where and when active sensory experience (movement patterns) is encoded in human brain activity (EEG) in the three sensory conditions. Then, to offer a neurocomputational account of active multisensory decision formation, we used these neural representations of active sensing to inform a drift diffusion model of decision-making behavior. This revealed a multisensory enhancement of the neural representation of active sensing, which led to faster and more accurate multisensory decisions. We then dissected the interactions between the V, H, and VH representations using a novel information-theoretic methodology. Ultimately, we identified a synergistic neural interaction between the two unisensory (V, H) representations over contralateral somatosensory and motor locations that predicted multisensory (VH) decision-making performance. SIGNIFICANCE STATEMENT In real-world settings, perceptual decisions are made during active behaviors, such as crossing the road on a rainy night, and include information from different senses (e.g., car lights, slippery ground). Critically, it remains largely unknown how sensory evidence is combined and translated into perceptual decisions in such active scenarios. Here we address this knowledge gap. First, we show that the simultaneous exploration of information across senses (multi-sensing) enhances the neural encoding of active sensing movements. Second, the neural representation of active sensing modulates the evidence available for decision; and importantly, multi-sensing yields faster evidence accumulation. Finally, we identify a cross-modal interaction in the human brain that correlates with multisensory performance, constituting a putative neural mechanism for forging active multisensory perception.

Unravelling the multisensory learning advantage: Different patterns of within and across frequency-specific interactions drive uni- and multisensory neuroplasticity

Closing the loop on multisensory interactions: A neural architecture for multisensory causal inference and recalibration

Multisensory integration and cross-modal learning in synaesthesia: A unifying model

Neural Encoding of Active Multi-Sensing Enhances Perceptual Decision-Making via a Synergistic Cross-Modal Interaction

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex

Enriched learning: behavior, brain, and computation

Multi-timescale neural dynamics for multisensory integration

The brain can develop conflicting multisensory principles to guide behavior

Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human-Computer Interaction

Critical Learning Periods for Multisensory Integration in Deep Networks

Anticipating multisensory environments: Evidence for a supra-modal predictive system

Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception

Multisensory integration in the mouse cortical connectome using a network diffusion model

Unconscious multisensory integration: behavioral and neural evidence from subliminal stimuli

Experience transforms crossmodal object representations in the anterior temporal lobes

Long-horizon associative learning explains human sensitivity to statistical and network structures in auditory sequences

Antagonistic Cross-Links and Reciprocal Couplings Emerge in Optimal Multisensory Integration

Research on Cross-sensory Information Enhancement Effect Based on the Analysis of Multi-mode Brain Imaging

Parallel and converging multisensory cascades in the Drosophila connectome

Functional gradients through the cortex, multisensory integration and scaling laws in brain dynamics