Abstract:Most perceptual decisions rely on the active acquisition of evidence from the environment involving stimulation from multiple senses. However, our understanding of the neural mechanisms underlying this process is limited. Crucially, it remains elusive how different sensory representations interact in the formation of perceptual decisions. To answer these questions, we used an active sensing paradigm coupled with neuroimaging, multivariate analysis, and computational modeling to probe how the human brain processes multisensory information to make perceptual judgments. Participants of both sexes actively sensed to discriminate two texture stimuli using visual (V) or haptic (H) information or the two sensory cues together (VH). Crucially, information acquisition was under the participants' control, who could choose where to sample information from and for how long on each trial. To understand the neural underpinnings of this process, we first characterized where and when active sensory experience (movement patterns) is encoded in human brain activity (EEG) in the three sensory conditions. Then, to offer a neurocomputational account of active multisensory decision formation, we used these neural representations of active sensing to inform a drift diffusion model of decision-making behavior. This revealed a multisensory enhancement of the neural representation of active sensing, which led to faster and more accurate multisensory decisions. We then dissected the interactions between the V, H, and VH representations using a novel information-theoretic methodology. Ultimately, we identified a synergistic neural interaction between the two unisensory (V, H) representations over contralateral somatosensory and motor locations that predicted multisensory (VH) decision-making performance. SIGNIFICANCE STATEMENT In real-world settings, perceptual decisions are made during active behaviors, such as crossing the road on a rainy night, and include information from different senses (e.g., car lights, slippery ground). Critically, it remains largely unknown how sensory evidence is combined and translated into perceptual decisions in such active scenarios. Here we address this knowledge gap. First, we show that the simultaneous exploration of information across senses (multi-sensing) enhances the neural encoding of active sensing movements. Second, the neural representation of active sensing modulates the evidence available for decision; and importantly, multi-sensing yields faster evidence accumulation. Finally, we identify a cross-modal interaction in the human brain that correlates with multisensory performance, constituting a putative neural mechanism for forging active multisensory perception.

The Dynamic Multisensory Engram: Neural Circuitry Underlying Crossmodal Object Recognition in Rats Changes with the Nature of Object Experience

Experience transforms crossmodal object representations in the anterior temporal lobes

Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study

Neuronal dynamics supporting formation and recombination of cross-modal olfactory-tactile association in the rat hippocampal formation

Posterior parietal cortex represents sensory history and mediates its effects on behaviour

Multisensory-Guided Associative Learning Enhances Multisensory Representation in Primary Auditory Cortex.

Multisensory learning binds neurons into a cross-modal memory engram

Severe cross-modal object recognition deficits in rats treated sub-chronically with NMDA receptor antagonists are reversed by systemic nicotine: implications for abnormal multisensory integration in schizophrenia

Subpopulations of neurons in the perirhinal cortex enable both modality-specific and modality-invariant recognition of objects

Task and Behavior-Related Variables Are Encoded by the Postrhinal and Medial Entorhinal Cortex During Non-Spatial Associative Learning

Neural correlates of object identity and reward outcome in the sensory cortical-hippocampal hierarchy: coding of motivational information in perirhinal cortex

Auditory Cortex Learns to Discriminate Audiovisual Cues through Selective Multisensory Enhancement

Neural Encoding of Active Multi-Sensing Enhances Perceptual Decision-Making via a Synergistic Cross-Modal Interaction

Cross-Modal Competition: the Default Computation for Multisensory Processing

Choice-dependent cross-modal interaction in the medial prefrontal cortex of rats

Auditory, Visual, and Cross-Modal Mismatch Negativities in the Rat Auditory and Visual Cortices

The brain can develop conflicting multisensory principles to guide behavior

Backward Signal from Medial Temporal Lobe in Neural Circuit Reorganization of Primate Inferotemporal Cortex

Multisensory integration in rodent tactile but not visual thalamus

Implicit multisensory associations influence voice recognition

Perirhinal cortex supports object perception by integrating over visuospatial sequences