Abstract:Abstract Stimuli associated with high reward modulate perception and such value-driven effects have been shown to originate from the modulation of the earliest stages of sensory processing in the brain. In natural environments objects comprise multiple features (imagine a rolling soccer ball, with its black and white patches and the swishing sound made during its motion), where each feature may signal different associations with previously encountered rewards. How perception of such an object is affected by the value associations of its constituent parts is unknown. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of visual perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual orientation discrimination task in the presence of previously rewarded visual or auditory cues (intra- and cross-modal cues, respectively) that were concurrently presented with the target stimulus. During the conditioning phase, when reward associations were learned and reward cues were the target of the task, reward value of both modalities enhanced the electrophysiological correlates of sensory processing in visual cortex. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value-enhanced behavioral measures of visual sensitivity whereas intra-modal value led to a trend for suppression. A similar pattern of modulations was found in the simultaneously recorded event-related potentials (ERPs) of posterior electrodes. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal cues led to a later value-driven modulation, with an enhancement of response positivity for high-compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses of the posterior electrodes. These results indicate that visual cortex is modulated by the reward value of visual as well as auditory cues. Previously rewarded, task-irrelevant cues from the same or different sensory modality have a different effect on visual perception, as intra-modal high-value cues may interfere with the target processing, whereas cross-modal high-value cues boost the perception of the target.

Crossmodal visual predictions elicit spatially specific early visual cortex activity but later than real visual stimuli

Salient Sounds Activate Human Visual Cortex Automatically

Multisensory representations of space and time in sensory cortices

Sounds activate visual cortex and improve visual discrimination.

Modulation of early auditory processing by visual information: Prediction or bimodal integration?

Differential effects of intra-modal and cross-modal reward value on visual perception: ERP evidence

Spatiotemporal dynamics across visual cortical laminae support a predictive coding framework for interpreting mismatch responses

The Time Course Of Spatial And Semantic Processing Between Audition And Vision: Evidence From Event-Related Potentials

Selective attention to sound features mediates cross-modal activation of visual cortices

Onset timing of letter processing in auditory and visual sensory cortices

Pre-stimulus alpha oscillations encode stimulus-specific visual predictions

Electrical neuroimaging evidence that spatial frequency-based selective attention affects V1 activity as early as 40-60 ms in humans

Intra- and Cross-Modal Cuing of Visual Spatial Attention: the Hyper-Effective Simultaneous Auditory Peripheral Cues

The impact of crossmodal predictions on the neural processing of aesthetic stimuli

Spatial and Semantic Processing Between Audition and Vision: an Event-Related Potential Study

Cross-modal Representations in Early Visual and Auditory Cortices Revealed by Multi-Voxel Pattern Analysis.

Cross-Modal Selective Attention to Visual and Auditory Stimuli Modulates Endogenous Erp Components

Cross-modal Orienting of Visual Attention.

Visual omitted stimulus potentials are not retinotopic

High-level visual prediction errors in early visual cortex

Suppressed Sensory Response to Predictable Object Stimuli throughout the Ventral Visual Stream