Abstract:The role of attention in speech comprehension is not well understood. We used fMRI to study the neural correlates of auditory word, pseudoword, and nonspeech (spectrally rotated speech) perception during a bimodal (auditory, visual) selective attention task. In three conditions, Attend Auditory (ignore visual), Ignore Auditory (attend visual), and Visual (no auditory stimulation), 28 subjects performed a one-back matching task in the assigned attended modality. The visual task, attending to rapidly presented Japanese characters, was designed to be highly demanding in order to prevent attention to the simultaneously presented auditory stimuli. Regardless of stimulus type, attention to the auditory channel enhanced activation by the auditory stimuli (Attend Auditory>Ignore Auditory) in bilateral posterior superior temporal regions and left inferior frontal cortex. Across attentional conditions, there were main effects of speech processing (word+pseudoword>rotated speech) in left orbitofrontal cortex and several posterior right hemisphere regions, though these areas also showed strong interactions with attention (larger speech effects in the Attend Auditory than in the Ignore Auditory condition) and no significant speech effects in the Ignore Auditory condition. Several other regions, including the postcentral gyri, left supramarginal gyrus, and temporal lobes bilaterally, showed similar interactions due to the presence of speech effects only in the Attend Auditory condition. Main effects of lexicality (word>pseudoword) were isolated to a small region of the left lateral prefrontal cortex. Examination of this region showed significant word>pseudoword activation only in the Attend Auditory condition. Several other brain regions, including left ventromedial frontal lobe, left dorsal prefrontal cortex, and left middle temporal gyrus, showed Attention x Lexicality interactions due to the presence of lexical activation only in the Attend Auditory condition. These results support a model in which neutral speech presented in an unattended sensory channel undergoes relatively little processing beyond the early perceptual level. Specifically, processing of phonetic and lexical-semantic information appears to be very limited in such circumstances, consistent with prior behavioral studies.

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

Spatial alignment between faces and voices improves selective attention to audio-visual speech

Audiovisual angle and voice incongruence do not affect audiovisual verbal short-term memory in virtual reality

"May I Speak?": Multi-modal Attention Guidance in Social VR Group Conversations

The Influence of Multisensory Input On Voice Perception and Production Using Immersive Virtual Reality

Prediction and constraint in audiovisual speech perception

Acoustic scene complexity affects motion behavior during speech perception in audio-visual multi-talker virtual environments

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

Shared attention in virtual immersive reality enhances electrophysiological correlates of implicit sensory learning

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication

Visual Facial Enhancements Can Significantly Improve Speech Perception in the Presence of Noise

Accessible Nonverbal Cues to Support Conversations in VR for Blind and Low Vision People

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

On the Use of Multi-sensory Cues in Symmetric and Asymmetric Shared Collaborative Virtual Spaces

Attentional and linguistic interactions in speech perception

Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Multisensory integration of speech signals: the relationship between space and time

Incorporating Virtual Reality Agents During a Dichotic Speech Reception Task: Insights From the Heart

Auditory stimuli degrade visual performance in virtual reality

Perceptual Conflicts in a Multi-Stereoscopic Immersive Virtual Environment: Case Study on Face-to-Face Interaction through an Avatar