Abstract:Lip-reading is crucial for understanding speech in challenging conditions. But how the brain extracts meaning from, silent, visual speech is still under debate. Lip-reading in silence activates the auditory cortices, but it is not known whether such activation reflects immediate synthesis of the corresponding auditory stimulus or imagery of unrelated sounds. To disentangle these possibilities, we used magnetoencephalography to evaluate how cortical activity in 28 healthy adult humans (17 females) entrained to the auditory speech envelope and lip movements (mouth opening) when listening to a spoken story without visual input (audio-only), and when seeing a silent video of a speaker articulating another story (video-only). In video-only, auditory cortical activity entrained to the absent auditory signal at frequencies <1 Hz more than to the seen lip movements. This entrainment process was characterized by an auditory-speech-to-brain delay of ~70 ms in the left hemisphere, compared with ~20 ms in audio-only. Entrainment to mouth opening was found in the right angular gyrus at <1 Hz, and in early visual cortices at 1–8 Hz. These findings demonstrate that the brain can use a silent lip-read signal to synthesize a coarse-grained auditory speech representation in early auditory cortices. Our data indicate the following underlying oscillatory mechanism: seeing lip movements first modulates neuronal activity in early visual cortices at frequencies that match articulatory lip movements; the right angular gyrus then extracts slower features of lip movements, mapping them onto the corresponding speech sound features; this information is fed to auditory cortices, most likely facilitating speech parsing. SIGNIFICANCE STATEMENT Lip-reading consists in decoding speech based on visual information derived from observation of a speaker's articulatory facial gestures. Lip-reading is known to improve auditory speech understanding, especially when speech is degraded. Interestingly, lip-reading in silence still activates the auditory cortices, even when participants do not know what the absent auditory signal should be. However, it was uncertain what such activation reflected. Here, using magnetoencephalographic recordings, we demonstrate that it reflects fast synthesis of the auditory stimulus rather than mental imagery of unrelated, speech or non-speech, sounds. Our results also shed light on the oscillatory dynamics underlying lip-reading.

How silent is silent reading? Intracerebral evidence for top-down activation of temporal voice areas during reading

Silent music reading: auditory imagery and visuotonal modality transfer in singers and non-singers

Processing Speech and Thoughts during Silent Reading: Direct Reference Effects for Speech by Fictional Characters in Voice-Selective Auditory Cortex and a Theory-of-Mind Network

Spatio-Temporal Progression of Cortical Activity Related to Continuous Overt and Covert Speech Production in a Reading Task

Hemodynamics of the left cerebral hemisphere during silent reading:analysis using near-infrared spectroscopy

Revealing the co-existence of written and spoken language coding neural populations in the left-ventral occipitotemporal cortex

Selective attention to human voice enhances brain activity bilaterally in the superior temporal sulcus

Examining the Neural Markers of Speech Rhythm in Silent Reading Using Mass Univariate Statistics of EEG Single Trials

Lip-Reading Enables the Brain to Synthesize Auditory Features of Unknown Silent Speech

How reading differs from object naming at the neuronal level

Reading the mind's eye: Online detection of visuo-spatial working memory and visual imagery in the inferior temporal lobe

Bilateral Prefrontal Cortex Blood Flow Dynamics during Silent and Oral Reading Using Near-Infrared Spectroscopy

Auditory cortex encodes lipreading information through spatially distributed activity

Will you read how I will read? Naturalistic fMRI predictors of emergent reading

Intracranial Mapping of Response Latencies and Task Effects for Spoken Syllable Processing in the Human Brain

Differential weighting of information during aloud and silent reading: Evidence from representational similarity analysis of fMRI data

Revealing Spatiotemporal Brain Dynamics Of Speech Production Based On Eeg And Eye Movement

Cortical cognitive processing during reading captured using functional-near infrared spectroscopy

Dissociable roles of neural pattern reactivation and transformation during recognition of words read aloud and silently: An MVPA study of the production effect

Modulation of Brain Activity by Selective Attention to Audiovisual Dialogues

Subtitled speech: the neural mechanisms of ticker-tape synaesthesia