Abstract:For many deaf people, lip-reading plays a major role in verbal communication. However, lip movements are by nature ambiguous, so that lip reading does not allow for a full understanding of speech. The resulting language access difficulties may have serious consequences on language, cognitive and social development. Cued speech (CS) was developed to eliminate this ambiguity by complementing lip-reading with hand gestures, giving access to the entire phonological content of speech through the visual modality alone. Despite its proven efficiency for improving linguistic and communicative abilities, the mechanisms of CS perception remain largely unknown. The goal of the present study is to delineate the brain regions involved in cued speech perception and identify their role in visual and language-related processes. Three matched groups of participants were scanned during two fMRI experiments: Prelingually deaf users of cued speech, hearing users of cued speech, and naive hearing controls. In Experiment 1, we presented videos of silent CS sentences, isolated lip movements, isolated gestures, plus CS sentences with speech sounds and meaningless CS sentences. In Experiment 2, we presented pictures of faces, bodies, written words, tools, and houses in order to understand the contribution of category-specific visual regions to CS perception. We delineated a number of mostly left-hemisphere brain regions involved in CS perception. We first found that language areas were activated in all groups by both silent CS sentences and isolated lip movements, and by gestures in deaf participants only. Despite overlapping activations when perceiving CS, several findings differentiated experts from novices. As such, CS expertise was associated with a leftward functional lateralization of the lateral occipital cortex, possibly driven by the expert identification of hand gestures. The Visual Word Form Area, which supports the interface between vision and language during reading, did not contribute to CS perception. Moreover, the integration of lip movements and gestures took place in a temporal language-related region in deaf users, and in movement-related regions in hearing users, reflecting their different profile of expertise in CS comprehension and production. Finally, we observed a strong involvement of the Dorsal Attentional Network in hearing users of CS, and identified the neural correlates of the variability in individual proficiency. Cued speech constitutes a novel pathway for accessing core language processes, halfway between speech perception and reading. The current study provides a delineation of the common and specific brain structures supporting those different modalities of language input, paving the way for further research.

Auditory cortex encodes lipreading information through spatially distributed activity

Temporal coding of speech in human auditory cortex

Cortical Neural Coding of Speech in Simple and Complex Auditory Scenes.

Neural coding of continuous speech in auditory cortex during monaural and dichotic listening.

Robust cortical encoding of slow temporal modulations of speech.

Cortical encoding of acoustic and linguistic rhythms in spoken narratives

Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Lip-Reading Enables the Brain to Synthesize Auditory Features of Unknown Silent Speech

Seeing speech: neural mechanisms of cued speech perception in prelingually deaf and hearing users

Lipreading and covert speech production similarly modulate human auditory-cortex responses to pure tones

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects

Lip movements enhance speech representations and effective connectivity in auditory dorsal stream

Lipreading a naturalistic narrative in a female population: Neural characteristics shared with listening and reading

Speech decoding using cortical and subcortical electrophysiological signals

Decoding speech perception from single cell activity in humans

Congruent audiovisual speech enhances auditory attention decoding with EEG.

Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

Prediction and constraint in audiovisual speech perception

Spoken language processing activates the primary visual cortex

Visual speech enhances auditory onset timing and envelope tracking through distinct mechanisms