Abstract:When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers' visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers' latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

Language-familiarity effect on voice recognition by blind listeners

How Noise and Language Proficiency Influence Speech Recognition by Individual Non-Native Listeners.

Superiority of blind over sighted listeners in voice recognition

Neural basis of language familiarity effects on voice recognition: An fNIRS study

How Different Types of Linguistic Information Impact Voice Perception: Evidence From the Language-Familiarity Effect

Visual Deprivation Alters Functional Connectivity of Neural Networks for Voice Recognition: A Resting-State fMRI Study

The mechanism of phonetic information in voice identity discrimination: a comparative study based on sighted and blind people

Hierarchical contributions of linguistic knowledge to talker identification: Phonological versus lexical familiarity

Familiarity Is Key: Exploring the Effect of Familiarity on the Face-Voice Correlation

Language-general versus language-specific processes in bilingual voice learning

Phonological Experience Modulates Voice Discrimination: Evidence from Functional Brain Networks Analysis

How Do Blind People Use Semantic and Phonological Cues for Spoken Word Recognition? an ERP Study

Neural Substrates for Voice Familiarity:A Fmri Study

EXPRESS: Prior multisensory learning can facilitate auditory-only voice-identity and speech recognition in noise

Effect of blindness on mismatch responses to Mandarin lexical tones, consonants, and vowels

The role of long-term target and masker talker familiarity in children's speech-in-speech recognition

How does voice familiarity affect speech intelligibility?

Algorithmic voice transformations reveal the phonological basis of language-familiarity effects in cross-cultural emotion judgments

Speech Perception in a Second Language: Hearing is Believing-Seeing is Not.

Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language

Formant Discrimination of Speech and Non-Speech Sounds in Temporally Modulated Noise: Effect of Language Experience