Abstract:When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers' visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers' latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

‘Face’-related expressions in the Minnan Dialect of Chinese

Conceptual Structure and Measurement of Face Need:An Exploratory Research

A Quantitative Study of Face Negotiation Behaviors in Intercultural Conflicting Context

The Interpretation of Disjunction in the Scope of Dou in Child Mandarin.

A Multi-dimensional Approach to Register Variations in Mandarin Chinese

Re-conceptualizing the Chinese concept of face from a face-sensitive perspective: A case study of a modern Chinese TV drama

Can Familiarity Lessen the Effect of Locality? A Case Study of Mandarin Chinese Subjects and the Following Adverbials

Unattended Emotional Prosody Affects Visual Processing of Facial Expressions in Mandarin-Speaking Chinese: A Comparison With English-Speaking Canadians

Cognitive discrepancy in Chinese “face”: Mian and lian, and their impact on cognition of country-of-origin image

What Makes a Face a Face:A Review of Studies on Face Specificity

Understanding the Use of Quantifiers in Mandarin

Children's Use of Phonological Information in Ambiguity Resolution: a View from Mandarin Chinese.

The neurocognitive basis of Chinese idiomatic constructions and processing differences between native speakers and L2 learners of Mandarin

Coda Nasal Perception in Mandarin by Fuqing Min Speakers

Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language

Exploring Cross-cultural Travel of “Face”from the Perspective of Neo-Causal-Historical Theory

How Different Types of Linguistic Information Impact Voice Perception: Evidence From the Language-Familiarity Effect

Effects of Language Experience on Prosodic Realization of Focus in Southern Min-Mandarin Bilinguals.

How Dialects of Chinese Language Influence L1-Speakers Phonological and Phonetic Acquisition of English

The phonetic properties of the non-modal phonation in Shanghainese

The Phonological Coordination Relationship of Hancheng Dialect in Shaanxi Province