Abstract:Purpose: We explore a new approach to the study of cognitive effort involved in listening to speech by measuring the brain activity in a listener in relation to the brain activity in a speaker. We hypothesize that the strength of this brain-to-brain synchrony (coupling) reflects the magnitude of cognitive effort involved in verbal communication and includes both listening effort and speaking effort. We investigate whether interbrain synchrony is greater in native-to-native versus native-to-nonnative communication using functional near-infrared spectroscopy (fNIRS). Method: Two speakers participated, a native speaker of American English and a native speaker of Korean who spoke English as a second language. Each speaker was fitted with the fNIRS cap and told short stories. The native English speaker provided the English narratives, and the Korean speaker provided both the nonnative (accented) English and Korean narratives. In separate sessions, fNIRS data were obtained from seven English monolingual participants ages 20–24 years who listened to each speaker's stories. After listening to each story in native and nonnative English, they retold the content, and their transcripts and audio recordings were analyzed for comprehension and discourse fluency, measured in the number of hesitations and articulation rate. No story retellings were obtained for narratives in Korean (an incomprehensible language for English listeners). Utilizing fNIRS technique termed sequential scanning, we quantified the brain-to-brain synchronization in each speaker–listener dyad. Results: For native-to-native dyads, multiple brain regions associated with various linguistic and executive functions were activated. There was a weaker coupling for native-to-nonnative dyads, and only the brain regions associated with higher order cognitive processes and functions were synchronized. All listeners understood the content of all stories, but they hesitated significantly more when retelling stories told in accented English. The nonnative speaker hesitated significantly more often than the native speaker and had a significantly slower articulation rate. There was no brain-to-brain coupling during listening to Korean, indicating a break in communication when listeners failed to comprehend the speaker. Conclusions: We found that effortful speech processing decreased interbrain synchrony and delayed comprehension processes. The obtained brain-based and behavioral patterns are consistent with our proposal that cognitive effort in verbal communication pertains to both the listener and the speaker and that brain-to-brain synchrony can be an indicator of differences in their cumulative communicative effort. Supplemental Material: https://doi.org/10.23641/asha.25452142

Speaker-listener neural coupling correlates with semantic and acoustic features of naturalistic speech

Robust cortical encoding of slow temporal modulations of speech.

Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment

Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Temporal coding of speech in human auditory cortex

Cortical encoding of acoustic and linguistic rhythms in spoken narratives

Speaker-listener neural coupling reveals a right-lateralized mechanism for non-native speech-in-noise comprehension

EEG-based Speaker-Listener Neural Coupling Reflects Speech-Selective Attentional Mechanisms Beyond the Speech Stimulus

Auditory-Articulatory Neural Alignment Between Listener and Speaker During Verbal Communication.

How a speaker herds the audience: multibrain neural convergence over time during naturalistic storytelling

Reduced Listener-speaker Neural Coupling Underlies Speech Understanding Difficulty in Older Adults

How a speaker herds the audience: Multi-brain neural convergence over time during naturalistic storytelling

How the Listener’s Attention Dynamically Switches Between Different Speakers During a Natural Conversation

The "two-brain" approach reveals the active role of task-deactivated default mode network in speech comprehension

Evaluating Speaker–Listener Cognitive Effort in Speech Communication Through Brain-to-Brain Synchrony: A Pilot Functional Near-Infrared Spectroscopy Investigation

Neural Mechanisms for Selectively Tuning in to the Target Speaker in a Naturalistic Noisy Situation

Bidirectional Connectivity Between Broca's Area and Wernicke's Area During Interactive Verbal Communication

Natural language processing models reveal neural dynamics of human conversation

Study on neural entrainment to continuous speech using dynamic source connectivity analysis

Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN)

The structure and statistics of language jointly shape cross-frequency neural dynamics during spoken language comprehension