Abstract:Purpose: We explore a new approach to the study of cognitive effort involved in listening to speech by measuring the brain activity in a listener in relation to the brain activity in a speaker. We hypothesize that the strength of this brain-to-brain synchrony (coupling) reflects the magnitude of cognitive effort involved in verbal communication and includes both listening effort and speaking effort. We investigate whether interbrain synchrony is greater in native-to-native versus native-to-nonnative communication using functional near-infrared spectroscopy (fNIRS). Method: Two speakers participated, a native speaker of American English and a native speaker of Korean who spoke English as a second language. Each speaker was fitted with the fNIRS cap and told short stories. The native English speaker provided the English narratives, and the Korean speaker provided both the nonnative (accented) English and Korean narratives. In separate sessions, fNIRS data were obtained from seven English monolingual participants ages 20–24 years who listened to each speaker's stories. After listening to each story in native and nonnative English, they retold the content, and their transcripts and audio recordings were analyzed for comprehension and discourse fluency, measured in the number of hesitations and articulation rate. No story retellings were obtained for narratives in Korean (an incomprehensible language for English listeners). Utilizing fNIRS technique termed sequential scanning, we quantified the brain-to-brain synchronization in each speaker–listener dyad. Results: For native-to-native dyads, multiple brain regions associated with various linguistic and executive functions were activated. There was a weaker coupling for native-to-nonnative dyads, and only the brain regions associated with higher order cognitive processes and functions were synchronized. All listeners understood the content of all stories, but they hesitated significantly more when retelling stories told in accented English. The nonnative speaker hesitated significantly more often than the native speaker and had a significantly slower articulation rate. There was no brain-to-brain coupling during listening to Korean, indicating a break in communication when listeners failed to comprehend the speaker. Conclusions: We found that effortful speech processing decreased interbrain synchrony and delayed comprehension processes. The obtained brain-based and behavioral patterns are consistent with our proposal that cognitive effort in verbal communication pertains to both the listener and the speaker and that brain-to-brain synchrony can be an indicator of differences in their cumulative communicative effort. Supplemental Material: https://doi.org/10.23641/asha.25452142

Auditory-Articulatory Neural Alignment Between Listener and Speaker During Verbal Communication.

Temporal coding of speech in human auditory cortex

Emergence of Neural Encoding of Auditory Objects While Listening to Competing Speakers

Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment

Separate Neural Subsystems Support Goal-Directed Speech Listening.

Bidirectional Connectivity Between Broca's Area and Wernicke's Area During Interactive Verbal Communication

EEG-based Speaker-Listener Neural Coupling Reflects Speech-Selective Attentional Mechanisms Beyond the Speech Stimulus

Speaker-listener neural coupling correlates with semantic and acoustic features of naturalistic speech

Speaker-listener neural coupling reveals a right-lateralized mechanism for non-native speech-in-noise comprehension

Neural Correlates of Temporal Complexity and Synchrony During Audiovisual Correspondence Detection

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Testing Native Language Neural Commitment at the Brainstem Level: A Cross-Linguistic Investigation of the Association Between Frequency-Following Response and Speech Perception.

Predictive Brain Mechanisms in Sound-to-Meaning Mapping During Speech Processing.

The "two-brain" approach reveals the active role of task-deactivated default mode network in speech comprehension

Spatial alignment between faces and voices improves selective attention to audio-visual speech

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

Immediate auditory feedback regulates inter-articulator speech coordination in service to phonetic structure

Congruent audiovisual speech enhances auditory attention decoding with EEG.

Evaluating Speaker–Listener Cognitive Effort in Speech Communication Through Brain-to-Brain Synchrony: A Pilot Functional Near-Infrared Spectroscopy Investigation

Neural Mechanisms for Selectively Tuning in to the Target Speaker in a Naturalistic Noisy Situation

How a speaker herds the audience: multibrain neural convergence over time during naturalistic storytelling