Selective cortical representation of attended speaker in multi-talker speech perception

Nima Mesgarani,Edward F. Chang

DOI: https://doi.org/10.1038/nature11020

IF: 64.8

2012-04-18

Nature

Abstract:The neural correlates of how attended speech is internally represented are described, shedding light on the ‘cocktail party problem’.

multidisciplinary sciences

What problem does this paper attempt to address?

Mechanisms Underlying Selective Neuronal Tracking of Attended Speech at a “cocktail Party”

Elena M. Zion Golumbic,Nai Ding,Stephan Bickel,Peter Lakatos,Catherine A. Schevon,Guy M. McKhann,Robert R. Goodman,Ronald Emerson,Ashesh D. Mehta,Jonathan Z. Simon,David Poeppel,Charles E. Schroeder

DOI: https://doi.org/10.1016/j.neuron.2012.12.037

IF: 16.2

2013-01-01

Neuron

Abstract:The ability to focus on and understand one talker in a noisy social environment is a critical social-cognitive capacity, whose underlying neuronal mechanisms are unclear. We investigated the manner in which speech streams are represented in brain activity and the way that selective attention governs the brain's representation of speech using a "Cocktail Party" paradigm, coupled with direct recordings from the cortical surface in surgical epilepsy patients. We find that brain activity dynamically tracks speech streams using both low-frequency phase and high-frequency amplitude fluctuations and that optimal encoding likely combines the two. In and near low-level auditory cortices, attention "modulates" the representation by enhancing cortical tracking of attended speech streams, but ignored speech remains represented. In higher-order regions, the representation appears to become more "selective," in that there is no detectable tracking of ignored speech. This selectivity itself seems to sharpen as a sentence unfolds.
Emergence of Neural Encoding of Auditory Objects While Listening to Competing Speakers

Nai Ding,Jonathan Z. Simon

DOI: https://doi.org/10.1073/pnas.1205381109

IF: 11.1

2012-01-01

Proceedings of the National Academy of Sciences

Abstract:A visual scene is perceived in terms of visual objects. Similar ideas have been proposed for the analogous case of auditory scene analysis, although their hypothesized neural underpinnings have not yet been established. Here, we address this question by recording from subjects selectively listening to one of two competing speakers, either of different or the same sex, using magnetoencephalography. Individual neural representations are seen for the speech of the two speakers, with each being selectively phase locked to the rhythm of the corresponding speech stream and from which can be exclusively reconstructed the temporal envelope of that speech stream. The neural representation of the attended speech dominates responses (with latency near 100 ms) in posterior auditory cortex. Furthermore, when the intensity of the attended and background speakers is separately varied over an 8-dB range, the neural representation of the attended speech adapts only to the intensity of that speaker but not to the intensity of the background speaker, suggesting an object-level intensity gain control. In summary, these results indicate that concurrent auditory objects, even if spectrotemporally overlapping and not resolvable at the auditory periphery, are neurally encoded individually in auditory cortex and emerge as fundamental representational units for top-down attentional modulation and bottom-up neural adaptation.
Neural coding of continuous speech in auditory cortex during monaural and dichotic listening.

Nai Ding,Jonathan Z Simon

DOI: https://doi.org/10.1152/jn.00297.2011

2012-01-01

Journal of Neurophysiology

Abstract:Ding N, Simon JZ. Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. J Neurophysiol 107: 78-89, 2012. First published October 5, 2011; doi: 10.1152/jn.00297.2011.-The cortical representation of the acoustic features of continuous speech is the foundation of speech perception. In this study, noninvasive magnetoencephalography (MEG) recordings are obtained from human subjects actively listening to spoken narratives, in both simple and cocktail party-like auditory scenes. By modeling how acoustic features of speech are encoded in ongoing MEG activity as a spectrotemporal response function, we demonstrate that the slow temporal modulations of speech in a broad spectral region are represented bilaterally in auditory cortex by a phase-locked temporal code. For speech presented monaurally to either ear, this phase-locked response is always more faithful in the right hemisphere, but with a shorter latency in the hemisphere contralateral to the stimulated ear. When different spoken narratives are presented to each ear simultaneously (dichotic listening), the resulting cortical neural activity precisely encodes the acoustic features of both of the spoken narratives, but slightly weakened and delayed compared with the monaural response. Critically, the early sensory response to the attended speech is considerably stronger than that to the unattended speech, demonstrating top-down attentional gain control. This attentional gain is substantial even during the subjects' very first exposure to the speech mixture and therefore largely independent of knowledge of the speech content. Together, these findings characterize how the spectrotemporal features of speech are encoded in human auditory cortex and establish a single-trial-based paradigm to study the neural basis underlying the cocktail party phenomenon.
Identification of Attended Speech Stream Using Single-Trial Electroencephalography Recording

Ala Somarowthu,Nai Ding,Ying-Yee Kong

DOI: https://doi.org/10.1121/1.4950389

2016-01-01

Abstract:Selective attention differentially modulates neural responses to simultaneously presented speech streams. In this study, single-trial EEG classification was performed to identify the attended speech from a two-talker speech mixture. During EEG recordings, normal-hearing listeners paid attention to one speech stream while listening to speech mixtures. The target-to-masker ratios (TMRs) varied from −9 to + 9 dB. Individual speech streams were processed with head-related transfer functions to simulate different spatial locations. Two simulated spatial conditions (0 vs. + /−90 and + 45 vs. −45 degree azimuth) were tested for each TMR. Features related to (1) cross-correlation values between EEG signals and temporal envelope of each speech stream, or (2) correlation values between reconstructed speech from the EEG signals with the acoustic stimuli, were fed to the classifiers. The dimensionality of the feature vector was reduced using Principal Component Analysis. Linear Discriminant Analysis and Support Vector Machine were used to classify the EEG signals. Classifiers were trained and tested with a five-fold cross validation method on data pooled across TMRs and source locations for trial lengths from 50 s to 10 s. Average classification accuracy was 85% with a 50 s trial length and maintained high at 70% with a reduced trial length of 10 s.
Cortical Neural Coding of Speech in Simple and Complex Auditory Scenes.

Nai Ding,Jonathan Z. Simon

DOI: https://doi.org/10.1121/1.3587731

2011-01-01

Abstract:The neural representation of continuous speech in human auditory cortex was obtained noninvasively via magnetoencephalography. The neural response was recorded from human subjects listening to a spoken narrative, either in clean or in the presence of interfering speech. The cortical neural response to clean speech is demonstrated to precisely track the slow temporal modulations (<10 Hz) of speech in a broad spectral region between 400 Hz and 2 kHz. The neural code is sufficiently faithful to decode acoustic features of speech. To examine the robustness of, and the role of attention in, this neural code, another spoken narrative was presented simultaneously, either to a different ear (dichotically) or to the same ear (diotically), instructing the subjects to focus on only one of the two speech signals. The cortical representation of the attended speech is found to be substantially stronger than that of the unattended speech. This attentional effect is significant during the subjects’ first exposure to the spoken narratives. These results demonstrate that auditory cortex precisely represents the slow temporal modulations of speech and maintains separate neural representations for concurrent speech signals that can be individually and strongly modulated by attention. [Work supported by the NIDCD Grant No. R01‐DC‐008342.]
Robust cortical encoding of slow temporal modulations of speech.

Nai Ding,Jonathan Z Simon

DOI: https://doi.org/10.1007/978-1-4614-1590-9_41

2013-01-01

Abstract:This study investigates the neural representation of speech in complex listening environments. Subjects listened to a narrated story, masked by either another speech stream or by stationary noise. Neural recordings were made using magnetoencephalography (MEG), which can measure cortical activity synchronized to the temporal envelope of speech. When two speech streams are presented simultaneously, cortical activity is predominantly synchronized to the speech stream the listener attends to, even if the unattended, competing-speech stream is more intense (up to 8 dB). When speech is presented together with spectrally matched stationary noise, cortical activity remains precisely synchronized to the temporal envelope of speech until the noise is 9 dB more intense. Critically, the precision of the neural synchronization to speech predicts subjectively rated speech intelligibility in noise. Further analysis reveals that it is longer-latency (similar to 100 ms) neural responses, but not shorter-latency (similar to 50 ms) neural responses, that show selectivity to the attended speech and invariance to background noise. This indicates a processing transition, from encoding the acoustic scene to encoding the behaviorally important auditory object, in auditory cortex. In sum, it is demonstrated that neural synchronization to the speech envelope is robust to acoustic interference, whether speech or noise, and therefore provides a strong candidate for the neural basis of acoustic-background invariant speech recognition.
Temporal coding of speech in human auditory cortex

Jonathan Z. Simon,Nai Ding

2012-01-01

Abstract:Human listeners can reliably recognize speech in complex listening environments. The underlying neural mechanisms, however, remain unclear and cannot yet be emulated by any artificial system. In this dissertation, we study how speech is represented in the human auditory cortex and how the neural representation contributes to reliable speech recognition. Cortical activity from normal hearing human subjects is noninvasively recorded using magnetoencephalography, during natural speech listening. It is first demonstrated that neural activity from auditory cortex is precisely synchronized to the slow temporal modulations of speech, when the speech signal is presented in a quiet listening environment. How this neural representation is affected by acoustic interference is then investigated. Acoustic interference degrades speech perception via two mechanisms, informational masking and energetic masking, which are addressed respectively by using a competing speech stream and a stationary noise as the interfering sound. When two speech streams are presented simultaneously, cortical activity is predominantly synchronized to the speech stream the listener attends to, even if the unattended, competing speech stream is 8 dB more intense. When speech is presented together with spectrally matched stationary noise, cortical activity remains precisely synchronized to the temporal modulations of speech until the noise is 9 dB more intense. Critically, the accuracy of neural synchronization to speech predicts how well individual listeners can understand speech in noise. Further analysis reveals that two neural sources contribute to speech synchronized cortical activity, one with a shorter response latency of about 50 ms and the other with a longer response latency of about 100 ms. The longer-latency component, but not the shorter-latency component, shows selectivity to the attended speech and invariance to background noise, indicating a transition from encoding the acoustic scene to encoding the behaviorally important auditory object, in auditory cortex. Taken together, we have demonstrated that during natural speech comprehension, neural activity in the human auditory cortex is precisely synchronized to the slow temporal modulations of speech. This neural synchronization is robust to acoustic interference, whether speech or noise, and therefore provides a strong candidate for the neural basis of acoustic background invariant speech recognition.
Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Nai Ding,Jonathan Z. Simon

DOI: https://doi.org/10.1523/jneurosci.5297-12.2013

2013-01-01

Abstract:Speech recognition is remarkably robust to the listening background, even when the energy of background sounds strongly overlaps with that of speech. How the brain transforms the corrupted acoustic signal into a reliable neural representation suitable for speech recognition, however, remains elusive. Here, we hypothesize that this transformation is performed at the level of auditory cortex through adaptive neural encoding, and we test the hypothesis by recording, using MEG, the neural responses of human subjects listening to a narrated story. Spectrally matched stationary noise, which has maximal acoustic overlap with the speech, is mixed in at various intensity levels. Despite the severe acoustic interference caused by this noise, it is here demonstrated that low-frequency auditory cortical activity is reliably synchronized to the slow temporal modulations of speech, even when the noise is twice as strong as the speech. Such a reliable neural representation is maintained by intensity contrast gain control and by adaptive processing of temporal modulations at different time scales, corresponding to the neural δ and θ bands. Critically, the precision of this neural synchronization predicts how well a listener can recognize speech in noise, indicating that the precision of the auditory cortical representation limits the performance of speech recognition in noise. Together, these results suggest that, in a complex listening environment, auditory cortex can selectively encode a speech stream in a background insensitive manner, and this stable neural representation of speech provides a plausible basis for background-invariant recognition of speech.
Selective Attention Enhances Beta-Band Cortical Oscillation to Speech under "Cocktail-Party" Listening Conditions.

Yayue Gao,Qian Wang,Yu Ding,Changming Wang,Haifeng Li,Xihong Wu,Tianshu Qu,Liang Li

DOI: https://doi.org/10.3389/fnhum.2017.00034

IF: 3.473

2017-01-01

Frontiers in Human Neuroscience

Abstract:Human listeners are able to selectively attend to target speech in a noisy environment with multiple-people talking. Using recordings of scalp electroencephalogram (EEG), this study investigated how selective attention facilitates the cortical representation of target speech under a simulated "cocktail-party" listening condition with speech-on-speech masking. The result shows that the cortical representation of target-speech signals under the multiple-people talking condition was specifically improved by selective attention relative to the non-selective-attention listening condition, and the beta-band activity was most strongly modulated by selective attention. Moreover, measured with the Granger Causality value, selective attention to the single target speech in the mixed-speech complex enhanced the following four causal connectivities for the beta-band oscillation: the ones (1) from site FT7 to the right motor area, (2) from the left frontal area to the right motor area, (3) from the central frontal area to the right motor area, and (4) from the central frontal area to the right frontal area. However, the selective-attention-induced change in beta-band causal connectivity from the central frontal area to the right motor area, but not other beta-band causal connectivities, was significantly correlated with the selective-attention-induced change in the cortical beta-band representation of target speech. These findings suggest that under the "cocktail-party" listening condition, the beta-band oscillation in EEGs to target speech is specifically facilitated by selective attention to the target speech that is embedded in the mixed-speech complex. The selective attention-induced unmasking of target speech may be associated with the improved beta-band functional connectivity from the central frontal area to the right motor area, suggesting a top-down attentional modulation of the speech-motor process.
Attention-Driven Modulation of Auditory Cortex Activity during Selective Listening in a Multispeaker Setting

Sebastian Puschmann,Mor Regev,Kayson Fakhar,Robert J Zatorre,Christiane M Thiel

DOI: https://doi.org/10.1523/JNEUROSCI.1157-23.2023

2024-04-10

Abstract:Real-world listening settings often consist of multiple concurrent sound streams. To limit perceptual interference during selective listening, the auditory system segregates and filters the relevant sensory input. Previous work provided evidence that the auditory cortex is critically involved in this process and selectively gates attended input toward subsequent processing stages. We studied at which level of auditory cortex processing this filtering of attended information occurs using functional magnetic resonance imaging (fMRI) and a naturalistic selective listening task. Forty-five human listeners (of either sex) attended to one of two continuous speech streams, presented either concurrently or in isolation. Functional data were analyzed using an inter-subject analysis to assess stimulus-specific components of ongoing auditory cortex activity. Our results suggest that stimulus-related activity in the primary auditory cortex and the adjacent planum temporale are hardly affected by attention, whereas brain responses at higher stages of the auditory cortex processing hierarchy become progressively more selective for the attended input. Consistent with these findings, a complementary analysis of stimulus-driven functional connectivity further demonstrated that information on the to-be-ignored speech stream is shared between the primary auditory cortex and the planum temporale but largely fails to reach higher processing stages. Our findings suggest that the neural processing of ignored speech cannot be effectively suppressed at the level of early cortical processing of acoustic features but is gradually attenuated once the competing speech streams are fully segregated.
Neural Mechanisms for Selectively Tuning in to the Target Speaker in a Naturalistic Noisy Situation

Bohan Dai,Chuansheng Chen,Yuhang Long,Lifen Zheng,Hui Zhao,Xialu Bai,Wenda Liu,Yuxuan Zhang,Li Liu,Taomei Guo,Guosheng Ding,Chunming Lu

DOI: https://doi.org/10.1038/s41467-018-04819-z

IF: 16.6

2018-01-01

Nature Communications

Abstract:The neural mechanism for selectively tuning in to a target speaker while tuning out the others in a multi-speaker situation (i.e., the cocktail-party effect) remains elusive. Here we addressed this issue by measuring brain activity simultaneously from a listener and from multiple speakers while they were involved in naturalistic conversations. Results consistently show selectively enhanced interpersonal neural synchronization (INS) between the listener and the attended speaker at left temporal–parietal junction, compared with that between the listener and the unattended speaker across different multi-speaker situations. Moreover, INS increases significantly prior to the occurrence of verbal responses, and even when the listener’s brain activity precedes that of the speaker. The INS increase is independent of brain-to-speech synchronization in both the anatomical location and frequency range. These findings suggest that INS underlies the selective process in a multi-speaker situation through neural predictions at the content level but not the sensory level of speech.
Cortical tracking of speakers' spectral changes predicts selective listening

Francisco Cervantes Constantino,Angel A. Caputi

DOI: https://doi.org/10.1101/2024.05.23.595545

2024-10-20

Abstract:A social scene is particularly informative when people are distinguishable. To understand somebody amid a 'cocktail party' chatter, we automatically index their voice. This ability is underpinned by parallel processing of vocal spectral contours from speech sounds, but it has not yet been established how this occurs in the brain's cortex. We investigate single-trial neural tracking of slow frequency modulations in speech using electroencephalography. Participants briefly listened to unfamiliar single speakers, and in addition, they performed a cocktail party comprehension task. Quantified through stimulus reconstruction methods, robust tracking was found in neural responses to slow (delta-theta range) modulations of frequency contours in the fourth and fifth formant band, equivalent to the 3.5-5 KHz audible range. Instantaneous frequency spacing (DF), which also yields indexical information from the vocal tract, was similarly decodable. Moreover, EEG evidence of listeners' spectral tracking abilities predicted their chances of succeeding at selective listening when faced with two-speaker speech mixtures. In summary, the results indicate that the communicating brain can rely on locking of cortical rhythms to major changes led by upper resonances of the vocal tract. Their corresponding articulatory mechanics hence continuously issue a fundamental credential for listeners to target in real time.

Neuroscience
Attention-driven modulation of auditory cortex activity during selective listening in a multi-speaker setting

Sebastian Puschmann,Mor Regev,Kayson Fakhar,Robert J Zatorre,Christiane M Thiel,Robert J. Zatorre,Christiane M. Thiel

DOI: https://doi.org/10.1523/jneurosci.1157-23.2023

2024-02-22

Abstract:Real-world listening settings often consist of multiple concurrent sound streams. To limit perceptual interference during selective listening, the auditory system segregates and filters the relevant sensory input. Previous work provided evidence that auditory cortex is critically involved in this process, and selectively gates attended input towards subsequent processing stages. We studied at which level of auditory cortex processing this filtering of attended information occurs using functional magnetic resonance imaging (fMRI) and a naturalistic selective listening task. Forty-five human listeners (of either sex) attended to one of two continuous speech streams, presented either concurrently or in isolation. Functional data were analyzed using an inter-subject analysis to assess stimulus-specific components of ongoing auditory cortex activity. Our results suggest that stimulus-related activity in primary auditory cortex and the adjacent planum temporale are hardly affected by attention, whereas brain responses at higher stages of the auditory cortex processing hierarchy become progressively more selective for the attended input. Consistent with these findings, a complementary analysis of stimulus-driven functional connectivity further demonstrated that information on the to-be-ignored speech stream is shared between primary auditory cortex and the planum temporale, but largely fails to reach higher processing stages. Our findings suggest that the neural processing of ignored speech cannot be effectively suppressed at the level of early cortical processing of acoustic features, but is gradually attenuated once the competing speech streams are fully segregated. Significance statement When listening selectively to one of multiple speech streams, the attended stream is segregated from the acoustic mixture and gated to further information processing, whereas the processing of irrelevant speech must be attenuated to minimize distraction. We here used functional magnetic resonance imaging to investigate the role of auditory cortex for this attentional filtering of auditory input. Our data suggest that stimulus representations in primary auditory cortex and the planum temporale are hardly affected by attention, whereas responses further along the auditory cortex processing hierarchy get more and more selective for the attended input. This suggests that ignored speech cannot be effectively suppressed on the acoustic feature level but is gradually attenuated once the competing speech streams are fully segregated.

neurosciences
Left Superior Temporal Gyrus Is Coupled to Attended Speech in a Cocktail-Party Auditory Scene

M. Op de beeck,V. Wens,Marc Vander Ghinst,M. Bourguignon,R. Hari,P. van Bogaert,B. Marty,S. Hassid,G. Choufani,X. De Tiège,S. Goldman,V. Jousmäki

DOI: https://doi.org/10.1523/JNEUROSCI.1730-15.2016

2016-02-03

Journal of Neuroscience

Abstract:Using a continuous listening task, we evaluated the coupling between the listener's cortical activity and the temporal envelopes of different sounds in a multitalker auditory scene using magnetoencephalography and corticovocal coherence analysis. Neuromagnetic signals were recorded from 20 right-handed healthy adult humans who listened to five different recorded stories (attended speech streams), one without any multitalker background (No noise) and four mixed with a “cocktail party” multitalker background noise at four signal-to-noise ratios (5, 0, −5, and −10 dB) to produce speech-in-noise mixtures, here referred to as Global scene. Coherence analysis revealed that the modulations of the attended speech stream, presented without multitalker background, were coupled at ∼0.5 Hz to the activity of both superior temporal gyri, whereas the modulations at 4–8 Hz were coupled to the activity of the right supratemporal auditory cortex. In cocktail party conditions, with the multitalker background noise, the coupling was at both frequencies stronger for the attended speech stream than for the unattended Multitalker background. The coupling strengths decreased as the Multitalker background increased. During the cocktail party conditions, the ∼0.5 Hz coupling became left-hemisphere dominant, compared with bilateral coupling without the multitalker background, whereas the 4–8 Hz coupling remained right-hemisphere lateralized in both conditions. The brain activity was not coupled to the multitalker background or to its individual talkers. The results highlight the key role of listener's left superior temporal gyri in extracting the slow ∼0.5 Hz modulations, likely reflecting the attended speech stream within a multitalker auditory scene. SIGNIFICANCE STATEMENT When people listen to one person in a “cocktail party,” their auditory cortex mainly follows the attended speech stream rather than the entire auditory scene. However, how the brain extracts the attended speech stream from the whole auditory scene and how increasing background noise corrupts this process is still debated. In this magnetoencephalography study, subjects had to attend a speech stream with or without multitalker background noise. Results argue for frequency-dependent cortical tracking mechanisms for the attended speech stream. The left superior temporal gyrus tracked the ∼0.5 Hz modulations of the attended speech stream only when the speech was embedded in multitalker background, whereas the right supratemporal auditory cortex tracked 4–8 Hz modulations during both noiseless and cocktail-party conditions.

Medicine,Biology,Psychology
At what time is the cocktail party? A late locus of selective attention to natural speech

Alan J. Power,John J. Foxe,Emma‐Jane Forde,Richard B. Reilly,Edmund C. Lalor,Emma-Jane Forde

DOI: https://doi.org/10.1111/j.1460-9568.2012.08060.x

IF: 3.698

2012-03-30

European Journal of Neuroscience

Abstract:Distinguishing between speakers and focusing attention on one speaker in multi-speaker environments is extremely important in everyday life. Exactly how the brain accomplishes this feat and, in particular, the precise temporal dynamics of this attentional deployment are as yet unknown. A long history of behavioral research using dichotic listening paradigms has debated whether selective attention to speech operates at an early stage of processing based on the physical characteristics of the stimulus or at a later stage during semantic processing. With its poor temporal resolution fMRI has contributed little to the debate, while EEG-ERP paradigms have been hampered by the need to average the EEG in response to discrete stimuli which are superimposed onto ongoing speech. This presents a number of problems, foremost among which is that early attention effects in the form of endogenously generated potentials can be so temporally broad as to mask later attention effects based on the higher level processing of the speech stream. Here we overcome this issue by utilizing the AESPA (auditory evoked spread spectrum analysis) method which allows us to extract temporally detailed responses to two concurrently presented speech streams in natural cocktail-party-like attentional conditions without the need for superimposed probes. We show attentional effects on exogenous stimulus processing in the 200-220 ms range in the left hemisphere. We discuss these effects within the context of research on auditory scene analysis and in terms of a flexible locus of attention that can be deployed at a particular processing stage depending on the task.

neurosciences
EEG-based Speaker-Listener Neural Coupling Reflects Speech-Selective Attentional Mechanisms Beyond the Speech Stimulus

Jiawei Li,Bo Hong,Guido Nolte,Andreas K. Engel,Dan Zhang

DOI: https://doi.org/10.1093/cercor/bhad347

IF: 4.861

2023-01-01

Cerebral Cortex

Abstract:When we pay attention to someone, do we focus only on the sound they make, the word they use, or do we form a mental space shared with the speaker we want to pay attention to? Some would argue that the human language is no other than a simple signal, but others claim that human beings understand each other because they form a shared mental ground between the speaker and the listener. Our study aimed to explore the neural mechanisms of speech-selective attention by investigating the electroencephalogram-based neural coupling between the speaker and the listener in a cocktail party paradigm. The temporal response function method was employed to reveal how the listener was coupled to the speaker at the neural level. The results showed that the neural coupling between the listener and the attended speaker peaked 5 s before speech onset at the delta band over the left frontal region, and was correlated with speech comprehension performance. In contrast, the attentional processing of speech acoustics and semantics occurred primarily at a later stage after speech onset and was not significantly correlated with comprehension performance. These findings suggest a predictive mechanism to achieve speakerlistener neural coupling for successful speech comprehension.
Selective attention to audiovisual speech routes activity through recurrent feedback-feedforward loops between different nodes of the speech network

Patrik Wikman,Viljami Salmela,Eetu Sjöblom,Miika Leminen,Matti Laine,Kimmo Alho

DOI: https://doi.org/10.1101/2023.07.17.549287

2024-02-12

Abstract:Selective attention related top-down modulation plays a significant role in separating relevant speech from irrelevant background speech when vocal attributes separating concurrent speakers are small and continuously evolving. Electrophysiological studies have shown that such top-down modulation enhances neural tracking of attended speech. Yet, the specific cortical regions involved remain unclear due to the limited spatial resolution of most electrophysiological techniques. To overcome such limitations, we collected both EEG (high temporal resolution) and fMRI (high spatial resolution), while human participants selectively attended to speakers in audiovisual scenes containing overlapping cocktail party speech. To utilize the advantages of the respective techniques, we analysed neural tracking of speech using the EEG data and performed representational dissimilarity-based EEG-fMRI fusion. We observed that attention enhanced neural tracking and modulated EEG correlates throughout the latencies studied. Further, attention related enhancement of neural tracking fluctuated in predictable temporal profiles. We discuss how such temporal dynamics could arise from a combination of interactions between attention and prediction as well as plastic properties of the auditory cortex. EEG-fMRI fusion revealed attention related iterative feedforward-feedback loops between hierarchically organised nodes of the ventral auditory object related processing stream. Our findings support models where attention facilitates dynamic neural changes in the auditory cortex, ultimately aiding discrimination of relevant sounds from irrelevant ones while conserving neural resources.

Neuroscience
Temporal coherence shapes cortical responses to speech mixtures in a ferret cocktail party

Neha Joshi,Wing Yiu Ng,Karan Thakkar,Daniel Duque,Pingbo Yin,Jonathan Fritz,Mounya Elhilali,Shihab Shamma

DOI: https://doi.org/10.1038/s42003-024-07096-3

IF: 6.548

2024-10-26

Communications Biology

Abstract:Perceptual segregation of complex sounds such as speech and music simultaneously emanating from multiple sources is a remarkable ability that is common in humans and other animals alike. Unlike animal physiological experiments with simplified sounds or human investigations with spatially broad imaging techniques, this study combines insights from animal single-unit recordings with segregation of speech-like sound mixtures. Ferrets are trained to attend to a female voice and detect a target word, both in presence and absence of a concurrent equally salient male voice. Recordings are made in primary and secondary auditory cortical fields, and in frontal cortex. During task performance, representation of the female words becomes enhanced relative to the male in all, but especially in higher cortical regions. Analysis of the temporal and spectral response characteristics during task performance reveals how speech segregation gradually emerges in the auditory cortex. A computational model evaluated on the same voice mixtures replicates and extends these results to different attentional targets (attention to female or male voices). These findings underscore the role of the principle of temporal coherence whereby attention to a target voice binds together all neural responses coherently modulated with the target, thus ultimately forming and extracting a common auditory stream.

biology
Attention to speech: Mapping distributed and selective attention systems

Galit Agmon,Paz Har-Shai Yahav,Michal Ben-Shachar,Elana Zion Golumbic

DOI: https://doi.org/10.1101/2021.02.13.431098

2021-02-13

Abstract:Abstract Daily life is full of situations where many people converse at the same time. Under these noisy circumstances, individuals can employ different listening strategies to deal with the abundance of sounds around them. In this fMRI study we investigated how applying two different listening strategies – Selective vs. Distributed attention – affects the pattern of neural activity. Specifically, in a simulated ‘cocktail party’ paradigm, we compared brain activation patterns when listeners attend selectively to only one speaker and ignore all others, versus when they distribute their attention and attempt to follow two or four speakers at the same time. Results indicate that the two attention types activate a highly overlapping, bilateral fronto-temporal-parietal network of functionally connected regions. This network includes auditory association cortex (bilateral STG/STS) and higher-level regions related to speech processing and attention (bilateral IFG/insula, right MFG, left IPS). Within this network, responses in specific areas were modulated by the type of attention required. Specifically, auditory and speech-processing regions exhibited higher activity during Distributed attention, whereas fronto-parietal regions were activated more strongly during Selective attention. This pattern suggests that a common perceptual-attentional network is engaged when dealing with competing speech-inputs, regardless of the specific task at hand. At the same time, local activity within nodes of this network varies when implementing different listening strategies, reflecting the different cognitive demands they impose. These results nicely demonstrate the system’s flexibility to adapt its internal computations to accommodate different task requirements and listener goals. Significance Statement Hearing many people talk simultaneously poses substantial challenges for the human perceptual and cognitive systems. We compared neural activity when listeners applied two different listening strategy to deal with these competing inputs: attending selectively to one speaker vs. distributing attention among all speakers. A network of functionally connected brain regions, involved in auditory processing, language processing and attentional control was activated when applying both attention types. However, activity within this network was modulated by the type of attention required and the number of competing speakers. These results suggest a common ‘attention to speech’ network, providing the computational infrastructure to deal effectively with multi-speaker input, but with sufficient flexibility to implement different prioritization strategies and to adapt to different listener goals.

Selective cortical representation of attended speaker in multi-talker speech perception

Mechanisms Underlying Selective Neuronal Tracking of Attended Speech at a “cocktail Party”

Emergence of Neural Encoding of Auditory Objects While Listening to Competing Speakers

Neural coding of continuous speech in auditory cortex during monaural and dichotic listening.

Identification of Attended Speech Stream Using Single-Trial Electroencephalography Recording

Cortical Neural Coding of Speech in Simple and Complex Auditory Scenes.

Robust cortical encoding of slow temporal modulations of speech.

Temporal coding of speech in human auditory cortex

Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Selective Attention Enhances Beta-Band Cortical Oscillation to Speech under "Cocktail-Party" Listening Conditions.

Attention-Driven Modulation of Auditory Cortex Activity during Selective Listening in a Multispeaker Setting

Neural Mechanisms for Selectively Tuning in to the Target Speaker in a Naturalistic Noisy Situation

Cortical tracking of speakers' spectral changes predicts selective listening

Attention-driven modulation of auditory cortex activity during selective listening in a multi-speaker setting

Left Superior Temporal Gyrus Is Coupled to Attended Speech in a Cocktail-Party Auditory Scene

At what time is the cocktail party? A late locus of selective attention to natural speech

EEG-based Speaker-Listener Neural Coupling Reflects Speech-Selective Attentional Mechanisms Beyond the Speech Stimulus

Selective attention to audiovisual speech routes activity through recurrent feedback-feedforward loops between different nodes of the speech network

Temporal coherence shapes cortical responses to speech mixtures in a ferret cocktail party

Attention to speech: Mapping distributed and selective attention systems