Abstract:To transform continuous speech into words, the human brain must resolve variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke context invariant speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such models may not capture the complete picture because they do not model the brain mechanism that categorizes sounds and consequently may overlook associated neural representations. By providing end-to-end accounts of speech-to-text transformation, new deep-learning systems could enable more complete brain models. We model EEG recordings of audiobook comprehension with the deep-learning speech recognition system Whisper. We find that (1) Whisper provides a self-contained EEG model of an intermediary representational stage that reflects elements of prelexical and lexical representation and prediction; (2) EEG modeling is more accurate when informed by 5-10s of speech context, which traditional context invariant categorical models do not encode; (3) Deep Whisper layers encoding linguistic structure were more accurate EEG models of selectively attended speech in two-speaker "cocktail party" listening conditions than early layers encoding acoustics. No such layer depth advantage was observed for unattended speech, consistent with a more superficial level of linguistic processing in the brain. Most people effortlessly can understand different speakers with distinct voices and accents, no matter whether they whisper, shout or are happy or sad. This effortlessness belies the remarkable computational challenge that our brains solve to transform such variable speech sounds into words. It is believed that our brains deal with this variability by categorizing speech sounds into a flow of phonemes and/or syllable units that are consistent, no matter how a word is spoken. Compelling supporting evidence has come from electrophysiological recordings of brain activity–colloquially known as brain waves—taken as people listen to speech. Scientists have trained computational models to predict brain wave fluctuations that correlate with sequences of phoneme categories. However, modeling only phoneme categories may miss key stages in the recognition process, including how sounds are mapped to phonemes, and phonemes to words. New deep learning speech models that are trained to recognize a diverse range of speech and speakers may offer new opportunities to provide more complete accounts of brain activity. This article reveals that these models indeed predict hitherto unexplained fluctuations in speech brain waves that reflect elements of sub-words and words, and shows that fluctuations are context sensitive, which may reflect the brain anticipating upcoming speech.

Speech perception: a model of word recognition

Hearings and mishearings: decrypting the spoken word

The neural dynamics of auditory word recognition and integration

Minimal Effective Theory for Phonotactic Memory: Capturing Local Correlations due to Errors in Speech

Words from spontaneous conversational speech can be recognized with human-like accuracy by an error-driven learning algorithm that discriminates between meanings straight from smart acoustic features, bypassing the phoneme as recognition unit

A Neural Network Model of Lexical Competition during Infant Spoken Word Recognition

Phonological neighbors cooperate during spoken-sentence processing: Evidence from a nonword detection task

An interactive activation model of context effects in letter perception: I. An account of basic findings.

Perception of Phonological Assimilation by Neural Speech Recognition Models

Evaluating Models of Robust Word Recognition with Serial Reproduction

Maintenance of subcategorical information during speech perception: Revisiting misunderstood limitations

Speech Perception Difficulty Modulates Theta-Band Encoding of Articulatory Synergies

Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation

Word Forms Reflect Trade-Offs Between Speaker Effort and Robust Listener Recognition

Robust assessment of the cortical encoding of word-level expectations using the temporal response function

Lombard Effect for Bilingual Speakers in Cantonese and English: importance of spectro-temporal features

Decoding syntactic class from EEG during spoken word recognition

Detecting Speech Abnormalities with a Perceiver-based Sequence Classifier that Leverages a Universal Speech Model

Phonological and syntactic competition effects in spoken word recognition: evidence from corpus-based statistics

Neural Tracking of Speech Acoustics in Noise Is Coupled with Lexical Predictability as Estimated by Large Language Models

Recognizing Speech in a Novel Accent: The Motor Theory of Speech Perception Reframed