Abstract:In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.

EEG Correlates of Learning From Speech Presented in Environmental Noise

Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Robust cortical encoding of slow temporal modulations of speech.

Individual Differences in Cognition and Perception Predict Neural Processing of Speech in Noise for Audiometrically Normal Listeners.

Ear-EEG Measures of Auditory Attention to Continuous Speech

An ecological investigation of the effect of background noise on speech processing in a Virtual Classroom

Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli

'Are you even listening?' - EEG-based decoding of absolute auditory attention to natural speech

Assessing focus through ear-EEG: a comparative study between conventional cap EEG and mobile in- and around-the-ear EEG systems

Classification of Targets and Distractors in an Audiovisual Attention Task Based on Electroencephalography

Auditory and language contributions to neural encoding of speech features in noisy environments

Predicting EEG Responses to Attended Speech via Deep Neural Networks for Speech

Comparison of linear and nonlinear methods for decoding selective attention to speech from ear-EEG recordings

EEG Responses to auditory figure-ground perception

Attentional inhibition ability predicts neural representation during challenging auditory streaming

From Beeps to Streets: Unveiling Sensory Input and Relevance Across Auditory Contexts

Study of cognitive component of auditory attention to natural speech events

The cortical organization of listening effort: New insight from functional near-infrared spectroscopy

EEG-Based Intersubject Correlations Reflect Selective Attention in a Competing Speaker Scenario

NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals