Abstract:In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

Denoising based on spatial filtering

Fast EEG-Based Decoding Of The Directional Focus Of Auditory Attention Using Common Spatial Patterns

Real-Time Spatiotemporal Filtering for Artifact-Free EEG during Electrical Neurostimulation

Statistically Optimized Spatial Filtering in Decoding Steady-State Visual Evoked Potentials Based on Task-Related Component Analysis.

Spatiotemporal signal space separation for regions of interest: Application for extracting neuromagnetic responses evoked by deep brain stimulation

Real-Time Tracking of Selective Auditory Attention From M/EEG: A Bayesian Filtering Approach

Variational Bayes for spatiotemporal identification of event-related potential subcomponents

EEG denoising during transcutaneous auricular vagus nerve stimulation across simulated, phantom and human data

Non-invasive single-trial EEG detection of evoked human neocortical population spikes

Human Recognition Using Deep Neural Networks and Spatial Patterns of SSVEP Signals

Mining within-trial oscillatory brain dynamics to address the variability of optimized spatial filters

Multi-response deconvolution of auditory evoked potentials in a reduced representation space

Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli

EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses

A time-frequency denoising method for single-channel event-related EEG

ERP denoising in multichannel EEG data using contrasts between signal and noise subspaces

Extraction of Mismatch Negativity Using A Resampling-Based Spatial Filtering Method

Speech-reception-threshold estimation via EEG-based continuous speech envelope reconstruction

Single-trial ERPs Denoising Via Collaborative Filtering on ERPs Images.

Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity