Abstract:In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.

EEG Gamma-Band Activity During Audiovisual Speech Comprehension in Different Noise Environments.

Robust cortical encoding of slow temporal modulations of speech.

Correlation Between Audio–visual Enhancement of Speech in Different Noise Environments and SNR: A Combined Behavioral and Electrophysiological Study

A Method to Study Global Spatial Patterns Related to Sensory Perception in Scalp EEG.

Effects of Spectral Degradation on Attentional Modulation of Cortical Auditory Responses to Continuous Speech

Alpha-band oscillation during speech recognition under different sensory conditions

Pre-stimulus Beta and Gamma Oscillatory Power Predicts Perceived Audiovisual Simultaneity.

Neural Oscillations Reflect the Individual Differences in the Temporal Perception of Audiovisual Speech.

The Oscillatory Activities And Its Synchronization In Auditory-Visual Integration As Revealed By Event-Related Potentials To Bimodal Stimuli

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Differential modulations of theta and beta oscillations by audiovisual congruency in letter‐speech sound integration

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

Monitoring Alpha Oscillations and Pupil Dilation across a Performance-Intensity Function

Congruent audiovisual speech enhances auditory attention decoding with EEG.

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli

Attentional inhibition by alpha power is modulated by faster theta rhythm and audio-visual congruency during natural speech perception

Beyond alpha band: prestimulus local oscillation and interregional synchrony of the beta band shape the temporal perception of the audiovisual beep-flash stimulus

Auditory Noise Leads to Increased Visual Brain-Computer Interface Performance: A Cross-Modal Study

Delta, theta, beta, and gamma brain oscillations index levels of auditory sentence processing