Abstract:In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. Recent studies have used encoding models to predict electroencephalography (EEG) responses to speech in noise-free listening situations, sometimes referred to as "speech tracking." Researchers have analyzed how speech tracking changes with different types of background noise. It is unclear, however, whether neural responses from acoustically rich, naturalistic environments with and without background noise can be generalized to more controlled stimuli. If encoding models for acoustically rich, naturalistic stimuli are generalizable to other tasks, this could aid in data collection from populations of individuals who may not tolerate listening to more controlled and less engaging stimuli for long periods of time. We recorded noninvasive scalp EEG while 17 human participants (8 male/9 female) listened to speech without noise and audiovisual speech stimuli containing overlapping speakers and background sounds. We fit multivariate temporal receptive field encoding models to predict EEG responses to pitch, the acoustic envelope, phonological features, and visual cues in both stimulus conditions. Our results suggested that neural responses to naturalistic stimuli were generalizable to more controlled datasets. EEG responses to speech in isolation were predicted accurately using phonological features alone, while responses to speech in a rich acoustic background were more accurate when including both phonological and acoustic features. Our findings suggest that naturalistic audiovisual stimuli can be used to measure receptive fields that are comparable and generalizable to more controlled audio-only stimuli.SIGNIFICANCE STATEMENT Understanding spoken language in natural environments requires listeners to parse acoustic and linguistic information in the presence of other distracting stimuli. However, most studies of auditory processing rely on highly controlled stimuli with no background noise, or with background noise inserted at specific times. Here, we compare models where EEG data are predicted based on a combination of acoustic, phonetic, and visual features in highly disparate stimuli-sentences from a speech corpus and speech embedded within movie trailers. We show that modeling neural responses to highly noisy, audiovisual movies can uncover tuning for acoustic and phonetic information that generalizes to simpler stimuli typically used in sensory neuroscience experiments.

Contrastive Learning of Shared Spatiotemporal EEG Representations Across Individuals for Naturalistic Neuroscience

Contrastive Learning of Shared Spatiotemporal EEG Representations Across Individuals for Naturalistic Neuroscience

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

GC-STCL: A Granger Causality-Based Spatial–Temporal Contrastive Learning Framework for EEG Emotion Recognition

Semantic-aware Contrastive Learning for Electroencephalography-to-Text Generation with Curriculum Learning

Assisting Schizophrenia Diagnosis Using Clinical Electroencephalography and Interpretable Graph Neural Networks: a Real-World and Cross-Site Study

Emotion recognition of EEG signals based on contrastive learning graph convolutional model

STILN: A novel spatial-temporal information learning network for EEG-based emotion recognition

A Novel Multi-Source Contrastive Learning Approach for Robust Cross-Subject Emotion Recognition in Eeg Data

Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation.

Temporal Contrastive Learning for Spiking Neural Networks

Contrastive Hierarchical Augmentation Learning for Modeling Cognitive and Multimodal Brain Network

CLEP: Contrastive Learning for Epileptic Seizure Prediction Using a Spatio-Temporal-Spectral Network

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

Consistent spectro-spatial features of human ECoG successfully decode naturalistic behavioral states

Identifying group-wise consistent sub-networks via spatial sparse representation of natural stimulus FMRI data

Single-Trial EEG Classification Using Spatio-Temporal Weighting and Correlation Analysis for RSVP-Based Collaborative Brain Computer Interface.

Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex

Leveraging temporal dependency for cross-subject-MI BCIs by contrastive learning and self-attention

Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli

EEG Based Parkinson Detection Through Supervised Information Enhanced Contrastive Learning