Abstract:A central challenge for the brain is how to combine separate sources of information from different sensory modalities to optimally represent objects and events in the external world, such as combining someone's speech and lip movements to better understand them in a noisy environment. At the level of individual neurons, audiovisual stimuli often elicit super-additive interactions, where the neural response is greater than the sum of auditory and visual responses. However, investigations using electroencephalography (EEG) to record brain activity have revealed inconsistent interactions, with studies reporting a mix of super- and sub-additive effects. A possible explanation for this inconsistency is that standard univariate analyses obscure multisensory interactions present in EEG responses by overlooking multivariate changes in activity across the scalp. To address this shortcoming, we investigated EEG responses to audiovisual stimuli using inverted encoding, a population tuning approach that uses multivariate information to characterise feature-specific neural activity. Participants (n=41) completed a spatial localisation task for both unisensory stimuli (auditory clicks, visual flashes) and combined audiovisual stimuli (spatiotemporally congruent clicks and flashes). To assess multivariate changes in EEG activity, we used inverted encoding to recover stimulus location information from event-related potentials (ERPs). Participants localised audiovisual stimuli more accurately than unisensory stimuli alone. For univariate ERP analyses we found an additive multisensory interaction. By contrast, multivariate analyses revealed a super-additive interaction ~180 ms following stimulus onset, such that the location of audiovisual stimuli was decoded more accurately than that predicted by maximum likelihood estimation. Our results suggest that super-additive integration of audiovisual information is reflected within multivariate patterns of activity rather than univariate evoked responses.

Better Together: Integrating Multivariate with Univariate Methods, and MEG with EEG to Study Language Comprehension

[The Neural Encoding of Continuous Speech - Recent Advances in EEG and MEG Studies].

Language Cognition and Language Computation—human and Ma-Chine Language Understanding

Elife Assessment: Finding Structure During Incremental Speech Comprehension

Analyzing Differences in Processing Nouns and Verbs in the Human Brain using Combined EEG and MEG Measurements

Time-domain Analysis of Neural Tracking of Hierarchical Linguistic Structures

A Multivariate analysis on evoked components of Chinese semantic congruity: an OP-MEG study with EEG

Modulation of brain activity by psycholinguistic information during naturalistic speech comprehension and production

A 10-hour within-participant magnetoencephalography narrative dataset to test models of language comprehension

Parallel or sequential? Decoding conceptual and phonological/phonetic information from MEG signals during language production

An inclusive multivariate approach to neural localization of language components

Predictive coding across the left fronto-temporal hierarchy during language comprehension

Temporal and Hemispheric Dynamics in Neural Processing of Auditory and Speech Stimuli Across Linguistic Complexity: A MEG Source Space Study

The Structure and Statistics of Language Jointly Shape Cross-frequency Neural Dynamics During Spoken Language Comprehension

Integrating LLM, EEG, and Eye-Tracking Biomarker Analysis for Word-Level Neural State Classification in Semantic Inference Reading Comprehension

Divergences between Language Models and Human Brains

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

Understanding language-elicited EEG data by predicting it from a fine-tuned language model

Tripartite organization of brain state dynamics underlying spoken narrative comprehension