Abstract:Adaptive behavior rests on forming predictions based on previous statistical regularities encountered in the environment. Such regularities pertain not only to the contents of the stimuli ("what") but also their timing ("when"), and both interactively modulate sensory processing. In speech streams, predictions can be formed at multiple hierarchical levels, both in terms of contents (e.g. single syllables vs. words) and timing (e.g., faster vs. slower time scales). Whether and how these hierarchies map onto each other in terms of integrating "what" and "when" predictions remains unknown. Under one hypothesis neural hierarchies may link "what" and "when" predictions within sensory processing areas: with lower cortical regions mediating interactions for smaller units e.g., syllables, and higher cortical areas mediating interactions for larger units e.g., words. Alternatively, interactions between "what" and "when" predictions might rest on a generic, sensory-independent mechanism, mediated by common attention-related (e.g., frontoparietal) networks. To address those questions, we manipulated "what" and "when" predictions at two levels - single syllables and disyllabic pseudowords - while recording neural activity using magnetoencephalography (MEG) in healthy volunteers (N=22). We studied how syllable and/or word deviants are modulated by "when" predictability, both analyzing event-related fields and using source reconstruction and dynamic causal modeling to explain the observed effects in terms of the underlying effective connectivity. "When" predictions modulated "what" mismatch responses in a specific way with regards to speech hierarchy, such that mismatch responses to deviant words (vs. syllables) were amplified by temporal predictions at a slower (vs. faster) time scale. However, these modulations were source-localized to a shared network of cortical regions, including frontal and parietal sources. Effective connectivity analysis showed that, while mismatch responses to violations of "what" predictions modulated connectivity between regions, the integration of "what" and "when" predictions selectively modulated connectivity within regions, consistent with gain effects. These results suggest that the brain integrates "what" and "when" predictions that are congruent with respect to their hierarchical level, but this integration is mediated by a shared and distributed cortical network. This contrasts with recent studies indicating separable networks for different levels of hierarchical speech processing.

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Perceptual uncertainty explains activation differences between audiovisual congruent speech and McGurk stimuli

Prediction and constraint in audiovisual speech perception

Concurrent generative models inform prediction error in the human auditory pathway

Dynamic Causal Modeling Analysis Reveals the Modulation of Motor Cortex and Integration in Superior Temporal Gyrus During Multisensory Speech Perception

Development of a Bayesian Estimator for Audio-Visual Integration: A Neurocomputational Study

Auditory-visual integration during nonconscious perception

Anticipating multisensory environments: Evidence for a supra-modal predictive system

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain

Prediction Errors but Not Sharpened Signals Simulate Multivoxel fMRI Patterns during Speech Perception

On the Role of Noise in AudioVisual Integration: Evidence from Artificial Neural Networks that Exhibit the McGurk Effect

"What" and "when" predictions jointly modulate speech processing

Multimodal mismatch responses in mouse auditory cortex

Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception

The neural dynamics of auditory word recognition and integration

The noisy encoding of disparity model predicts perception of the McGurk effect in native Japanese speakers

Prediction tendency, eye movements, and attention in a unified framework of neural speech tracking

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Auditory, Visual, and Cross-Modal Mismatch Negativities in the Rat Auditory and Visual Cortices