Abstract:The integral capacity of human language together with semantic memory drives the linkage of words and their meaning, which theoretically is subject to cognitive control. However, it remains unknown whether, across different language modalities and input/output formats, there is a shared system in the human brain for word-meaning binding and how this system interacts with cognitive control. Here, we conducted a functional magnetic resonance imaging experiment based on a large cohort of subjects (50 females, 50 males) to comprehensively measure the brain responses evoked by semantic processing in spoken and written word comprehension and production tasks (listening, speaking, reading, and writing). We found that heteromodal word input and output tasks involved distributed brain regions within a frontal-parietal-temporal network and focally coactivated the anterior lateral visual word form area (VWFA), which is located in the basal occipitotemporal area. Directed connectivity analysis revealed that the VWFA was invariably under significant top-down modulation of the frontoparietal control network and interacts with regions related to attention and semantic representation. This study reveals that the VWFA is a key site subserving general semantic processes linking words and meaning, challenging the predominant emphasis on this area's specific role in reading or more general visual processes. Our findings also suggest that the dynamics between semantic memory and cognitive control mechanisms during word processing are largely independent of the modalities of input or output.SIGNIFICANCE STATEMENT Binding words and their meaning into a coherent whole during retrieval requires accessing semantic memory and cognitive control, allowing our thoughts to be expressed and comprehended through mind-external tokens in multiple modalities, such as written or spoken forms. However, it is still unknown whether multimodal language comprehension and production share a common word-meaning binding system in human brains and how this system is connected to a cognitive control mechanism. By systematically measuring brain activity evoked by spoken and written verbal input and output tasks tagging word-meaning binding processes, we demonstrate a general word-meaning binding site within the visual word form area (VWFA) and how this site is modulated by the frontal-parietal control network.

Interactions Between Modal and Amodal Semantic Areas in Spoken Word Comprehension.

Chinese semantic processing cerebral areas

Working Memory Asymmetrically Modulates Auditory and Linguistic Processing of Speech.

Neural Substrates of Amodal and Modality-Specific Semantic Processing Within the Temporal Lobe: A Lesion-Behavior Mapping Study of Semantic Dementia.

Modality independence of word comprehension

Semantic Congruent Audiovisual Integration During the Encoding Stage of Working Memory: an ERP and Sloreta Study

Task modulation of spatiotemporal dynamics in semantic brain networks: An EEG/MEG study

A Heteromodal Word-Meaning Binding Site in the Visual Word Form Area under Top-Down Frontoparietal Control

The role of variation in phonological and semantic working memory capacities in sentence comprehension: neural evidence from healthy and brain-damaged individuals

Frequency-specific directed interactions between whole-brain regions during sentence processing using multimodal stimulus

Investigating Inner Properties of Multimodal Representation and Semantic Compositionality with Brain-based Componential Semantics

Task-modulated Activation and Functional Connectivity of the Temporal and Frontal Areas During Speech Comprehension

Modality-Specific and Amodal Language Processing by Single Neurons

Differential Contributions of Bilateral Ventral Anterior Temporal Lobe and Left Anterior Superior Temporal Gyrus to Semantic Processes

No Morphological Markers, No Problem: ERP Study Reveals Semantic Contribution to Distinct Neural Substrates Between Noun and Verb Processing in Online Sentence Comprehension

Basic Linguistic Composition Recruits the Left Anterior Temporal Lobe and Left Angular Gyrus During Both Listening and Reading

A social-semantic working-memory account for two canonical language areas

Modality- and Task-specific Brain Regions Involved in Chinese Lexical Processing

Spatial and Semantic Processing Between Audition and Vision: an Event-Related Potential Study

Both the Middle Temporal Gyrus and the Ventral Anterior Temporal Area Are Crucial for Multimodal Semantic Processing: Distortion-corrected fMRI Evidence for a Double Gradient of Information Convergence in the Temporal Lobes

A Novel Audiovisual P300-Speller Paradigm Based on Cross-Modal Spatial and Semantic Congruence