Abstract:Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper attempts to explore how modern language models (LMs) handle different types of data across languages and modalities. Specifically, the authors propose a hypothesis that these models achieve this capability by learning a shared representation space. In this shared representation space, semantically similar inputs are mapped to nearby locations, even if they come from different languages or modalities. This hypothesis is referred to as the "Semantic Hub Hypothesis," inspired by the "Hub-and-Spoke Model" in neuroscience, which posits that semantic knowledge in the human brain is integrated through a cross-modal "semantic hub" that connects various modality-specific regions. ### Main Research Content 1. **Multilingual Processing**: - The authors demonstrate that when processing texts in different languages, the representations in the intermediate layers of the model are similar and can be interpreted through the model's primary pre-training language (usually English). - Experiments confirm that translated sentences have high similarity in the intermediate layer representations of the model. 2. **Cross-Modal Processing**: - The authors further explore the model's behavior when handling other types of data (such as arithmetic expressions, code, visual, and audio inputs). - Experiments show that these different modalities also exhibit similarity in the intermediate layer representations of the model and can be interpreted through the primary pre-training language. 3. **Intervention Experiments**: - The authors conducted intervention experiments and found that intervening in one type of data in the shared representation space can predictably affect the model's output when processing other types of data. - This indicates that the shared representation space is not merely a byproduct of large-scale training but a structure actively utilized by the model when processing inputs. ### Experimental Methods - **Multilingual Experiments**: - Using parallel text datasets, comparing the similarity of sentence representations in different languages in the intermediate layers of the model. - Verifying the dominant role of the primary language in intermediate layer representations by calculating the similarity between hidden states and the embedding vectors of the target language vocabulary. - **Arithmetic Expression Experiments**: - Comparing the similarity of arithmetic expressions represented by Arabic numerals and English words in the intermediate layers of the model. - Analyzing representation changes when predicting numbers using the Logit Lens method. - **Code Experiments**: - Analyzing the model's behavior when processing code, finding that the model tends to use free-form English to express future predictions in the intermediate layers rather than following program syntax. - Verifying the similarity of representations to semantic roles when processing list literals and function call parameters using the Logit Lens method. - **Formal Semantics Experiments**: - Using the COGS dataset to verify the similarity of sentence representations and their formal semantic structures in the intermediate layers of the model. - Ensuring the robustness of results through strictly controlled experiments. ### Conclusion The paper validates the "Semantic Hub Hypothesis" through a series of experiments, showing that modern language models indeed handle different types of data across languages and modalities by learning a shared representation space. This finding not only helps to better understand the internal mechanisms of current models but also provides new directions for future research aimed at better controlling and optimizing these models.

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

Evaluating the Representational Hub of Language and Vision Models

A Tri-network Model of Human Semantic Processing.

Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models

Investigating Inner Properties of Multimodal Representation and Semantic Compositionality with Brain-based Componential Semantics

Bilingual language processing relies on shared semantic representations that are modulated by each language

Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

Probing Multimodal Large Language Models for Global and Local Semantic Representations

Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models

Unraveling lexical semantics in the brain: Comparing internal, external, and hybrid language models

MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models

Semantic Composition in Visually Grounded Language Models

AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Thirteen Modalities

Relative Representations of Latent Spaces enable Efficient Semantic Channel Equalization

AllSpark: a Multimodal Spatiotemporal General Model

A Neurobiologically Constrained Cortex Model of Semantic Grounding With Spiking Neurons and Brain-Like Connectivity

Can computers understand words like humans do? Comparable semantic representation in neural and computer systems