Abstract:Abstract Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel ‘compositional probe’ setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds.

Olive Oil is Made of Olives, Baby Oil is Made for Babies: Interpreting Noun Compounds using Paraphrases in a Neural Model

Paraphrase to Explicate: Revealing Implicit Noun-Compound Relations

Chinese Noun Compound Interpretation Based on Paraphrasing Verbs

Noun Compound Interpretation With Relation Classification and Paraphrasing

Visualizing and Understanding Neural Models in NLP

A Systematic Comparison of English Noun Compound Representations

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

Transfer and Multi-Task Learning for Noun-Noun Compound Interpretation

How is a “Kitchen Chair” like a “Farm Horse”? Exploring the Representation of Noun-Noun Compound Semantics in Transformer-based Language Models

"Covid vaccine is against Covid but Oxford vaccine is made at Oxford!" Semantic Interpretation of Proper Noun Compounds

SemEval-2013 Task 4: Free Paraphrases of Noun Compounds

Large-Scale Noun Compound Interpretation Using Bootstrapping and the Web as a Corpus

Semantic interpretation of compound nominalization using bilingual lexical ontologies

Using the Web as an Implicit Training Set: Application to Noun Compound Syntax and Semantics

Extracting paraphrases from a parallel corpus

Learning to Predict Novel Noun-Noun Compounds

Human and Automatic Interpretation of Romanian Noun Compounds

Neural Syntactic Preordering for Controlled Paraphrase Generation

Interpreting Verbal Metaphors by Paraphrasing

Acquisition of Phrase Correspondences using Natural Deduction Proofs

Beyond Pivot For Extracting Chinese Paraphrases