Abstract:Abstract Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel ‘compositional probe’ setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds.

Paraphrase to Explicate: Revealing Implicit Noun-Compound Relations

Chinese Noun Compound Interpretation Based on Paraphrasing Verbs

Noun Compound Interpretation With Relation Classification and Paraphrasing

Visualizing and Understanding Neural Models in NLP

Olive Oil is Made of Olives, Baby Oil is Made for Babies: Interpreting Noun Compounds using Paraphrases in a Neural Model

Transfer and Multi-Task Learning for Noun-Noun Compound Interpretation

A Systematic Comparison of English Noun Compound Representations

Extracting paraphrases from a parallel corpus

SemEval-2013 Task 4: Free Paraphrases of Noun Compounds

Using the Web as an Implicit Training Set: Application to Noun Compound Syntax and Semantics

Integrating Linguistic Knowledge to Sentence Paraphrase Generation.

Semantic interpretation of compound nominalization using bilingual lexical ontologies

Neural Syntactic Preordering for Controlled Paraphrase Generation

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

How is a “Kitchen Chair” like a “Farm Horse”? Exploring the Representation of Noun-Noun Compound Semantics in Transformer-based Language Models

Paraphrase Generation with Collaboration Between the Forward and the Backward Decoder

Hierarchical Sketch Induction for Paraphrase Generation

Neural paraphrasing by automatically crawled and aligned sentence pairs

Exploring Diverse Expressions for Paraphrase Generation

Bidirectional Long Short-Term Memory with Gated Relevance Network for Paraphrase Identification

Automated Generalization of Phrasal Paraphrases from the Web.