Abstract:The meaning of polysemous words often varies in a highly productive yet predictable way. Generalizing the regularity between conventional senses to derive novel word meaning is crucial for automated processing of non-literal language uses such as figurative expressions. We introduce a novel task called systematic word meta-sense extension (SWORME) to test and improve language models' ability to extend word meaning to denote new semantic domains (also called meta-senses) that bear regular semantic relations with existing senses. We found that language models prefer incremental lexical semantic change toward conceptually similar meta-senses such as logical metonymy, and are much worse at predicting highly non-literal meaning extensions such as metaphors. We propose a novel analogy-based method of word meaning extension, and show that it effectively improves language model systematicity in making both gradual and radical types of meta-sense extension. We further demonstrate that learning systematic meta-sense extensions benefits language models on multiple benchmarks of figurative language understanding.
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve
This paper aims to address the inadequacy of language models in extending word meanings to represent new semantic domains (referred to as metonymy). Specifically, the authors propose a new task—Systematic Word Metonymy Extension (SWORME)—to test and improve the ability of language models to predict word meaning extensions in natural contexts.
### Background and Motivation
1. **Polysemy and Meaning Extension**: Many words have multiple related but distinct meanings. For example, "get" and "grasp" can be used for both physical objects and abstract knowledge acquisition. Humans can extend these conventional meanings to new, metaphorical expressions, such as "steal information."
2. **Limitations of Existing Models**: Although existing distributed semantic models (such as word embeddings and contextual language models) can distinguish related word meanings and identify regular relationships between lexical items in some tasks, they perform poorly in generating and understanding new language usages, especially when dealing with highly non-literal meaning extensions (such as metaphors).
3. **Systematicity and Cognitive Theory**: Linguists and cognitive scientists believe that the process of extending polysemous words from conventional meanings to new meanings follows the same generative rules and that these extensions exhibit systematicity. However, neural language models often fail to generate new word meanings that have a predictable systematic relationship with existing meanings, consistent with their lack of systematicity in other areas.
### Main Contributions
1. **SWORME Task**: The authors introduce the SWORME task to evaluate the ability of language models to predict word meaning extensions in natural contexts. Specifically, given two semantically related domains (such as objects and information) and the polysemous usage of words describing these domains, the task is to extend the meaning of a word from one domain to another.
2. **Analogy-Based Word Meaning Extension Method**: The authors propose a new analogy-based method to infer new word meanings through the relational similarity between word meanings. This method effectively improves the systematicity of language models in both gradual and radical word meaning extensions.
3. **Experimental Validation**: Through experiments on the SWORME task, the authors demonstrate the superior performance of the analogy-based method in predicting word meaning extensions. This systematic extension capability helps improve the performance of language models on multiple metaphorical language understanding benchmarks.
### Experimental Results
1. **Model Performance**: All BERT-based models significantly outperformed the random baseline on the SWORME task, and model performance gradually improved with the increase in word meaning instances in the training data.
2. **Advantage of Analogy Chain Models**: Analogy-based chain models outperformed association-based chain models in predicting radical word meaning extensions, indicating that analogy or relational similarity between semantic domains is more important for systematic word meaning extension.
3. **Application to Metaphorical Language Understanding**: After learning the SWORME task, the performance of language models on metaphorical language understanding tasks significantly improved, especially in dealing with issues requiring objective knowledge, visual metaphors, social understanding, and cultural metaphors.
### Conclusion
By introducing the SWORME task and the analogy-based word meaning extension method, the authors successfully improved the systematicity of language models in generating and understanding new word meanings, which is crucial for handling metaphorical and other non-literal language usages.