Abstract:Recently, much work has concerned itself with the enigma of what exactly PLMs (pretrained language models) learn about different aspects of language, and how they learn it. One stream of this type of research investigates the knowledge that PLMs have about semantic relations. However, many aspects of semantic relations were left unexplored. Only one relation was considered, namely hypernymy. Furthermore, previous work did not measure humans' performance on the same task as that solved by the PLMs. This means that at this point in time, there is only an incomplete view of models' semantic relation knowledge. To address this gap, we introduce a comprehensive evaluation framework covering five relations beyond hypernymy, namely hyponymy, holonymy, meronymy, antonymy, and synonymy. We use six metrics (two newly introduced here) for recently untreated aspects of semantic relation knowledge, namely soundness, completeness, symmetry, asymmetry, prototypicality, and distinguishability and fairly compare humans and models on the same task. Our extensive experiments involve 16 PLMs, eight masked and eight causal language models. Up to now only masked language models had been tested although causal and masked language models treat context differently. Our results reveal a significant knowledge gap between humans and models for almost all semantic relations. Antonymy is the outlier relation where all models perform reasonably well. In general, masked language models perform significantly better than causal language models. Nonetheless, both masked and causal language models are likely to confuse non-antonymy relations with antonymy.

Exploring Accurate and Generic Simile Knowledge from Pre-trained Language Models

Can Pre-trained Language Models Interpret Similes As Smart As Human?

Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

Neural Multitask Learning for Simile Recognition

A Study of Pre-trained Language Models in Natural Language Processing

Sem4SAP: Synonymous Expression Mining From Open Knowledge Graph For Language Model Synonym-Aware Pretraining

Better Simultaneous Translation with Monotonic Knowledge Distillation.

A Comprehensive Evaluation of Semantic Relation Knowledge of Pretrained Language Models and Humans

A Survey on Knowledge-Enhanced Pre-trained Language Models

ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models

Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge

Writing Polishment with Simile: Task, Dataset and A Neural Approach

I run as fast as a rabbit, can you? A Multilingual Simile Dialogue Dataset

MAPS-KB: A Million-scale Probabilistic Simile Knowledge Base

Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work

A Survey of Knowledge Enhanced Pre-trained Language Models

A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models