Abstract:Learning vectors that capture the meaning of concepts remains a fundamental challenge. Somewhat surprisingly, perhaps, pre-trained language models have thus far only enabled modest improvements to the quality of such concept embeddings. Current strategies for using language models typically represent a concept by averaging the contextualised representations of its mentions in some corpus. This is potentially sub-optimal for at least two reasons. First, contextualised word vectors have an unusual geometry, which hampers downstream tasks. Second, concept embeddings should capture the semantic properties of concepts, whereas contextualised word vectors are also affected by other factors. To address these issues, we propose two contrastive learning strategies, based on the view that whenever two sentences reveal similar properties, the corresponding contextualised vectors should also be similar. One strategy is fully unsupervised, estimating the properties which are expressed in a sentence from the neighbourhood structure of the contextualised word embeddings. The second strategy instead relies on a distant supervision signal from ConceptNet. Our experimental results show that the resulting vectors substantially outperform existing concept embeddings in predicting the semantic properties of concepts, with the ConceptNet-based strategy achieving the best results. These findings are furthermore confirmed in a clustering task and in the downstream task of ontology completion.

Unsupervised Post-processing of Word Vectors via Conceptor Negation

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Visualizing and Understanding Neural Models in NLP

Neural Vector Conceptualization for Word Vector Space Interpretation

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding

An Empirical Study on Post-processing Methods for Word Embeddings

Conceptor-Aided Debiasing of Large Language Models

Adversarial Training for Unknown Word Problems in Neural Machine Translation

Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models

Learning Sparse Overcomplete Word Vectors Without Intermediate Dense Representations

Word vector computation based on implicit expression

Correcting the Common Discourse Bias in Linear Representation of Sentences using Conceptors

WordNet-based Concept Vector Space Model for Text Classification

Enriching Word Vectors with Subword Information

Effect of Post-processing on Contextualized Word Representations

Sentence Vector Model Based on Implicit Word Vector Expression

Improving Word Vector Model with Part-of-speech and Dependency Grammar Information

Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings

Improving Word Vector with Prior Knowledge in Semantic Dictionary.

Visual Concepts Tokenization