Abstract:Words change their meaning over time as well as in different contexts. The sense-aware contextualised word embeddings (SCWEs) such as the ones produced by XL-LEXEME by fine-tuning masked langauge models (MLMs) on Word-in-Context (WiC) data attempt to encode such semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal several novel insights such as (a) although there exist a smaller number of axes that are responsible for semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA. Source code is available at <a class="link-external link-https" href="https://github.com/LivNLP/svp-dims" rel="external noopener nofollow">this https URL</a> .

Dynamic Contextualized Word Embeddings

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge

Redefining Context Windows for Word Embedding Models: An Experimental Study

A Survey on Contextual Embeddings

Enriching Word Embeddings with Temporal and Spatial Information

Learning Context-Specific Word/Character Embeddings.

Dynamic Bernoulli Embeddings for Language Evolution

Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Dynamic Word Embeddings for Evolving Semantic Discovery

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Contextual Word Representations: A Contextual Introduction

Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

Investigating the Contextualised Word Embedding Dimensions Responsible for Contextual and Temporal Semantic Changes

Fast Extraction of Word Embedding from Q-contexts

Context-Dependent Sense Embedding.

Different Contexts Lead to Different Word Embeddings

Learning Word Embeddings from Intrinsic and Extrinsic Views

Exploring the Value of Personalized Word Embeddings

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Semantics or spelling? Probing contextual word embeddings with orthographic noise