Abstract:Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space. This work considers a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations. We introduce a Poincare probe, a structural probe projecting these embeddings into a Poincare subspace with explicitly defined hierarchies. We focus on two probing objectives: (a) dependency trees where the hierarchy is defined as head-dependent structures; (b) lexical sentiments where the hierarchy is defined as the polarity of words (positivity and negativity). We argue that a key desideratum of a probe is its sensitivity to the existence of linguistic structures. We apply our probes on BERT, a typical contextualized embedding model. In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean. In a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings. We demonstrate the findings with our Poincare probe via extensive experiments and visualization. Our results can be reproduced at https://github.com/FranxYao/PoincareProbe.

Deep Clustering of Text Representations for Supervision-Free Probing of Syntax

Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing

Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach

Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation

Wave to Syntax: Probing spoken language models for syntax

When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes

Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT

Cluster-norm for Unsupervised Probing of Knowledge

Probing BERT in Hyperbolic Spaces

A Latent-Variable Model for Intrinsic Probing

Probing a pretrained RoBERTa on Khasi language for POS tagging

Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs

Probing Pretrained Language Models for Lexical Semantics

A Matter of Framing: The Impact of Linguistic Formalism on Probing Results

Modelling the Lexicon in Unsupervised Part of Speech Induction

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

Syntax-augmented Multilingual BERT for Cross-lingual Transfer

Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing

Syntax-Enhanced Pre-trained Model

A Cascaded Unsupervised Model for PoS Tagging

On Eliciting Syntax from Language Models via Hashing